CAFIN Fuses CUDA Kernels to Accelerate the AdaFisher Second-Order Optimizer
Summary:CAFIN targets the GPU execution overhead of the AdaFisher optimizer by progressively fusing Fisher Information Matrix preconditioner construction, exponential moving-average updates, gradient preconditioning, momentum accumulation, weight decay, and parameter updates into native CUDA kernels. Rather than changing AdaFisher's optimization formulation, the framework reduces kernel-launch overhead, repeated memory traffic, and temporary allocations through shared-memory reuse, register-resident com
Core Idea: Preserve the Optimizer, Redesign GPU Execution
CAFIN targets the GPU execution overhead of AdaFisher, an optimizer that approximates second-order curvature using diagonal Kronecker structures in the Fisher Information Matrix (FIM) and exponential moving averages. The paper argues that the main bottleneck is not the mathematical formulation but the PyTorch implementation, which splits preconditioner construction, moving-average updates, gradient scaling, momentum accumulation, weight decay, and parameter writes into sequential CUDA kernels. This sequence creates launch overhead, temporary allocations, and repeated memory traffic.
CAFIN leaves the AdaFisher update formulation unchanged and reorganizes its GPU execution path. It progressively fuses the optimizer stages into native CUDA kernels, using shared-memory reuse, register-resident computation, and in-place parameter updates to reduce kernel launches and unnecessary HBM transfers.
Fusion Pipeline
The fused pipeline covers six major operations:
- FIM preconditioner construction;
- exponential moving-average updates;
- gradient preconditioning;
- momentum accumulation;
- weight decay;
- parameter updates.
Shared memory retains data across dependent stages, while intermediate computations stay in registers where possible. In-place updates avoid allocating separate tensors for each optimizer phase. The result is a more continuous GPU data flow instead of a sequence of small kernels connected through global memory.
This is particularly relevant for optimizers, which are often bandwidth-bound. Much of their runtime is spent reading gradients, optimizer statistics, and parameters, then writing updated values. Fusion can improve throughput by enabling on-chip reuse, although the realized gain still depends on tensor sizes, parameter grouping, occupancy, and compiler instruction selection.
Measured Results
The experiments use ResNet-18 on CIFAR-10, CIFAR-100, and Tiny ImageNet, with NVIDIA GTX 1660 Ti and RTX 3060 Laptop GPUs.
- Isolated FIM construction latency falls by up to 78.4%.
- Full optimizer-step latency decreases by 2.1%–26.9%.
- Approximately 18.91 MB of intermediate GPU allocations is removed per optimizer step.
- Numerical outputs remain consistent within the predefined FP32 tolerance.
- Convergence accuracy is comparable to the original implementation across the evaluated datasets and random seeds.
The much larger improvement in isolated FIM construction indicates that preconditioner generation and fragmented neighboring operators are the main bottleneck. End-to-end optimizer gains are naturally smaller because a training step also includes data loading, backward propagation, gradient generation, and other components outside the fused region.
Engineering Trade-offs and Evaluation Limits
The principal benefit of CAFIN is that it preserves AdaFisher's optimization semantics, allowing teams to replace the execution backend without changing the underlying algorithm. That does not guarantee universal applicability, however.
Fused kernels typically require assumptions about parameter shapes, model structure, and precision layouts. Irregular parameter groups or dynamic tensor sizes can reduce register reuse, shared-memory efficiency, and occupancy. Native CUDA also narrows portability: systems without NVIDIA GPUs need an alternative backend or a fallback to the PyTorch implementation.
The experiments focus on ResNet-18 and small-scale image-classification workloads. They do not establish whether the same gains transfer to large language models, where parameter scale, gradient distributions, optimizer dynamics, sharding, and distributed synchronization differ substantially. The summary also omits full training curves, repeated-run uncertainty, end-to-end training time, and comparisons with a broader set of GPU-aware optimizers. The results support a narrower conclusion: this implementation can materially reduce AdaFisher's execution overhead, not that every second-order optimizer will receive the same acceleration.
Practical Adoption Guidance
Teams evaluating CAFIN should profile the original PyTorch implementation before replacing it. Useful baselines include optimizer-step latency, peak memory, training throughput, final validation accuracy, and numerical error across different parameter groups.
Research deployments should first test models with uniform parameter shapes and stable grouping, as these are the most favorable cases for shared-memory reuse and register residency. It is also important to retain the original implementation as a fallback and use a CUDA profiler or CUDA Graph analysis to identify remaining launch, synchronization, and memory-transaction overhead.
For large-model training, CAFIN should be evaluated alongside optimizer-state quantization, ZeRO or FSDP sharding, and computation-communication overlap. CAFIN optimizes the internal execution of an optimizer step; it does not remove the costs of gradient synchronization, parameter communication, or distributed training. Its practical value therefore depends on how large a share of total training time is consumed by the optimizer itself.
CAFIN demonstrates a pragmatic systems approach: rather than introducing another curvature approximation, it removes implementation overhead through GPU-native fusion. For workloads limited by memory bandwidth and kernel launches, this can deliver more immediately reproducible engineering benefits than adding complexity to the optimizer mathematics.
— END —Source: ArXiv Machine Learning (cs.LG) (2026-11-13)
Tags: #AdaFisher #CUDA Kernel Fusion #Second-Order Optimization #Fisher Information Matrix #GPU Performance
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments