Profiling Huginn on B200: Shared Weights Do Not Guarantee Lower HBM Traffic

Summary:Dinara DL profiled the Huginn looped Transformer on an NVIDIA B200 to determine whether reusing the same weights across iterations actually reduces HBM traffic. A chain of eight matrix multiplications using one shared 72 MiB weight ran 24% faster than the same chain with eight separate copies and performed almost no HBM reads. At 288 MiB, however, the working set exceeded the L2 cache and the advantage largely disappeared. A full Huginn Core run with random weights showed eight iterations readi

---

Core Finding

Huginn reuses the same weights across iterations, but that does not automatically make the GPU read less data from HBM. Dinara DL's profiling on an NVIDIA B200 shows that the benefit depends on whether those weights can remain resident in the chip's cache between iterations. With one shared 72 MiB weight, an eight-matrix-multiplication chain ran 24% faster than the equivalent chain with eight separate copies and performed almost no HBM reads. At 288 MiB, however, the working set exceeded L2 capacity and the advantage largely disappeared.

The result separates two ideas that are often conflated: weight sharing reduces storage requirements, while weight residency reduces memory traffic. Reusing the same tensor is not enough to guarantee the latter.

Architecture: What Loop Recomputation Actually Saves

Huginn advances a sequence state by repeatedly executing computation blocks with shared parameters. An eight-iteration loop can be represented as eight matrix multiplications over the same weight set. One implementation stores a single weight tensor and references it repeatedly; the other stores eight independent copies.

The mathematical operations are equivalent, but storage and data movement are not:

• Shared weights reduce weight capacity from 8 × S to S, where S is the size of one weight set.
• On-chip residency requires the shared weight to remain in SRAM or L2 until its next use, avoiding another HBM fetch.
• If the weight is evicted between iterations, L2 must reload it from HBM, and total traffic approaches 8 × S again.
• If it remains resident, HBM reads can approach zero, although all eight rounds of computation still have to execute.

Weight sharing therefore offers a clear capacity benefit. It does not unconditionally provide a bandwidth benefit.

B200 Results: Cache Capacity Changes the Outcome

In a controlled chain of eight matrix multiplications, sharing one 72 MiB weight ran 24% faster than using eight separate copies and generated almost no HBM reads. The shared working set fit within the B200's cache hierarchy, allowing later iterations to reuse already-loaded data.

When a single weight grew to 288 MiB and exceeded L2 capacity, the performance difference largely disappeared. Cache misses and memory stalls again dominated execution. A shared-weight label therefore cannot replace direct analysis of cache behavior.

| Test | Weight layout | HBM behavior | Result | |---|---|---|---| | 72 MiB weight, 8 matrix multiplications | One shared tensor | Almost no HBM reads | 24% faster than 8 separate copies | | 288 MiB weight, 8 matrix multiplications | One shared tensor | No stable cache residency | Advantage largely disappeared | | Full Huginn Core, 8 iterations | Looped shared weights | About 8× the data read | Runtime was about 8× a single iteration |

Why the Full Model Did Not Inherit the Microbenchmark Gain

The full Huginn Core uses a teaching implementation with random weights. Its eight iterations read roughly eight times as much HBM data and took about eight times as long as one iteration. This does not invalidate the cache effect observed in the microbenchmark; it shows that a complete execution graph introduces a different memory-access pattern.

Potential causes include:

1. The working set is no longer a single tensor. Activations, temporary buffers, and runtime data compete for cache capacity.
2. The scheduler does not guarantee cross-iteration residency. Synchronization, queue changes, or eviction can occur between kernel launches.
3. Other operators add memory traffic. Reductions, activation functions, normalization, and tensor transformations may read or rewrite intermediate data.
4. Weight access is not explicitly cache-persistent. Conventional GPU kernels rely on the normal L1/L2 hierarchy rather than reserving a weight for the lifetime of a recurrent loop.
5. The teaching implementation is not residency-optimized. It is closer to a structural reference implementation than a production-grade training or inference kernel.

The more precise conclusion is that tensor aliasing or loop reuse alone cannot guarantee on-chip persistence across kernels.

Engineering Trade-offs

Capacity versus Locality

Shared weights can reduce static memory consumption, but they also concentrate cache pressure. A weight reused immediately in the next iteration is a strong residency candidate. If intervening operations evict it, memory capacity may shrink while cache thrashing worsens.

Bandwidth versus Compute

Even successful residency does not remove the FLOPs of each iteration. It can reduce stalls and improve throughput, but the matrix computations still have to run. The roughly eightfold runtime of the full loop illustrates that end-to-end gains depend on arithmetic intensity and pipeline efficiency as well as bandwidth.

General-purpose versus Persistent Kernels

A persistent kernel can execute the entire recurrent loop inside one kernel launch and explicitly manage on-chip data residency. This avoids invalidation at kernel boundaries and may expose additional reuse opportunities.

The costs are equally important:

• Persistent kernels are harder to integrate with general compiler and library ecosystems.
• Dynamic shapes, sparse operators, and complex control flow are difficult to optimize.
• Long-lived SM and on-chip storage occupancy may reduce concurrency for other work.
• Cache capacities and coherence behavior vary across GPU generations, limiting portability.

Evaluation Boundaries

The current results come from a controlled matrix-multiplication chain and a Huginn Core teaching implementation using random weights. They establish a useful relationship among cache capacity, weight sharing, and HBM traffic, but they do not yet show that real training workloads receive the same speedup.

A complete evaluation should report:

• End-to-end runtime and per-kernel timing breakdowns;
• HBM reads and writes, plus L1/L2 hit rates;
• Activation sizes, fusion strategy, and intermediate-buffer traffic;
• Results across batch sizes, sequence lengths, and weight shapes;
• Stability with trained weights as well as random weights;
• Fair comparisons against normal expanded kernels, CUDA Graphs, fusion, and persistent-kernel implementations.

The reported 24% improvement is specific to the microbenchmark and should not be generalized into an end-to-end Huginn training-throughput gain.

Deployment Recommendations

Teams implementing recurrent weight residency should proceed in stages:

1. Use Nsight Compute to collect per-iteration HBM and cache counters before changing the architecture.
2. Keep weight sharing, weight residency, and operator fusion separate as engineering questions.
3. Establish a performance upper bound using weight sizes that fit comfortably in the on-chip cache.
4. Test single-kernel loops, CUDA Graphs, and persistent kernels while measuring launch and synchronization overhead.
5. For larger weights, explore tiling, hierarchical caching, or compression instead of assuming reuse implies cache hits.
6. Evaluate throughput, latency, peak memory, and convergence behavior on the real model.

Practical Significance

Huginn's profiling establishes a concrete boundary for looped Transformers: sharing weights is an effective way to reduce model state and static memory usage, but it becomes a bandwidth optimization only when cache capacity, scheduling, and explicit residency mechanisms are aligned. The optimization target is not whether two operations point to the same tensor; it is whether the weight remains in the appropriate storage tier across iterations.

---

Source: Reddit r/MachineLearning (2026-11-04)

— END —

Tags: #Huginn Follow #Looped Transformers Follow #GPU Profiling Follow #HBM Traffic Follow #B200 Follow