DepthGraft Reconstructs Cross-Layer KV Sharing to Cut Cache by One Third
Summary:DepthGraft introduces a hierarchical cross-layer KV-sharing architecture for Pre-LN language models. Instead of computing every layer's key-value state independently, it reconstructs deep-layer KV representations from stride-aligned shallow and middle layers. The method provides both cache savings and a direct gradient path from deep layers to source KV projections, potentially improving retrieval geometry and value subspaces. The reported experiments span dense and MoE models from 0.5B to 16B
Core Idea: Making Deep-Layer KV Benefit from Cross-Layer Sharing
DepthGraft addresses a structural weakness in Pre-LN Transformers: as depth increases, information in the residual stream can be diluted, so adding layers does not always produce more effective computation. Existing cross-layer connections often introduce extra computation or memory traffic. DepthGraft revisits cross-layer KV sharing, reconstructing deep-layer KV states from shallow and middle layers. This saves cache space while also creating direct gradient paths from deep layers to the source KV projections.
The proposal operates on two levels. The first is explicit engineering efficiency: fewer independent KV states must be stored and written during autoregressive decoding. The second is an implicit structural regularizer. Because deep-layer losses can directly update the source projections, the shared representation is shaped for downstream use rather than merely reused as a passive cache artifact. The paper argues that a suitable topology sharpens key-retrieval directions and enriches the value-content subspace.
This differs from conventional KV-cache compression. Standard compression typically changes cache quantity or quantization precision during inference. DepthGraft incorporates sharing into the pretrained architecture, allowing the cache structure to participate in representation learning.
Hierarchical KV Reconstruction
DepthGraft reconstructs deep-layer KV representations from stride-aligned shallow and middle source pairs. Compared with adjacent-layer sharing, this topology carries gradients across larger network distances. Compared with a single global anchor, it avoids forcing all layers to depend continuously on one early representation and distributes non-local gradient flow among multiple sources.
The main engineering benefit is reduced demand for independent deep-layer KV projections and storage. Although the exact memory formula differs between GQA and MLA, both generally scale with sequence length, layer count, KV-head count, and head dimension. Reconstructing or sharing deep-layer states can therefore reduce capacity and bandwidth pressure during long-context decoding.
KV sharing does not automatically guarantee better quality. Adjacent layers may be highly correlated, but sharing can also constrain the formation of new deep representations. A global anchor may be stable while encouraging excessive dependence on one early state. DepthGraft attempts to balance representation diversity, gradient-path length, and cache reduction through layered source pairs and stride alignment.
Sharing as Structural Regularization
In a conventional Transformer, deep layers influence early projections only indirectly through the residual stream and sequential computation. A KV-reuse edge introduces a direct path from deep-layer loss to the source KV projection. The model must therefore learn source states that remain geometrically useful to deeper tasks, not merely learn to read them.
The proposed explanation has two components. First, sharing can concentrate key-space retrieval directions and reduce unproductive degrees of freedom in the query-to-key relationship. Second, sharing can broaden the value subspace, allowing deeper layers to extract more diverse information instead of repeatedly relying on a narrow early state. Because incorrect sharing can also cause gradient interference or representation collapse, topology quality is central rather than incidental.
Experimental Results and Limits
The paper evaluates dense and MoE models from 0.5B to 16B parameters under both GQA and MLA. The most prominent result comes from a 16B MoE model: absolute gains of 7.0 percentage points on MMLU-Pro, 6.6 on CMMLU, and 6.4 on C3. KV-cache usage falls by roughly one third while retaining the claimed prefill early-completion behavior.
The cross-model and cross-attention results are encouraging, but the available summary does not disclose the full training budget, parameter-matched controls, computational overhead, or inference throughput. During pretraining, shallow and middle layers may carry extra reconstruction responsibilities, while routing and reconstruction operations can offset part of the cache benefit.
The benchmark gains also require caution. MMLU-Pro, CMMLU, and C3 primarily measure knowledge and reasoning; they do not independently establish long-context quality, generation reliability, or training stability. Reproduction should examine:
- Gains against standard Pre-LN baselines with equivalent parameter counts, token budgets, and optimizer settings;
- Interactions between shared topology, KV quantization, and grouped attention;
- Whether results from a 16B MoE model transfer to larger activated and total parameter scales;
- Effects on streaming latency, batched throughput, and multi-GPU execution;
- Whether the learned sharing structure can be applied to existing checkpoints without retraining.
Practical Adoption Guidance
For research teams, the priority should be to reproduce the topology, gradient-path, and cache ablations rather than focus only on the headline MMLU-Pro gain. Measurements should track independent KV states per layer, cache bandwidth, training FLOPs, and inference throughput. A smaller cache does not automatically imply a faster end-to-end system.
For inference infrastructure teams, implementation should focus on memory layouts, prefill scheduling, and batched decoding for cross-layer reconstruction. If reconstruction requires additional matrix operations, theoretical memory savings may not translate into latency gains. Actual performance depends on cache capacity, sequence length, batch size, and GPU memory bandwidth.
Developers should not assume that existing checkpoints can adopt the structure without loss. DepthGraft changes representation formation and gradient flow during pretraining rather than acting as a simple inference-time plugin. Before deployment, teams should verify licensing, code and checkpoint availability, and rerun perplexity, instruction-following, long-context, and multilingual evaluations.
Conclusion
DepthGraft's central contribution is not merely reducing KV-cache usage by about one third. It reframes cross-layer sharing as a structural regularizer that participates in pretraining. Its key hypothesis is that deep-layer supervision should directly shape the KV geometry of shallow and middle layers. If this holds at larger scale, DepthGraft may provide a new path for scaling deep Pre-LN models while controlling inference memory.
— END —Source: Hugging Face Daily Papers (2026-11-02)
Tags: #DepthGraft #KV Cache #Cross-Layer Sharing #Pre-LN #MoE
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments