LowRAM Compresses KV Cache Asymmetrically for Up to 5x Faster Decoding

Summary:LowRAM treats the key and value caches as structurally different rather than compressing them uniformly. Within each attention head, a lightweight predictor reconstructs value vectors from their corresponding keys, while residual storage is reserved for tokens that the predictor reconstructs poorly. Cross-layer low-rank compression is then applied to keys, producing an asymmetric pipeline that the authors report remains close to lossless across three model families. At 4x compression, LowRAM ac

---

What LowRAM Changes

LowRAM does not compress keys and values uniformly. It exploits their different structures: values are reconstructed from their corresponding keys, while keys are compressed through a cross-layer low-rank representation. Across three model families, the authors report near-lossless behavior at 4x compression and up to 5x higher peak decoding throughput. When combined with quantization and token eviction, the pipeline reaches 26.9x compression while retaining 94.5% of baseline performance.

Conventional KV-cache compression often searches for redundancy across tokens, layers, feature dimensions, or numerical precision. LowRAM adds another axis: within an attention head, value vectors are often predictable from the corresponding key vectors.

Core Mechanism: Predict Values and Keep Difficult Residuals

Autoregressive decoding caches the keys and values required by every attention layer. Approximate cache memory scales as:

[ M_{KV}=2,L,H_{kv},d_h,N,s ]

where (L) is the number of layers, (H_{kv}) the number of KV heads, (d_h) the head dimension, (N) the context length, and (s) the bytes per scalar. The goal is not to shrink model parameters, but to avoid KV-cache capacity limiting batch size, concurrency, and context length.

LowRAM fits a lightweight value predictor for each attention head. It reconstructs values from keys and stores residuals only for tokens that the predictor reconstructs poorly. Formally, if (\hat{V}=f(K)) explains most of the value structure, the cache can retain only (\delta=V-\hat{V}) for selected tokens.

Error-driven token selection is essential. If the predictor works well for most tokens, storage falls sharply. Tokens with high reconstruction error must retain residuals because otherwise errors can propagate through attention across layers and across a long sequence. In generative workloads, small numerical deviations can also change sampling trajectories, so average reconstruction error alone is not a sufficient quality metric.

The paper reports 4x near-lossless value compression. However, "near-lossless" still depends on the model, task, and evaluation protocol. Perplexity or average task scores should be supplemented with worst-case token error, long-context retrieval accuracy, and output-trajectory divergence.

Cross-Layer Low-Rank Key Compression

Values exploit within-head predictability, while keys are compressed through cross-layer low-rank structure. The assumption is that keys in adjacent or related layers share a lower-dimensional subspace and can therefore be represented through a shared basis and layer-specific coefficients.

If (K_l\approx U_lV_l^\top), the storage cost can shift from a full matrix that grows with token count to smaller basis matrices and low-dimensional coefficients. Low-rank approximation reduces memory, but it also introduces approximation error and recurring projection or matrix-multiplication work during decoding. Without fused dequantization and projection kernels, memory savings may not translate into throughput.

The complete pipeline is therefore asymmetric: keys follow a cross-layer low-rank path, while values follow key-based prediction plus selective residual storage. This is not simply assigning different compression rates to keys and values; it treats them as different statistical structures.

Fitting and Initialization Costs

The value predictor can be initialized in two ways:

1. Fit from scratch at test time. This suits workloads such as multi-query inference that can absorb a one-off cost. It can adapt to the active workload but introduces slow cold starts and greater latency variance.
2. Meta-learned initialization. This reduces online prefill overhead by moving more adaptation work offline, but requires robustness across model weights, context distributions, and quantization configurations.

Production deployments should bind predictors to model fingerprints. Mixing a predictor with another base model, LoRA adapter, KV-head layout, or cache format can cause silent reconstruction errors.

Why Peak Throughput Can Improve by 5x

Compression does not automatically improve speed. The benefit appears only if lower KV-cache usage is converted into larger batches, higher request concurrency, or reduced host-device and inter-GPU transfer.

The paper reports that LowRAM reconstructs only selected KV pairs at each decoding step, limiting additional compute. At 4x compression, peak decoding throughput increases by as much as 5x, suggesting that capacity relief and reduced memory traffic can outweigh reconstruction cost in the evaluated regime.

The 5x figure is a peak result, not necessarily an average end-to-end gain for every workload. Evaluation should report prefill time, time to first token, time per output token, steady-state throughput, concurrent requests, peak KV-cache memory, and saturation behavior. A serving workload dominated by low-concurrency, short prompts may realize far less benefit than a high-concurrency benchmark.

Composition with Quantization and Eviction

LowRAM can be layered with quantization and KV-token eviction. Low-rank key compression reduces the base representation, value prediction reduces value storage, quantization lowers per-element cost, and eviction removes low-value historical tokens.

The paper reports 26.9x compression on long contexts while retaining 94.5% of baseline performance. The composability is attractive because each mechanism targets a different source of redundancy. Yet errors can accumulate: low-rank approximation alters key directions, prediction residuals retain value bias, quantization introduces rounding noise, and eviction may remove critical evidence.

End-to-end long-context evaluations are therefore essential. Multi-hop reasoning, repository-scale code understanding, cross-document retrieval, and tasks requiring recall of early tokens are especially sensitive to reconstruction errors or accidental eviction.

How to Interpret the Evaluation

Testing three model families is stronger evidence than a single-model demonstration, but it does not establish universal architecture-level validity. Differences in GQA, Multi-Query Attention, RoPE, head dimensions, and pretraining distributions can change key-value predictability.

The most important checks are:

• The baseline models and quality metric behind the 4x near-lossless claim;
• The context length and task set behind the 26.9x result;
• Whether predictor fitting and cache reconstruction are included in throughput measurements;
• Whether temporary reconstruction buffers and allocator fragmentation are counted in peak memory;
• Variance across seeds, prompt distributions, and worst-case inputs;
• Hardware-matched comparisons with H2O, SnapKV, StreamingLLM, KVQuant, and related methods.

Deployment Guidance

Four items deserve priority before production adoption:

1. Measure capacity first, then speed. Verify that the compressed cache reduces GPU memory and can actually increase batch size on the target accelerator.
2. Fuse reconstruction into the attention path. Standalone deserialization and projection kernels can erase gains through memory traffic and kernel-launch overhead.
3. Version predictors with model fingerprints. Base weights, LoRA adapters, KV-head layouts, and cache formats must remain tightly coupled.
4. Validate with production traffic. Test long-prompt prefill, short-output high concurrency, sustained multi-turn conversations, and retrieval-heavy tasks while tracking TTFT, TPOT, throughput, memory, and quality.

For individual developers, LowRAM is better treated as an inference-engine or cache-management research component than as a manual model-weight modification. Its central contribution is demonstrating that keys and values should not be forced into the same representation or storage policy. Whether that structural asymmetry becomes a production gain depends on cache layout, kernel fusion, and real serving workloads.

---

Source: Hugging Face Daily Papers (2026-11-04)

— END —

Tags: #KV Cache Follow #Low-Rank Compression Follow #Prediction Residuals Follow #Long-Context Inference Follow #Attention Optimization Follow