RidgeFill Accelerates Long-Context Prefill by Skipping Upper-Layer Computation Without Training
Summary:RidgeFill is a training-free prefill optimization for long-context inference. It computes lower transformer layers exactly, uses closed-form ridge regression to predict the inputs of upper layers, and writes the predicted states into the KV cache. No prompt tokens are dropped, while selected rows, the final few thousand tokens, and every decoding step still use the full model. Experiments report a 1.25–1.31x improvement in time to first token for 16K–120K-token prompts on an A100 and up to a 1.
Core Idea: Turn Part of Prefill into Cacheable Approximate Computation
RidgeFill targets the most expensive phase of long-context inference: prefill. A standard Transformer computes every layer for the prompt, and attention cost generally grows approximately quadratically with context length. GQA, MQA, and linear-attention alternatives reduce KV-cache pressure, but they do not necessarily eliminate the full prompt computation in standard attention.
RidgeFill introduces a split layer:
- Layers below the split are computed exactly as usual.
- Starting from the split-layer state, an independent map predicts the input of every upper layer.
- Those predictions are used to populate the corresponding Key and Value caches.
- Selected rows, the final few thousand tokens, and every decoding step still run through the full model.
The method therefore does not drop prompt tokens or simply skip entire layers. It replaces part of the exact upper-layer computation with a cache-filling approximation.
Mapping Mechanism: Closed-Form Ridge Regression
RidgeFill fits the mapping with ridge regression. Let the split-layer states be (X\in\mathbb{R}^{d\times n}), the target-layer inputs be (Y\in\mathbb{R}^{m\times n}), and the linear map be (W\in\mathbb{R}^{m\times d}). Its closed-form solution is:
[ W=(Y X^\top+\lambda I)^{-1}X X^\top ]
The regularization term limits sensitivity to collinear calibration features and controls prediction variance. Once fitted, each map can be cached and applied directly to the split-layer state. No gradient-based model training is required.
The authors note that YOCO and DeepSeek-V4.1-Flash train a similar projection into the model itself. RidgeFill differs by retrofitting the mechanism to models that were never trained for it while leaving each layer's existing projections and attention implementation intact. It can also operate on architectures whose approximation errors are distributed unevenly among Keys, Values, and recurrent states.
Why Standard Evaluation Is Insufficient
One of the paper's strongest contributions is its measurement methodology. Loss averaged over every token can hide errors in the approximate cache region. Even a cache that fails to retrieve one critical fact may look nearly unchanged when averaged across thousands of mostly irrelevant tokens.
RidgeFill combines three diagnostics:
- Needle retrieval inside the filled region to test whether approximate cache entries remain accessible.
- Loss on tokens that depend on distant context to expose long-range dependency errors.
- Splice tests that replace approximate Keys or Values with exact ones to identify which component carries the error.
On Llama-3.1-8B, Keys account for much of the observed error, with roughly half its magnitude. On Qwen3.5-9B, Values and recurrent states are more significant. This means RidgeFill will not behave identically across Transformers: attention design, normalization placement, state dimensionality, and upper-layer sensitivity all affect the quality of the fitted maps.
Measured Results
For Qwen3.8-27B using a llama.cpp fork, keeping the maps in VRAM improves time to first token by approximately 1.25–1.31x at 16K–120K tokens. The cost is a roughly 1.2–1.4-point RULER decline and a 1.4-point LongBench v2 decline. The reported 95% intervals include zero, so the evidence does not establish a definite regression, but it also does not rule out losses of several points.
With Qwen3.8-Flash-Next in vLLM, the method reports a 1.50–1.60x improvement at 9K–32K tokens. RULER Needle reaches 99.8 versus 100.0 for exact computation, while perplexity on novel tokens is 2.65% higher on average, with a reported interval of -0.02% to +4.93%. The speedup therefore does not appear to come from catastrophic test corruption, although the error bound remains broad.
Across three 8B–9B reference models, the fitted maps outperform identity fill on the filled-region RULER metric by 49.1 and 49.6 points on two dense models under the same split and tail settings. One Llama configuration nevertheless has worse loss, and the long-document loss difference relative to the released SwiftKV checkpoint is approximately ±0.03 nat.
Engineering Trade-offs
RidgeFill improves time to first token, not decoding. Every generated token still requires full-model computation. The benefit is consequently strongest for long prompts, latency-sensitive services, and workloads with relatively short outputs. It may offer little advantage for short prompts, generation-heavy applications, or deployments optimized primarily for aggregate throughput.
Practical deployment must account for:
- Extra VRAM for the fitted maps, which depends on model dimensions, target-layer count, and sequence length.
- Numerical stability. Production code should prefer damped least squares, Cholesky, QR, or SVD instead of an unstable explicit inverse.
- Calibration coverage. Maps fitted on a narrow language or task distribution may not generalize reliably.
- Portability. A map is generally tied to a particular model, quantization format, attention implementation, and backend.
A conservative deployment starts with a later split layer so that more critical interactions remain in the exact-compute region, then searches for the Pareto frontier between latency and task quality.
Deployment Guidance
RidgeFill is best treated as an optional fast path in an inference engine rather than an unconditional replacement for exact prefill. Teams should track TTFT, end-to-end latency, throughput, peak memory, RULER/Needle retrieval, LongBench, perplexity, and real task success rates. Exact fallback should remain available for high-risk queries.
Average token loss alone is not a sufficient acceptance criterion. Evaluation should include retrieval from the approximated region, long-range dependent QA, exact symbol recall in code and tables, multilingual prompts, and batched workloads with shared prefixes. For citation-heavy or auditable applications, limiting the approximated token range or enabling RidgeFill only for low-risk traffic is safer.
RidgeFill offers a pragmatic long-context optimization path: modest offline calibration produces a 25%–60% prefill-latency improvement, with a measurable but not yet fully eliminated cost in retrieval quality and perplexity.
— END —Source: Hugging Face Daily Papers (2026-11-06)
Tags: #Long-Context Inference #KV Cache #Prefill Optimization #Ridge Regression #Inference Acceleration
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments