ZICQ
中 Log in / Sign up
Newsroom Research & Papers #quantization evaluation #decision flips #serving engines #vLLM #Gated DeltaNet

Quantization Noise Floor: Cloudflare Separates 4-Bit Decision Flips from Runtime Drift

Avatar of Mr.Xu

By Mr.Xu Compiled & Reviewed by Editorial

Published:

中文阅读 (Chinese) English Version
Co-reading Select any sentence in the article to highlight, annotate, or copy a quote card. Join discussion ↓

Summary:Cloudflare’s quantization study separates weight-format effects from the serving stack itself. On an NVIDIA GB10, the authors replay the same 2,400 public decisions through Clef-flash (9B) and Clef (27B), measuring output flips caused by engine swaps, restarts, batch-size changes, and an automatically selected Gated DeltaNet kernel. Non-quantization perturbations move 0–0.92% of decisions; within one engine, Q4_K_M and NVFP4 move 3.50–5.79%, with the reported 4-bit intervals separating them from


The question: a quantization comparison changes more than weights

A direct A/B test between a BF16 model and its quantized build does not isolate quantization. The serving engine, service instance, request batching, operator implementation, and even library import order can alter the final output. Unless those variables are controlled, the measured difference mixes model error with runtime drift.

Cloudflare’s study does not introduce a new quantization algorithm. Instead, it establishes a measurable noise floor for the deployment stack and then attributes only the residual change to quantization. That methodological correction is more broadly useful than a ranking of any single number format.

Experimental design: establish the noise floor first

The experiments use Cloudflare’s Clef-flash (9B) and Clef (27B) decision models on one NVIDIA GB10 system. For the same 2,400 public decisions, the authors introduce several controlled interventions:

  • Restarting the same service and replaying the records;
  • Switching between llama.cpp and vLLM under BF16;
  • Changing request batch size;
  • Changing library import relationships to observe whether transformers selects a different Gated DeltaNet kernel without an explicit request;
  • Comparing BF16, Q8_0, FP8, Q4_K_M, and NVFP4 only within a fixed engine.

Quantization effects are then expressed as the residual above these non-quantization sources. Instead of asking whether two endpoint scores differ, the design asks whether the difference is attributable to the weight representation or to the execution path.

Why decision flips expose what aggregate accuracy hides

Quantization can be represented as Wq = Q(W), with rounding error ΔW = Q(W) - W. That error propagates through layers and perturbs the final logits. If the correct class and a competing class have a small score margin, even a minor perturbation can change the argmax result.

For a correct class y and competitor j, let the logit margin be z_y - z_j = m. When |m| is smaller than the perturbation introduced by quantization or numerical execution, the decision can flip from y to j. This explains why aggregate accuracy may barely move while individual outputs change substantially: improvements and regressions cancel in the mean.

A decision flip is not automatically an incorrect answer, but it exposes sensitivity to the runtime environment. That matters most for safety-critical cases, long-tail inputs, and samples whose original logits were already close to a decision boundary.

Runtime drift also has plausible numerical causes. Floating-point addition is not strictly associative, and different kernels can use different reduction orders, blocking strategies, or accumulation precision. Batch-size changes may also trigger shape-specialized kernels. The paper does not trace every effect down to instruction level, but it shows that changing import relationships alone can switch the Gated DeltaNet kernel. Identical weights therefore do not imply an identical execution trajectory.

Main results: 4-bit effects clearly exceed non-quantization noise

The decision-flip rates measured within one engine are:

| Quantization format | Clef-flash (9B) | Clef (27B) | |---|---:|---:| | Q8_0 | 0.88% | 1.04% | | FP8 | 2.12% | 1.62% | | Q4_K_M | 5.79% | 3.50% | | NVFP4 | 5.79% | 5.67% |

Across the non-quantization interventions, total movement ranges from 0 to 0.92%. More specifically:

  • llama.cpp restarts remain bit-identical on every record for both models;
  • After vLLM restarts at BF16, FP8, and NVFP4, Clef-flash flips 0, 5, and 20 decisions, while Clef flips 4, 0, and 22;
  • In an exploratory third-instance comparison for Clef, FP8 remains at zero flips, while NVFP4 ranges from 4 to 22 flips depending on the instance pair;
  • At BF16, an engine change flips 6–16 decisions, a batch-size change flips 8–15, and the import-triggered Gated DeltaNet kernel change flips 10–14.

The paper reports that, for both models, the intervals for every 4-bit result exclude all non-quantization sources. Within this hardware, engine, model, and sample configuration, the Q4_K_M and NVFP4 perturbations therefore cannot be explained by restart or runtime noise alone. This is not a universal guarantee for other hardware and models.

Preregistration and the value of disconfirmation

Five predictions were preregistered. Two were not supported, and the same two failed when the tests were rerun on the 27B model: FP8 does not fall within restart noise, and changing engines moves fewer decisions than 8-bit quantization does.

These disconfirmations do not weaken the study. Preregistration prevents the authors from selecting only favorable post hoc comparisons, while the release of code, quantization recipes, per-record outputs, and the preregistration lets external teams reproduce the failed predictions rather than only the final accuracy table.

Accuracy can conceal record-level regressions

Across the full set of 31,275 public questions, Clef’s Q4_K_M build changes net accuracy by only six questions, or -0.02 percentage points, with a reported interval of [-0.20, +0.16]. Yet an exploratory record-level analysis finds that 436 decisions become worse.

The aggregate score therefore appears almost lossless even though individual behavior has become less stable. NVFP4 is easier to detect from the mean: it reduces accuracy by 0.95 points for Clef-flash and 0.65 points for Clef. The resulting deployment rule is straightforward: quantization reports need paired flip rates, counts of improved and worsened samples, and risk-stratified regressions in addition to overall accuracy.

Engineering trade-offs: smaller weights are not a free substitution

At the raw weight-storage level, FP8 and Q8_0 halve the bit width relative to BF16, while 4-bit formats reduce it to one quarter. End-to-end memory does not fall by exactly the same proportion because KV caches, activations, quantization scales, and dequantization buffers remain in play. Throughput depends on native hardware kernels, memory bandwidth, batch size, and operator scheduling.

The benefit of 4-bit inference is higher weight density and potentially lower bandwidth pressure. The cost is more decision flips and a narrower stability margin. FP8 changes more decisions than Q8_0 but may deliver better throughput through native acceleration, so format selection cannot be reduced to bit width alone. The supplied material does not report latency, throughput, peak memory, or energy figures, so it cannot establish which format offers the best production return.

Deployment recommendations

  1. Lock the complete execution environment. Pin not only the model revision but also the container digest, CUDA and driver versions, inference-engine version, quantization kernels, batch size, and import path.
  2. Measure a zero-difference baseline first. Restart services, switch engines, and vary batch size without changing weights to establish the deployment stack’s own decision noise floor.
  3. Use paired evaluations. Save outputs for every input and report flip rates, net accuracy, improved records, and worsened records instead of comparing only two aggregate scores.
  4. Change one factor at a time. Evaluate Q8_0, FP8, and 4-bit within a fixed engine before changing engines simultaneously; otherwise attribution becomes impossible.
  5. Audit boundary cases. Stratify results by low decision margin, task risk, long-context behavior, and workflows that depend on deterministic formatting.
  6. Use shadow traffic and gradual rollout. Apply record-level diffs to block high-risk outputs and retain BF16 or a more conservative format as a rollback path.

Bottom line

The central contribution is not a declaration that every 4-bit format is unusable. It is a demonstration that quantization evaluation must control runtime noise before attributing behavioral changes to the model format. Cloudflare separates the two effects through paired controls, preregistration, and public record-level outputs. Aggregate accuracy may remain nearly unchanged while decision stability has already deteriorated, making the latter a prerequisite for safe production deployment.


Source: Hugging Face Daily Papers (2026-11-11)

— END —

Tags: #quantization evaluation #decision flips #serving engines #vLLM #Gated DeltaNet

Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.

What's your take on this story?
🚀 Explore & Submit Works

Community Comments

Loading live comments and annotations…