LlamaIndex Optimizes OCRia-1 for Nearly 10× Token Generation on One B200
Summary:LlamaIndex reports that its inference-stack work on the OCRia-1 document parsing model raised token generation from roughly 285 to 2,800 tokens per second on a single NVIDIA B200—nearly a 10× improvement. The stack combines layer-sensitive BF16, FP8, and NVFP4 quantization with activation-aware calibration, DFlash2 block-diffusion speculative decoding, and state-management techniques designed for OCRia-1's Gated DeltaNet layers. The results show how aggressively memory traffic can dominate long
Core Finding
LlamaIndex has disclosed the results of an inference-stack overhaul for the OCRia-1 document parsing model. According to the company, token generation on a single NVIDIA B200 rose from roughly 285 to 2,800 tokens per second—nearly a 10× improvement. The gain does not come from one new kernel. It combines mixed-precision quantization, DFlash2 speculative decoding, Gated DeltaNet state optimization, and a series of CPU and GPU overhead reductions.
The most useful lesson is the engineering sequence: identify decoding as a bandwidth-bound problem, then attack it from three directions—move fewer bytes per decoding step, generate more tokens per target-model forward pass, and eliminate state-rollback and runtime overhead.
Why Decoding Dominates Latency
OCRia-1 takes images and a text prompt and produces a structured textual representation of each document. Dense pages create long output sequences, and autoregressive decoding must generate those tokens one step at a time.
Inference has two broadly different phases:
- Prefill processes the input in parallel and builds the model's internal state. It is generally more compute-bound.
- Decode generates a small number of tokens per iteration while repeatedly reading model weights and recurrent state. It is usually constrained by HBM bandwidth.
In the bandwidth-bound regime, performance can be approximated by asking how many bytes must be moved to produce one token. Quantization directly reduces the weight bytes in that denominator, while speculative decoding allows one target-model forward pass to cover multiple output tokens. Larger batches improve arithmetic intensity but also increase the working set, recurrent-state pressure, and scheduling complexity. GPU peak compute alone therefore predicts document-parsing throughput poorly.
Mixed-Precision Quantization: Less Data Per Decode Step
OCRia-1 does not apply one precision to every parameter. The stack mixes BF16, FP8, and NVFP4, retaining higher precision for sensitive layers and compressing more aggressively elsewhere.
Moving from BF16 to FP8 reduces the main weight representation from two bytes per parameter to one. NVFP4 can compress further, although real-world savings still depend on scale factors, metadata, operator fusion, and native hardware support. The team also uses activation-aware quantization: calibration reflects how weights interact with activations from real, in-distribution inputs rather than rounding weights in isolation.
The vision encoder remains unquantized to reduce the risk of losing fine visual information. The team then evaluates combinations of layer precision against both speed and accuracy on held-out examples, reporting that the selected quantization is effectively lossless for the target tasks.
The engineering trade-offs are substantial:
- Calibration data must represent production document types or scaling factors may fail on long-tail layouts.
- Losslessness on the selected task cannot be generalized to every OCR, table, formula, reading-order, or layout metric.
- Mixed precision complicates kernel selection, operator coverage, and version validation. If the runtime cannot fuse the relevant formats, theoretical compression may not become throughput.
DFlash2 Speculative Decoding: More Tokens per Forward Pass
Even with a KV Cache that avoids recomputing earlier context, an autoregressive model generally reads its weights again for every decoding step. DFlash2 serves as a lightweight block-diffusion drafter: it proposes multiple future tokens in one forward pass, conditioned on hidden-state features extracted from the target model. OCRia-1 then verifies those candidates in parallel during one target-model forward pass.
When acceptance and rejection follow the correct target-distribution validation procedure, speculative decoding can preserve the target model's output distribution in theory. A flawed drafter should cost computation, not generation quality. DFlash1 accepted an average of nine tokens; DFlash2 added local convolutions and a candidate selector, raising the acceptance length above 12 and using DFlash1 weights to initialize the new model.
At an intermediate stage, LlamaIndex reports more than 1,400 tokens per second per user at batches of 8 to 12, versus about 200 tokens per second for the base model. The drafter nevertheless adds model memory, verification compute, and scheduling complexity. Its benefit depends heavily on acceptance length, which can vary with repetitive prose, tabular structures, and output length.
ReplaySSM and Gated DeltaNet State Rollback
OCRia-1 uses Gated Delta Networks. Unlike full-attention layers that retain historical keys and values, GDN layers compress history into a fixed-size recurrent state. Long outputs therefore avoid a continuously growing KV Cache. The tradeoff is that every token still reads that state from HBM, updates it, and writes it back. When speculative candidates are rejected, the system must also roll back or recompute state, increasing memory traffic further.
The article identifies ReplaySSM from Dao Lab as a way to lower verification cost. Its direction is to use state checkpoints and the necessary replay work to reduce repeated state materialization after rejected candidates. The public account does not disclose checkpoint granularity, replay scope, supported operator combinations, or an isolated speedup, so the full 10× result should not be attributed to this component alone.
ReplaySSM is also less portable than generic quantization. It depends on GDN state layouts, GPU kernels, and runtime implementation. The broader stack additionally targets unnecessary memory transfers, CPU-side preparation, and smaller scheduling inefficiencies, but no independent ablation is provided for those changes.
How to Interpret the Nearly 10× Result
The 2,800-token-per-second figure is a first-party result, not an independent benchmark. The published figures do not fully specify the final concurrency level, image resolutions and page counts, output lengths, warm-up procedure, draft-block size, acceptance-rate distribution, or structured-output post-processing cost. It should not be equated directly with time-to-first-token or total document completion time: prefill, vision encoding, and structural validation may still account for substantial latency.
Two measurement scopes also need to be kept separate. The approximately 200-to-1,400-token-per-second figures are per-user results from an intermediate batch-8-to-12 stage, while 285 to 2,800 tokens per second is the final single-GPU generation result.
A rigorous evaluation should report at least:
- TTFT, time per output token, and p50/p95 end-to-end latency under fixed models, prompts, image corpora, and output lengths;
- aggregate single-GPU throughput, per-user throughput, concurrency scaling, and peak memory;
- character error rate, reading-order accuracy, table structure, formula preservation, and layout fidelity;
- task-level quality regressions for each quantization mix and acceptance rates by document type;
- cost per thousand pages, energy use, and retry overhead.
Deployment Guidance
- Profile by stage. Measure vision encoding, prefill, decode, structured post-processing, and CPU scheduling separately rather than attributing all latency to the model forward pass.
- Choose precision per layer. Start from a BF16 baseline, then test BF16, FP8, and NVFP4 combinations with in-distribution calibration. Run separate quality regressions for the vision encoder and numerically sensitive layers.
- Add speculative decoding after profiling. Enable a drafter only when its average accepted length covers the extra draft and verification cost. Monitor acceptance by document type rather than relying on one global average.
- Audit rollback for stateful models. With GDN or other SSMs, measure state copying, replay, and memory transfer after candidate rejection; full-attention KV Cache optimization patterns do not transfer automatically.
- Lock the runtime stack. Quantization formats, CUDA versions, fused kernels, Tensor Core instructions, and drafter implementations can all change the result. Pin versions and run end-to-end regressions.
Bottom Line
The OCRia-1 work shows that long structured-document generation is often constrained by moving weights and recurrent state rather than by idle GPU compute. Layer-sensitive mixed precision reduces bytes per iteration, DFlash2 raises the useful output of each target forward pass, and ReplaySSM addresses rollback costs specific to recurrent architectures.
The broadly reusable elements are quantization, calibration, and exact speculative verification. ReplaySSM—and the nearly 10× headline—is much more tightly coupled to OCRia-1's architecture and custom kernels. Without complete configurations and independent reproduction, the disclosure is best treated as a valuable engineering reference rather than a general performance guarantee.
— END —Source: LlamaIndex Blog (2026-11-21)
Tags: #LlamaIndex #OCRia-1 #Speculative Decoding #Quantized Inference #Gated DeltaNet
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments