TTB Bakes Cross-View Attention into MLPs for Up to 23x Faster Novel-View Rendering
Summary:Test-Time Baking (TTB) accelerates decoder-only Large View Synthesis Models by converting cross-view attention into a lightweight MLP learned from Test-Time Training fast weights. It disentangles camera-only context tokens used as inputs from ground-truth tokens carrying image content used as mapping targets, reducing the modality mismatch between context prefilling and novel-view rendering. The paper reports up to a 23x rendering speedup while maintaining quality comparable to state-of-the-art
Core Idea: Turning Cross-View Attention into a Reusable Mapping
Test-Time Baking (TTB) targets the inference bottleneck in decoder-only Large View Synthesis Models (LVSMs). These models propagate global scene information across views through a KV cache, but novel-view rendering still requires costly cross-view attention. Rather than reconstructing explicit 3D geometry, TTB repurposes the fast-weight MLP in Test-Time Training (TTT) layers as a cross-view attention baker. Once the scene context is established, a lightweight MLP replaces repeated attention during rendering.
For a baseline model, the attention path between query tokens and cached context tokens has a cost roughly proportional to:
[
O\left((N_q+N_k)d^2 + N_qN_kd\right),
]
where (N_q) is the number of query tokens, (N_k) is the context-cache length, and (d) is the hidden dimension. TTB moves much of this computation into a scene-specific MLP, reducing each novel-view token to projections and nonlinear transforms rather than repeated quadratic attention interactions.
How TTB Bakes Cross-View Attention
The first key step is to reuse the input-output relation of the TTT fast-weight MLP so that it explicitly learns the cross-view attention mapping. TTB does not simply remove attention or substitute an unconstrained MLP for the entire attention module. The baked mapping therefore inherits the cross-view correspondence already established by the original LVSM.
The second step addresses modality misalignment between context prefilling and novel-view rendering. The paper separates context tokens into:
- camera-only context-rendering tokens, used as TTB inputs; and
- context ground-truth tokens containing image content, used as mapping targets.
This separation keeps the baker from having to generalize across a camera-conditioned input distribution and an image-generation target distribution. In implementation terms, it decouples the conditioning query from the supervision signal, which may improve mapping stability across scenes and camera trajectories.
Engineering Trade-offs
TTB trades online rendering work for scene-level preparation. Its benefit therefore depends on scene reuse. Interactive playback, multi-frame generation, and applications that request many novel views from the same context can amortize the baking cost. A one-off query may not gain much once that preparation is included.
The original KV cache can still support context encoding, but the baked MLP removes much of the renderer's dependence on large cross-view attention operations. The summary does not disclose baking latency, extra memory, MLP parameter count, batch efficiency, or cache-switching overhead. These details determine whether TTB is useful for real-time systems rather than only offline benchmark evaluation.
Performance and Evaluation Caveats
The paper reports up to a 23x rendering speedup while maintaining quality comparable to state-of-the-art methods across several benchmarks. The central claim is therefore an inference-efficiency improvement rather than quality obtained by reducing resolution, sampling frequency, or output detail.
However, “up to 23x” should not be interpreted as a universal multiplier. The summary does not specify view counts, resolutions, hardware, batch sizes, precision modes, or end-to-end timing boundaries. It also does not establish comparisons against competing KV-cache compression methods. Independent evaluation should examine geometric consistency, fine textures, specular surfaces, and camera extrapolation rather than relying solely on aggregate image metrics.
Deployment Guidance
- Measure scene reuse first. Multi-frame rendering and interactive navigation are the strongest use cases; single-request serving should include baking overhead in its latency budget.
- Profile the full pipeline. Report context prefill, baking, novel-view decoding, and output transfer separately to avoid inflated kernel-level speedups.
- Test quality-sensitive scenes. Evaluate texture-dense regions, thin structures, reflections, and cameras that move beyond the input distribution.
- Track memory and throughput. Include MLP caches, KV caches, intermediate activations, batch size, and both throughput and per-request latency.
- Account for scene changes. Frequent cache invalidation or scene switching can erase the amortization advantage in production workloads.
TTB does not introduce a new 3D representation. Its contribution is a different inference graph for existing LVSMs: repeated cross-view interactions are compiled into a lightweight, reusable mapping, shifting computation from dynamic per-view attention to scene-level preparation. If the full results reproduce across scenes and hardware, this could provide a practical route from research-scale view synthesis toward real-time novel-view applications.
— END —Source: Hugging Face Daily Papers (2026-11-12)
Tags: #LVSM #Test-Time Training #Cross-View Attention #Novel-View Synthesis #Inference Acceleration
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments