TraceQuery Framework Revolutionizes Diffusion Language Model Inference Efficiency: Trajectory-guided Sparse Decoding App

Summary:arXiv introduces TraceQuery, a lightweight sparse decoding framework for diffusion language models (dLLMs) that predicts a small set of trajectory-relevant positions for limited fresh computation. By focusing on full-denoising trajectories, TraceQuery reduces the number of token computations while maintaining high accuracy, outperforming existing cache-based methods in quality-computation trade-offs across various tasks and model variants.

---

1. Introduction

Diffusion Language Models (dLLMs) enable flexible text generation through iterative denoising, but this process incurs high inference costs due to the repeated computation of token representations at each denoising step. Existing cache-based methods reduce this cost by reusing intermediate results and recomputing only a subset of positions at each step, but these methods typically rely on fixed schedules or heuristic signals to select positions for recomputation.

2. Technical Mechanism Analysis

TraceQuery introduces a trajectory-guided querying approach that analyzes the full denoising trajectory to determine which unresolved positions should receive limited fresh computation. The core mechanisms include:

• Trajectory-guided Prediction: TraceQuery predicts future trajectory-relevant positions by analyzing the denoising trajectory and computes only these positions.
• Lightweight Sparse Decoding Framework: This framework predicts a small set of trajectory-relevant positions and lets the model backbone select which one to commit from their fresh predictions, enabling efficient decoding.
• Quality-Computing Trade-off Optimization: TraceQuery significantly reduces the number of token computations while maintaining high accuracy, offering the best quality-computing trade-off among current cache-based methods.

3. Engineering Trade-offs and Performance

• Advantages:
  • Significantly reduces inference computation.
  • Applicable to various tasks and model variants.
  • Maintains high accuracy.
• Disadvantages:
  • Requires additional analysis and prediction of the denoising trajectory, which may add some computational overhead.
  • In some extreme cases, trajectory prediction may not be accurate enough, affecting decoding quality.

Experimental results show that TraceQuery performs excellently in multiple benchmarks, especially in handling long-text generation tasks.

4. Developer Implementation and Deployment Recommendations

• Applicable Scenarios:
  • dLLM application scenarios that require efficient inference, such as real-time dialogue systems, text generation, etc.
  • Environments with limited resources.
• Deployment Recommendations:
  • In actual deployment, it is recommended to fine-tune TraceQuery according to specific task requirements and model characteristics to achieve the best performance.
  • Consider combining TraceQuery with existing cache methods to further optimize inference efficiency.

5. Conclusion

TraceQuery provides an innovative trajectory-guided sparse decoding method that significantly improves the inference efficiency of dLLMs while maintaining high accuracy by reducing the number of new token computations. This framework opens new possibilities for the widespread application of dLLMs, especially in resource-constrained environments.

---

Source: ArXiv NLP/LLM (cs.CL) (2026-10-27)

— END —

Tags: #Diffusion Models Follow #Inference Optimization Follow #Sparse Decoding Follow