Hugging Face Releases PreSTO: A New Method for Accelerating LLM Inference

Summary:Hugging Face has introduced PreSTO, a novel method for accelerating the inference process of large language models (LLMs) through predictive subtree prefetching. PreSTO enhances existing samplers like PowerMH, EntropyCut, and MultiTryMH by generating batches of future accept/reject decision subtrees without altering the Metropolis-Hastings (MH) transition kernel. Experimental results demonstrate that PreSTO achieves median speedups of 1.71x to 2.33x across seven datasets and five base LLMs, redu

---

Technical Mechanism Analysis

PreSTO accelerates acceleration through the following core mechanisms:

1. Predictive Subtree Prefetching: PreSTO views future Metropolis-Hastings (MH) decisions as a tree of accept/reject decisions. In each model call, it generates a batch of proposals for a prefetched subtree and then follows the realized path with the original MH acceptance rule. This approach significantly reduces the overhead of sequential computations, thereby accelerating inference.

2. Sparse Subtree Optimization: PreSTO observes that MH only accesses a sparse, nearly unary part of each prefetched subtree. Therefore, it designs a predictive traversal rule that selects subtrees to maximize the number of completed transitions, further optimizing inference efficiency.

3. Invariance Guarantee: PreSTO preserves the invariance of the MH transition kernel while accelerating inference, meaning its sampling process is theoretically equivalent to the original method, ensuring sampling quality.

Engineering Trade-offs and Performance

• Acceleration Effect: Across seven datasets and five base models, PreSTO achieves median speedups of 1.71x, 1.91x, and 2.33x for PowerMH, EntropyCut, and MultiTryMH, respectively.

• Model Call Reduction: For PowerMH and EntropyCut, PreSTO reduces the number of model calls by an average of 3.9x, with each call only taking 2.1x longer.

• Real-world Application: On the LCB V6 dataset with the Qwen3.5-9B model, PreSTO's predictive traversal rule is 1.25x to 1.37x faster than the default breadth-first traversal method.

Developer Implementation and Deployment Recommendations

PreSTO provides developers with a way to improve LLM inference efficiency without significantly increasing computational costs. Here are some deployment recommendations:

1. Hardware Adaptation: The resource requirements of PreSTO are similar to existing methods, but it is recommended to run on multi-core processors or GPUs to take full advantage of its parallelization benefits.

2. Model Integration: PreSTO can be seamlessly integrated into existing LLM inference pipelines without the need for major modifications to the model architecture.

3. Performance Tuning: Developers can adjust the depth and width of the prefetched subtrees according to specific application scenarios to find the best balance between acceleration effect and computational overhead.

Conclusion

PreSTO, through its innovative predictive subtree prefetching technology, significantly improves the inference efficiency of LLMs while maintaining sampling quality. Its excellent performance in multiple benchmarks demonstrates its great potential in practical applications, providing new technical support for the further popularization and application of large language models.

---

Source: Hugging Face Daily Papers (2026-10-09)

— END —

Tags: #Hugging Face Follow #LLMs & Foundation Models Follow #Inference Acceleration Follow #Metropolis-Hastings Follow #PreSTO Follow