Hugging Face Releases Mellum2.1-12B-A2.5B-Thinking-MTP: Enhanced Inference Efficiency with Multi-token Prediction Head

Summary:Hugging Face has released Mellum2.1-12B-A2.5B-Thinking-MTP, a language model enhanced with a Multi-token Prediction (MTP) head. The MTP head, an additional Transformer block, predicts the next token, enabling a 1.4 to 1.8x speedup in inference while maintaining output quality. By verifying drafted tokens, the model accepts approximately 90% of the drafts, significantly accelerating decoding without compromising the final output.

---

Technical Mechanism Analysis

The Mellum2.1-12B-A2.5B-Thinking-MTP introduces a Multi-token Prediction (MTP) head to significantly enhance inference efficiency. The MTP head, an extra Transformer block trained alongside the model, predicts the token following the one just sampled. Inference engines use it as a speculative draft: the model verifies each drafted token, ensuring the output remains unchanged while decoding runs 1.4 to 1.8 times faster, with approximately 90% of the drafted tokens accepted.

Engineering Trade-offs and Performance

• Balance of Speed and Quality: The introduction of the MTP head accelerates decoding without significantly impacting the model's output quality. Experiments show that 90% of the drafts are accepted, indicating high draft quality.
• Implementation Details: The MTP head is stored in the model-mtp.safetensors file and declared by num_nextn_predict_layers: 1 in config.json.
• Compatibility: The model is compatible with vLLM 0.33 or newer, transformers 5.20 or newer, and llama.cpp. Specific configuration methods are detailed above.

Developer Implementation and Deployment Recommendations

For developers, integrating the MTP head can significantly improve the model's inference efficiency, especially in applications requiring real-time responses. Here are some deployment recommendations:

• Using vLLM: Add the parameter --speculative-config '{ "method": "mtp", "num_speculative_tokens": 1 }' when starting the model.
• Using transformers: Pass the parameter use_mtp=True to the generate function.
• Using llama.cpp: Use the GGUF repository and add the parameters --spec-type draft-mtp --spec-draft-n-max 1.

Conclusion

The release of Mellum2.1-12B-A2.5B-Thinking-MTP marks a significant advancement in optimizing the inference efficiency of language models. By introducing the MTP head, developers can greatly enhance the model's inference speed without sacrificing output quality. This is particularly valuable for applications requiring high performance and real-time responses, such as dialogue systems and real-time translation.

---

Source: Reddit r/LocalLLaMA (2026-11-02)

— END —

Tags: #Hugging Face Follow #Mellum2.1 Follow #MTP Follow #Inference Acceleration Follow #Language Model Follow