CoDR Open-Sourced: Enhancing Inference Accuracy and Efficiency for Diffusion Language Models
By Mr.Xu
Published:
Summary:CoDR (Confidence Drift Remasking) is a training-free, sampler-agnostic refinement method designed to address inference errors in Masked Diffusion Language Models (MDLMs) caused by confidence drift. It estimates drift for all committed positions in only k forward passes via k-partition probing and remasks and regenerates only the tokens the model no longer endorses. Experiments demonstrate that CoDR improves average accuracy across various reasoning and coding tasks while maintaining low computat
Background
Masked Diffusion Language Models (MDLMs) decode by repeatedly committing tokens to masked positions, but these commitments are usually irreversible. Due to the model's confidence in a committed token dropping from its sparse commit-time context to the denser context available later, existing samplers mainly decide when to commit a token but rarely check whether an already committed token should still be kept, allowing early mistakes to propagate.
Core Innovation of CoDR
CoDR (Confidence Drift Remasking) addresses this issue through the following mechanisms:
- Confidence Drift Detection: CoDR estimates drift for all committed positions in only k forward passes via k-partition probing.
- Selective Regeneration: It remasks and regenerates only the tokens the model no longer endorses, avoiding unnecessary computational overhead.
- Training-Free and Sampler-Agnostic: CoDR requires no additional training and can be seamlessly integrated with various samplers.
Experimental Results
Experiments across two backbones, four reasoning and coding tasks, and three base samplers demonstrate that CoDR improves average accuracy in all evaluated configurations and enhances most individual task settings while maintaining low computational overhead.
Technical Highlights
- Efficiency: CoDR significantly reduces the number of forward passes through k-partition probing, making it more efficient than existing remasking methods.
- Accuracy Improvement: By targeted confidence-drift remasking, CoDR effectively enhances inference accuracy.
- Flexibility: Its training-free and sampler-agnostic nature makes it applicable to a wide range of scenarios.
Industry Impact and Developer Recommendations
The release of CoDR provides a new approach to optimizing inference in diffusion language models, particularly for applications that require high inference accuracy and efficiency. Developers can integrate CoDR into existing MDLMs to improve model performance. Additionally, the open-source nature of CoDR makes it an excellent foundation for further research and development by engineers and researchers.
Future Outlook
As diffusion models become more prevalent in AI, CoDR is expected to play a significant role in areas such as text generation and image generation. Future research can explore CoDR's performance on larger models and more complex tasks and further optimize its computational efficiency.
— END —Source: ArXiv NLP/LLM (cs.CL) (2026-10-08)
Tags: #CoDR #Diffusion Models #Inference Optimization #Open Source AI #Masked Language Models
Community Comments