ResCom: Activation Residual Compensation for Large Language Model Quantization
Summary:ResCom is a novel low-rank correction method for large language model quantization that compensates for layer-output errors using activation rounding residuals while maintaining low-bit weight and activation quantization. Experiments on the Llama-3-8B model with 4-bit weights and activations show that ResCom significantly improves performance on benchmarks like WikiText-2 perplexity, MMLU, and GSM8K. The method also demonstrates scalability across models ranging from 1B to 70B parameters and eff
Technical Mechanism Analysis
The core idea of ResCom is to use activation rounding residuals to compensate for the errors introduced during the quantization process. Quantization reduces memory and computational demands by converting weights and activations to low-bit representations, but this conversion leads to precision loss, especially when activations are rounded, causing the model outputs to deviate from the original results. ResCom addresses this through the following steps:
- Activation Residual Calculation: At each quantized layer, compute the rounding residual of the activation values, which is the difference between the original and quantized activation values.
- Low-Rank Correction: Apply a low-rank matrix to correct the residuals, reducing computational overhead while maintaining the effectiveness of the correction.
- Output-Aware Calibration: Analyze the impact of the correction on the model output, select the residual directions that contribute most to the reconstructed output, and compute the correction values in closed form without gradient training or updating the calibrated weights.
Engineering Trade-offs and Performance
ResCom significantly improves model accuracy while maintaining the advantages of low-bit quantization. Key experimental results include:
- On the Llama-3-8B model with 4-bit weights and activations, ResCom reduces the WikiText-2 perplexity from 7.708 to 7.147, improves MMLU by 2.89 percentage points, and raises the five-shot GSM8K accuracy from 23.96% to 36.16%.
- ResCom demonstrates scalability across models ranging from 1B to 70B parameters.
- The FP32 implementation adds 2.25% to the 4K prefill time, while the compact-storage implementation retains a 27.7% peak-memory reduction relative to FP16.
These results indicate that ResCom is highly effective in improving the accuracy of quantized models while having a minor impact on computational and memory overhead.
Developer Deployment Recommendations
For developers looking to deploy quantized large language models, ResCom offers an effective solution. Here are some recommendations:
- Model Selection: ResCom is applicable to models of various sizes. Developers can choose the appropriate model for quantization based on their specific needs.
- Calibration Process: Ensure the model is thoroughly calibrated before applying ResCom to achieve the best compensation effect.
- Resource Management: Although ResCom has a minor impact on memory and computational overhead, it is important to carefully assess its impact in resource-constrained environments.
Conclusion
ResCom is an innovative quantization correction method that significantly improves model accuracy while maintaining low-bit quantization. Its excellent performance in multiple benchmarks and good scalability make it a powerful tool in the field of large language model quantization.
— END —Source: ArXiv Machine Learning (cs.LG) (2026-10-28)
Tags: #Large Language Model #Quantization #Low-Rank Correction #ResCom #Deep Learning
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments