arXiv Proposes Novel Lightweight Language Model: Frozen Vocabulary for Efficient Inference and Storage Optimization
Summary:arXiv has released research on a lightweight language model that replaces the traditional trainable vocabulary with a fixed-size binary code table (16 MB). This approach significantly reduces the number of trainable parameters and storage requirements while maintaining or surpassing the performance of traditional models on benchmarks like LAMBADA, HellaSwag, and PIQA. The model also demonstrates high inference speed on low-resource devices, offering a promising solution for applications with lim
Background and Motivation
In small language models, the vocabulary is often the largest part of the model. For example, in the Gemma 3 270M model, the embedding table accounts for 168M out of 268M parameters. Due to quantization methods typically retaining the embedding in 16-bit, the embedding table still dominates the model file size even after compression. This raises the question: is such a large embedding table truly necessary?
Methodology and Innovation
The research team proposed a novel approach that replaces the traditional trainable vocabulary with a fixed-size 512-bit binary code table (16 MB). This table has no trainable parameters and is derived from a pretrained embedding using principal-component projection and sign binarization. The team trained the same Gemma-shaped model from scratch with and without the frozen table on the same 2.05B tokens.
Results and Analysis
The results show that while the trainable vocabulary table provides a 0.20 nats improvement in validation loss, it comes at the cost of 2.6 times the trainable parameters. On benchmarks like LAMBADA, HellaSwag, and PIQA, the frozen-table model either matches or outperforms the trainable table model, with only ARC-Easy favoring the trainable table.
Furthermore, the team tested a ternary BitNet b1.58 model, resulting in a 46 MB model file. This model ties the float model on LAMBADA and PIQA, and on a laptop CPU, it generates 615 to 740 tokens per second with 62 MB of memory, which is five to six times faster than the 8-bit GGUF float model in llama.cpp at a seventh of the memory.
The frozen table model excels on rare words, outperforming the trainable table by 1.7 to 2.3 nats, and can accept 10,000 new words after training with only a 3% cost, while the trainable table collapses by 43%.
Conclusion and Impact
This study demonstrates the significant potential of frozen vocabulary in small language models, particularly in resource-constrained environments. By reducing the number of trainable parameters and model size, this approach not only lowers storage requirements but also enhances inference speed, offering a promising solution for the lightweight development of AI models.
— END —Source: ArXiv NLP/LLM (cs.CL) (2026-10-19)
Tags: #Lightweight Models #Frozen Vocabulary #Quantization Techniques #Resource-Constrained Applications #Inference Optimization
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments