ZICQ
中 Log in / Sign up
Newsroom Research & Papers #Zero-Gradient Compilation #Circuit Grafting #LLM Arithmetic #Mellin-GLU #Model Weight Surgery

Compiling Exact Arithmetic Circuits into Pretrained LLMs Without Gradient Updates

Avatar of Mr.Xu

By Mr.Xu Compiled & Reviewed by Editorial

Published:

中文阅读 (Chinese) English Version
Co-reading Select any sentence in the article to highlight, annotate, or copy a quote card. Join discussion ↓

Summary:The paper introduces zero-gradient circuit grafting, a method that compiles deterministic computations directly into the weights of pretrained language models without training examples, gradient updates, or external tool calls. Its construction combines Mellin-GLU quadrature with normalization-aware cross-token cancellation, then extends the model width and adapts the tokenizer and output embeddings for deterministic arithmetic. On 1,200 six-digit arithmetic examples, the reported ALU-prefix ac


Core idea: compiling specialized circuits into model weights

Instead of asking a language model to learn arithmetic from additional examples, this work represents a deterministic function as a circuit and uses closed-form parameter solving to map that circuit onto the hidden states and output weights of a pretrained model. The targeted functions include four-to-eight-digit signed addition, subtraction, and multiplication, as well as three-literal Boolean functions, five-bit Majority, and Parity. The process uses no training examples or backpropagation, and inference requires no external interpreter such as Python or a calculator.

The claim that an LLM contains a CPU is metaphorical. The construction does not expose a general instruction set, arbitrary programs, control flow, or dynamic memory access. What is embedded is a family of specialized threshold circuits, making the result closer to an arithmetic coprocessor compiled into a neural network than to a programmable processor.

How the three mechanisms work together

1. Mellin-GLU absorbs RMSNorm scaling

RMSNorm applies a token-dependent multiplicative scale of the form x / sqrt(mean(x_i²)+ε). A compiler cannot assume that the exact normalization factor at runtime is known from the original checkpoint, so a direct closed-form calibration of the target circuit can fail.

The paper uses the Mellin-transform property that converts multiplicative scaling into translation in the logarithmic domain. A two-layer Mellin-GLU quadrature construction over bias-free SwiGLU or GeGLU layers then approximates the desired transfer relation on a log-spaced gate grid spanning more than 250 times in dynamic range. IEEE 754 error bounds are used to constrain floating-point deviations. The intended result is a fixed gate transformation that remains usable even when the effective RMSNorm scale is unknown at inference time.

2. Cross-token cancellation limits state drift

Compensating one layer is not enough when a multi-digit operation is distributed across tokens. The paper combines signed Chebyshev Indefinite-RoPE self-sink attention with memoryless SiLU-antisymmetric GatedDeltaNet shifts. The former maintains computation state through a self-sink attention structure, while the latter uses antisymmetric state updates to cancel a common scale factor.

The practical target is structural rather than data-driven. In long arithmetic chains, small errors can accumulate across positions; correcting only the current activation leaves earlier state untouched. Symmetric constructions are intended to remove shared scale components before they propagate into subsequent tokens.

3. Width extension and a canonical output path

The compiler expands hidden width from d to d+K, reserves the new channels for circuit state, and applies d/(d+K) norm compensation. Under the squared-norm interpretation, this keeps the expanded activation scale close to the original scale and avoids destabilizing RMSNorm through the enlarged denominator.

The output path uses canonical Base-1000 BPE, encoding decimal numbers in base-1,000 chunks rather than one token per decimal digit. This reduces decoding length and alignment ambiguity. Differential tied unembeddings with +254/-256 offsets separate output directions associated with different signs. The cost is structural: hidden width, embedding shape, and tokenizer vocabulary change, so the resulting checkpoint cannot necessarily be loaded using the original model configuration.

Constant depth does not mean negligible resources

The paper describes four-to-eight-digit addition and multiplication as constant-depth TC^0 circuits. That complexity statement constrains circuit depth, not total computation. As the number of bits grows, circuit width, parameter count, matrix-multiplication work, and state bandwidth can still increase substantially. On a GPU, each constant-depth circuit layer may become a broad matrix operation, while autoregressive decoding remains token by token.

“Zero-gradient” also does not mean “zero compilation cost.” Closed-form solving may still require large linear systems, discrete quadrature, conditioning analysis, or numerical search. The abstract does not report compilation time, peak compilation memory, solver failure rates, or scaling curves across digit counts. Likewise, “no runtime hooks” does not imply zero deployment overhead. Width expansion, sparse sidecar access, Base-1000 tokenization, and precision policies may require changes to inference configurations or kernels.

Reported empirical results

| Evaluation | Result reported in the abstract | |---|---| | Three-literal Boolean functions | 100% on N=1,728, with an additional audit on a 42-layer Gemma 4 E4B model | | Five-bit Majority and Parity | 100% on both tasks | | Granite 4.2 3B, six-digit arithmetic | 99.17% BF16 ALU-prefix accuracy and 100% with an FP32 sidecar; 98%—100% mid-sentence resumption | | Qwen 3.5 4B, six-digit arithmetic | 100% ALU-prefix accuracy with both BF16 and an FP32 sidecar; 98%—100% mid-sentence resumption | | Active nonzero parameters | Granite: 1.07—1.33 million, at most 0.036%; Qwen: 2.25—2.39 million, at most 0.057% | | LoRA comparison | With 12.8K examples, the two models score 42.2% and 61.7%, with 0% inline continuation | | Long-context regression | All 14/14 pre-registered Tango splits above 2K tokens pass, covering N=4,392; 0 triggers across 11,134 OOD cases | | A100 memory | Reported values are 7.89/10.65 GiB; the stated INT8/INT4 plus FP32-sidecar configurations use 4.60/8.14 GiB, although the abstract does not fully specify every pairing |

How to interpret the benchmarks

The results support two meaningful conclusions. First, small deterministic circuits can be written into existing models with high accuracy, and the construction can preserve mid-sentence natural-language behavior across two model families. Second, the comparison with a 12.8K-example LoRA baseline shows that deterministic circuit specialization and generic post-training solve fundamentally different problems.

These results do not establish general mathematical reasoning. The compiler knows the target circuit in advance and can allocate dedicated state channels and output paths, making the approach closer to hardware specialization than to an algorithm emerging from data. The six-digit evaluation contains only 1,200 examples, while confidence intervals, random seeds, prompt templates, and per-digit error distributions are not disclosed. It is also unclear whether the LoRA baseline received a comparable search or hyperparameter budget.

“Exact circuit” is not equivalent to “every deployed example is exact.” Granite's 99.17% BF16 result shows that floating-point rounding can still break end-to-end outputs. Pre-registered long-context splits and zero OOD triggers are encouraging, but cross-model replication, broader task coverage, and independent evaluation remain necessary.

Practical deployment guidance

  1. Verify architecture compatibility first. The compiler depends on bias-free SwiGLU/GeGLU blocks, RMSNorm, and specific positional-encoding and state-space structures. It should not be assumed to work unchanged on every Transformer, GQA, or hybrid-attention model.
  2. Treat the tokenizer and output layer as migration work. Base-1000 canonical BPE and tied unembedding changes affect vocabulary, embedding dimensions, and generation constraints. Multilingual deployments must define how numeric aliases, special tokens, and multiple vocabularies are handled.
  3. Run an offline numerical audit. Exhaustively test small widths, run randomized and boundary cases at the target width, and record RMSNorm drift, IEEE 754 error bounds, and the active range of every gate.
  4. Choose precision according to reliability requirements. An FP32 path is preferable when exact and auditable results are mandatory. INT8 or INT4 can reduce sidecar memory but may invalidate the paper's strict numerical guarantees; they should not be enabled based only on average accuracy.
  5. Measure compilation and serving costs separately. In addition to memory, record compilation time, sidecar access overhead, time to first token, sustained decoding throughput, and batch-size scaling. Verify that the expanded width does not break existing parallel inference kernels.
  6. Keep the application scope narrow. The method is most relevant to agents and domain models that require deterministic, auditable arithmetic. It should not yet be treated as a general calculator replacement or evidence that the model has acquired transferable mathematical ability.

Source: Hugging Face Daily Papers (2026-11-19)

— END —

Tags: #Zero-Gradient Compilation #Circuit Grafting #LLM Arithmetic #Mellin-GLU #Model Weight Surgery

Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.

What's your take on this story?
🚀 Explore & Submit Works

Community Comments

Loading live comments and annotations…