ZICQ
中 Log in / Sign up
Newsroom Research & Papers #Canon Layers #Linear Attention #Pretraining Token Efficiency #In-Context Learning #Synthetic Evaluation

Canon Layers Save 30% of Pretraining Tokens at 8B Scale, but Benchmarks Miss Retrieval Failures

Avatar of Mr.Xu

By Mr.Xu Compiled & Reviewed by Editorial

Published:

中文阅读 (Chinese) English Version
Co-reading Select any sentence in the article to highlight, annotate, or copy a quote card. Join discussion ↓

Summary:Researchers report 64 from-scratch pretraining runs comparing Transformers, gated linear attention, Gated DeltaNet, and Mamba2 at 1.3B–8B parameters and 100B–2T training tokens. At trillion-token scale, adding local token-mixing Canon layers reportedly lets an 8B Transformer reach the same MMLU accuracy with about 30% fewer pretraining tokens. With Canon layers, gated linear attention matches Gated DeltaNet and outperforms Mamba2 at 8B parameters and 1T tokens. The more consequential finding is


Core result

The paper compares Transformers, Gated Linear Attention, Gated DeltaNet, and Mamba2 through 64 from-scratch pretraining runs spanning 1.3B–8B parameters and 100B–2T Nemotron-CC tokens. Its central architectural finding is that adding local token-mixing Canon layers lets an 8B Transformer reach the same MMLU accuracy with roughly 30% fewer pretraining tokens. Gated linear attention equipped with Canon layers reportedly matches Gated DeltaNet and outperforms Mamba2 at 8B parameters and 1T tokens.

The paper does not establish that any one architecture has decisively replaced the Transformer. Its more consequential result is that standard benchmarks can conceal important architectural differences. Linear models that score highly still fail to reliably retrieve information from a 500-token context, and neither Transformers nor linear models reliably beat random guessing on a simple two-hop problem.

Mechanism: what Canon changes

A standard Transformer directly connects every token to every other token through self-attention. For sequence length L, each layer computes an L×L attention matrix, with interaction complexity around O(L²d + Ld²), while inference also maintains a KV Cache that grows with context length. Global, content-dependent mixing is powerful, but the model must learn local syntax, entity adjacency, and short-range structure from data rather than receiving a suitable inductive bias directly.

GLA, Gated DeltaNet, and Mamba2 use linear or recurrent mechanisms that avoid materializing the full attention matrix. Historical information is compressed into a recurrent state whose size is generally independent of sequence length, making these architectures attractive for long-context training and streaming inference. The tradeoff is that a bounded state may discard the exact information required for copying or distant lookup. The paper's retrieval result illustrates this tension: some linear models match Transformers on MMLU yet cannot reliably retrieve information from only 500 tokens.

Canon layers are described as adding local token mixing, giving neighboring tokens a shorter communication path instead of forcing all local information through global attention or recurrent-state updates. The abstract does not disclose the window size, convolution kernel, parameter sharing, or parallel implementation, so it is not possible to determine whether Canon uses fixed-window attention, a convolution-like operator, or another mechanism. If implemented with a fixed-width local kernel, its sequence-length term can be close to linear and its overhead may be much lower than full attention, but actual training throughput, memory bandwidth use, and inference latency still require measurement.

The study also introduces GLA5 and GDN2, which improve Gated Linear Attention and Gated DeltaNet through low-rank output gates. In generic terms, an output gate can be written as g(z) = σ(Wz). Factoring W = AB, where the intermediate rank r is much smaller than d, changes the operation to A(Bz). Such factorization can reduce gate parameters and computation while adding regularization. It can also restrict the gate's ability to select different state-update channels when the rank is too low. The abstract provides neither the chosen ranks nor parameter savings and ablations.

Why scale changes the conclusion

At an academic scale of 1.3B parameters and 100B tokens, carefully tuned architectures produce differences that almost disappear. Small-model comparisons can therefore misattribute gains to architecture when a broader learning-rate search, longer warmup, or better data mixture could compensate for a missing inductive bias.

At 1T–2T tokens, architectural differences become measurable. This need not mean that a model's theoretical ceiling suddenly changes. It may instead indicate that the bottleneck has shifted from optimization to data efficiency: at smaller budgets, spare capacity can learn compensating features, whereas at trillion-token scale, a bias that reduces wasted data continues to compound.

| Experimental setting | Reported result | Engineering implication | |---|---|---| | 1.3B, 100B tokens | Architectural differences nearly vanish after careful tuning | Small-scale runs cannot directly validate trillion-token architecture choices | | 1.3B–8B, 1T–2T tokens | Architectural differences become measurable | Architecture and training scale interact | | 8B Transformer | Canon layers reach equal MMLU accuracy with about 30% fewer pretraining tokens | This is conditional data efficiency, not an equal reduction in end-to-end cost | | 8B, 1T tokens | GLA with Canon matches Gated DeltaNet and exceeds Mamba2 | Local mixing may matter more than a more elaborate state-update mechanism | | Synthetic tasks below 100M parameters | Small diagnostic settings predict the later architectural differences | Synthetic environments can screen architectures, but cannot replace scale validation |

The reported 30% token reduction is one of the paper's most valuable engineering results, but its meaning must remain precise. It says that under the specified 8B setting and MMLU target, less pretraining data is required. It does not establish that training time, GPU expense, or total FLOPs fall by 30%. Full attention grows quadratically with sequence length, whereas local mixing and recurrent states have different operator efficiencies. The actual saving also depends on sequence length, kernel implementation, parallel efficiency, memory footprint, and optimizer state. With roughly $100,000 reported for each 8B run, external replication and ablation are expensive.

What standard benchmarks hide

MMLU measures aggregate knowledge and reasoning through multiple-choice questions. It does not directly test whether a model can retrieve an exact value from its context or explicitly combine two facts. Broad pretraining data may expose the model to many shallow patterns that support strong MMLU performance without producing a composable in-context algorithm.

The paper reports three more targeted diagnostics:

  • Linear models that match or exceed Transformers on standard benchmarks cannot reliably retrieve information from a context of only 500 tokens.
  • On a two-hop problem expressed in a few short sentences, neither Transformers nor linear models reliably beat random guessing at this scale.
  • Small synthetic environments can identify architectural differences below 100M parameters that later appear at 8B scale.

These results challenge the use of static benchmarks as substitutes for capability diagnosis. The current material does not disclose retrieval success rates, the exact two-hop generator, prompt templates, scorer, random seeds, or confidence intervals, so the claim that the models cannot reliably perform the tasks still needs verification against the full paper and code. Synthetic evaluation is not automatically fairer: if architecture selection and test-set tuning both use the same generator, a new form of overfitting can emerge.

Engineering trade-offs and reproducibility limits

| Design choice | Potential benefit | Cost and risk | |---|---|---| | Add Canon layers to a Transformer | Local inductive bias may reduce large-scale pretraining-token demand | Adds local operators and memory traffic; inference-latency effects are undisclosed | | Use GLA with Canon | A simpler linear mixer can match more elaborate state models | State compression may impair exact copying, retrieval, and compositional reasoning | | Use Gated DeltaNet or Mamba2 | More expressive state updates may help selected tasks | Scan kernels, state memory, and parallel efficiency add complexity; the reported comparison shows no clear advantage | | Use low-rank output gates | Potentially fewer gate parameters and stronger regularization | A low-rank bottleneck may restrict channel selection; rank ablations are needed | | Introduce in-context reasoning examples early | Retrieval and multi-hop behavior receive direct supervision during pretraining | The data mixture may be memorized; evaluation generators must remain isolated |

The most important open questions concern experimental control rather than the direction of the abstract's claims. Did the 64 runs receive symmetric hyperparameter-search budgets? Did every architecture use the same data ordering, batch sizes, and optimizers? Are results stable across random seeds? Are absolute retrieval and two-hop scores reported beyond MMLU? Are the exact implementations and code for Canon, GLA5, and GDN2 available? The collected material contains only the arXiv v1 abstract and omits author affiliations, absolute metrics, variance, hardware configuration, and code status. The 30% result should therefore be treated as a high-value finding awaiting independent replication.

Deployment guidance

  1. Build a diagnostic small-scale baseline first. Generate retrieval, single-hop, and multi-hop tasks with separate random seeds, isolate generator templates from evaluation samples, and test whether architectural differences emerge reproducibly below 100M parameters before committing to an 8B run.
  2. Control the tuning budget. At 1.3B parameters and 100B tokens, give Transformers, linear attention, and Canon variants equivalent hyperparameter-search budgets. Report accuracy-versus-token curves as well as final scores so that tuning gains are not mistaken for architectural gains.
  3. Move context capability into pretraining data. If a product depends on long-document lookup, tool use, or multi-fact composition, include composable in-context examples from early pretraining. Ablate early pretraining, mid-training, and post-training separately to avoid confusing data-recipe effects with architecture effects.
  4. Do not convert token savings directly into serving gains. Measure prefill throughput, time per decoded token, peak memory, KV or recurrent-state footprint, and long-context quality separately. A 30% pretraining-token reduction does not automatically produce a 30% online cost reduction.
  5. Use a two-layer evaluation matrix. Retain knowledge benchmarks such as MMLU, but add exact retrieval, multi-hop composition, induction, and shortcut-resistance tests. Report absolute scores, variance, failure categories, and degradation curves across context lengths.
  6. Wait for replication before replacing a production architecture. Systems requiring exact copying, very long context, or durable tool state must verify information fidelity in recurrent-state architectures. Teams whose primary constraint is pretraining cost can test Canon variants in an equal-budget A/B experiment now.

Bottom line

The important result is not a declaration that linear attention or the Transformer has won. Sixty-four real-scale pretraining runs show that architectural differences can be hidden by tuning and can remain invisible beneath strong standard-benchmark scores. Canon layers put local token mixing back at the center of architecture research, while synthetic diagnostics, exact retrieval, and multi-hop evaluation provide a more informative path than relying on MMLU alone.


Source: ArXiv NLP/LLM (cs.CL) (2026-11-20)

— END —

Tags: #Canon Layers #Linear Attention #Pretraining Token Efficiency #In-Context Learning #Synthetic Evaluation

Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.

What's your take on this story?
🚀 Explore & Submit Works

Community Comments

Loading live comments and annotations…