Mixture of Wide Experts: Sparse Routing for Native Interpretable Coordinates
Summary:The paper introduces Mixture of Wide Experts (MoWE), a sparse feed-forward layer whose selected experts read the full hidden representation but write only to fixed, disjoint output coordinates. This makes interpretable features native to the model rather than requiring a separate sparse autoencoder to reconstruct them post hoc. Under approximately matched training compute, MoWE reaches an evaluation cross-entropy of 2.417, better than a dense model's 2.452 but worse than a standard MoE's 2.22.
Core Idea
The paper introduces Mixture of Wide Experts (MoWE), a sparse feed-forward design in which concepts can be represented by stable, directly readable coordinates inside the model. Instead of relying on a separately trained sparse autoencoder to reconstruct features after the fact, MoWE learns an output space whose coordinates are partially isolated by construction.
Each token activates only a subset of experts. Selected experts read the full hidden representation, but each expert writes only to a fixed, disjoint group of output coordinates. Routing therefore determines the conditional computation path, while coordinate ownership determines which interpretable features can be activated.
Architecture and Computation
Let (h) be the hidden state, (r(h)) the router, (W_e^{in}) the input projection of expert (e), and (M_e) its fixed output mask. The MoWE layer can be expressed as:
[ y_{FFN}(h)=\sum_{e\in TopK(r(h))} M_e\odot\phi_e(W_e^{in}h) ]
where (\odot) denotes element-wise multiplication. Each expert still receives the full hidden state, preserving broad contextual access, but its output is restricted to a predetermined coordinate subset.
This is the main difference from conventional MoE. The objective is not merely to activate fewer parameters; it is to couple sparse conditional computation with stable feature ownership. The corresponding engineering constraint is equally important: coordinate partitions must remain fixed and semantically consistent across training and inference.
Performance–Interpretability Trade-off
The study trained two 19-layer runs of each architecture for 10 billion tokens under approximately matched compute:
| Architecture | Evaluation cross-entropy | Native-coordinate readability specificity | |---|---:|---:| | MoWE | 2.417 | 0.523 | | Dense FFN | 2.452 | 0.375 | | Standard MoE | 2.22 | 0.268 |
MoWE slightly outperforms the dense baseline on cross-entropy, showing that coordinate isolation does not necessarily damage predictive performance. It nevertheless remains 0.197 behind standard MoE. Conventional MoE experts can write into the full output space, while MoWE must preserve disjoint coordinate ownership and stable semantics.
For readability, a language-model judge generated descriptions of coordinates from strongly activating passages. Those descriptions were then tested on held-out passages, with a correction for descriptions that also fit other coordinates. MoWE reaches 0.523 specificity, compared with 0.375 for the dense model and 0.268 for standard MoE.
TopK SAEs trained on the same FFN updates score 0.489 for the dense model and 0.556 for MoWE. Native MoWE coordinates are therefore comparable to features recovered by a separately trained dense SAE, potentially reducing the cost and interpretation mismatch of post-hoc feature extraction.
Case Studies: Can Coordinates Intervene on Behavior?
In the first case study, the researchers identify a four-coordinate causal path associated with the model's prediction of degrees after A right angle is 90. They then copy seven angle-related coordinate values while the model processes A rectangle has four, changing the next-token prediction from sides to degrees.
This suggests that the coordinates are not merely descriptive but can participate in causal computation. The experiment is nevertheless a controlled intervention: it directly modifies the model's internal state and does not establish that arbitrary coordinates can be edited safely in production.
In the second case study, one fixed coordinate set detects hazardous requests, while another steers the next token toward refusal. This suggests a route for internal safety auditing, but the reported evidence does not yet establish false-positive rates, false-negative rates, adversarial robustness, or production stability.
Evaluation Limitations and Engineering Costs
Readability depends on an LLM judge. Generated descriptions may inherit the judge's biases or produce plausible but incorrect causal stories. Human annotation and independent classifiers should be used to validate the results.
Controlled tasks do not imply open-world interpretability. Experiments with known hidden causes are useful for mechanism testing, but they do not prove that coordinates remain stable across web text, code, multilingual data, or long contexts.
Sparse computation does not automatically mean low latency. TopK routing can reduce expert FLOPs, but routing, scheduling, and masked writes add overhead. The summary provides no throughput, memory, or end-to-end latency measurements, so deployment efficiency cannot be inferred from the architecture alone.
The performance gap against standard MoE remains significant. MoWE trades some predictive quality for clearer coordinate semantics. It may be less attractive for quality-only training, but more useful for debugging, model editing, safety auditing, and controllable generation.
Practical Guidance
- Test semantic stability. Verify whether a concept maps to the same coordinates late in training and across domains, prompt formats, and routing paths.
- Use independent interpretability evaluation. Combine human labeling, targeted probes, and LLM judging rather than relying on a single judge score.
- Measure systems cost. Track activated experts, routing balance, memory use, batch throughput, and P99 latency in addition to FLOPs and cross-entropy.
- Start with high-value debugging tasks. Internal probes, behavior audits, concept-level interventions, and safety-policy verification offer lower migration risk than replacing a mature production model wholesale.
- Constrain coordinate interventions. Apply activation thresholds, value bounds, rollback mechanisms, and test suites before allowing coordinates to be edited.
MoWE offers an alternative to post-hoc interpretability: it builds sparse conditional computation and native coordinate structure into the model itself. The controlled results show a promising quality–interpretability trade-off, but causal faithfulness in large-scale language models still requires broader datasets, stricter evaluation, and deeper systems measurements.
— END —Source: ArXiv Machine Learning (cs.LG) (2026-11-05)
Tags: #MoWE #Mixture of Experts #Interpretability #Sparse Autoencoders #Sparse Routing
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments