ZICQ
中 Log in / Sign up
ZICQ Info LLMs & Foundation Models #Maxwell-V1.0-Flash #MoE #Block-Sparse Attention #256K Long Context #Model Release

Maxwell-V1.0-Flash Debuts 35B MoE with 3B Active Parameters and 256K Context

Avatar of Mr.Xu

By Mr.Xu Community Post

Published: · 2 views

中文阅读 (Chinese) English Version
Co-reading Select any sentence in the article to highlight, annotate, or copy a quote card. Join discussion ↓

Summary:Maxwell Team has released Maxwell-V1.0-Flash, a 35B-total-parameter MoE language model that activates 3B parameters per token, excluding embeddings. It targets reasoning, agentic workflows, and contexts up to 256K tokens. Its central mechanism is Maxwell Block Sparse Attention, or MBSA: eight prefix blocks, eight local blocks, and up to 16 learned-indexed remote blocks limit each KV group's main attention span to 4,096 tokens, while the full-history KV cache remains available. The model card rep


Architecture Core: MBSA Bounds Long-Context Attention

Maxwell-V1.0-Flash uses a fine-grained MoE backbone with 35B total parameters and 3B active parameters per token, excluding embeddings. It is designed for reasoning, agentic workloads, and contexts of up to 256K tokens. Its central mechanism is not simply a larger KV cache: Maxwell Block Sparse Attention, or MBSA, limits the amount of history that participates in main attention for each KV group.

At a 256K context length, the sequence contains 262,144 tokens. Dense GQA reduces K/V projection and caching costs by sharing K/V heads, but every query still interacts with the full history. Its idealized per-layer attention workload remains close to O(S²d_head). Under the model's 128-token block formulation, MBSA retains eight prefix blocks and eight local blocks, then uses a learned indexer to select as many as 16 remote blocks. The resulting 32 blocks allow at most 4,096 tokens to participate in main attention per group. Ideally, this changes the main attention workload from O(S²d_head) to O(SBd_head), where B equals 4,096.

Learned remote-block selection preserves access to earlier history, but missed or incorrectly selected blocks create a new retrieval bottleneck. Two effects must be separated: the model card says the full-history KV cache remains available, meaning that MBSA primarily compresses the attention span rather than reducing KV-cache memory to a constant.

Fine-Grained MoE and Multiple Residual Streams

The 3B active-parameter figure means that only a subset of routed experts executes for each token. It is neither equivalent to storing only 3B parameters nor an exact statement of total FLOPs. All 35B weights must still reside in memory or across multiple GPUs, alongside routing, attention, always-active shared experts, embeddings, and communication overhead.

Fine-grained MoE uses more smaller experts to obtain finer-grained specialization, while shared experts provide a stable common computation path. A dynamic routing correction bias and a sequence-wise load-balancing loss are intended to suppress expert collapse and hotspot routing. The public material does not disclose the number of experts, Top-K configuration, load variance, or communication share. Multi-GPU expert parallelism typically relies on All-to-All communication, so finer experts do not automatically produce better kernel efficiency.

Manifold-Constrained Hyper-Connections introduce multiple residual streams and use input-dependent learned aggregation with constrained mixing to widen information flow. The model card states that the design does not add attention or feed-forward evaluations. Its primary overhead should instead lie in aggregation control, but the mathematical form of the constraint and the additional parameter count still require code-level verification.

Training and Post-Training Recipe

The model was pretrained on approximately 26T multilingual tokens, with the Muon optimizer applied to most hidden weight matrices. Training begins with dense attention to establish the base model. Block-sparse attention is introduced at the 128K stage, after the learned indexer has been warmed against the full dense history, and is then trained jointly with the backbone. This curriculum can reduce cold-start instability in the sparse indexer, but it does not automatically eliminate loss spikes or quality degradation at the dense-to-sparse transition.

Long-context training expands from 4K to 256K in stages. A script-aware multilingual tokenizer is designed to remain compact across languages, including low-resource Southeast Asian languages such as Thai, Lao, Myanmar, Khmer, and Shan. The relevant measurements are token compression, tokens per character, embedding quality, and downstream accuracy, rather than vocabulary size alone.

Post-training begins with all-domain SFT covering general, reasoning, and agentic capabilities, followed by harder reasoning and agent trajectories. General tasks, verifiable reasoning, and agentic domains such as search, terminal use, function calling, and software engineering are then handled by independent RL branches. The branches are merged through multi-teacher on-policy distillation: teachers supervise responses generated by the student itself, reducing the mismatch between offline teacher data and the student's actual sampling distribution. The tradeoffs are additional teacher inference cost, conflicting reward signals, and the difficulty of reconciling several specialist policies.

Performance Evidence: The 6.7x Figure Is Kernel-Level

| Published claim | Technical meaning | Missing evidence | |---|---|---| | MBSA kernel is 6.7x faster than dense GQA at 256K | Local acceleration of the sparse-attention kernel | Hardware, precision, batch size, warm-up, end-to-end throughput, and quality parity | | 35B total parameters with 3B active per token | Lower expert-forward computation | Expert count, Top-K, measured FLOPs, memory use, and communication cost | | Support for a 256K context | Interfaces and training objectives cover long sequences | Long-context accuracy, cross-segment reasoning, and recall degradation curves | | Approximately 26T pretraining tokens | Substantial training-data scale | Data composition, contamination audit, training compute, and ablations |

MBSA reduces the attention span to roughly 1/64 of the full history, yet the official figure reports only a 6.7x kernel gain. These observations are not contradictory: attention is only one part of end-to-end inference, and indexing, cache reads, reductions, memory bandwidth, and scheduling still limit the achievable gain. More importantly, a faster kernel does not guarantee unchanged answer quality because missed remote blocks can still remove key evidence.

The available material contains no auditable unified benchmark table and does not disclose comparable results for MMLU, GPQA, AIME, long-context retrieval, code generation, or agent success rate. It is therefore impossible to determine whether baselines used equal compute, whether prompt templates and sampling parameters were aligned, or whether 256K support reflects effective reasoning rather than merely accepting a longer sequence.

Engineering Trade-offs: Lower Compute Does Not Mean Lower Memory

First, MBSA reduces token interactions in main attention, but the full-history KV cache still grows linearly with sequence length. Second, fine-grained MoE lowers expert computation per token while preserving full-weight residency, expert routing, and cross-device communication costs. Third, remote-block selection converts dense attention recall into a sparse-retrieval problem, making the indexer a critical reliability component. Finally, on-policy distillation can smooth the integration of separate RL branches, but it requires extra teacher inference and repeated trajectory sampling.

A rough lower bound for KV-cache memory is M_KV ≈ 2 × L × H_KV × d_head × S × b, where L is the number of layers, H_KV the number of KV heads, d_head the head dimension, S the sequence length, and b the bytes per element. This is calculated per sequence and per batch item. For example, 32 layers, eight KV heads, a head dimension of 128, BF16 storage, and a 256K context produce roughly 32 GiB per sequence. The actual value changes with the GQA configuration, and batching multiplies it further. KV-cache quantization can reduce this footprint if implemented, but the model card does not establish support for it.

Deployment and Acceptance Recommendations

Before deployment, verify the completeness of the model card, tokenizer, configuration files, and GitHub implementation, then confirm the weight license, training-data documentation, and commercial-use terms. The 35B/3B split alone is insufficient for capacity planning: based on the non-embedding parameter count, raw weights occupy roughly 70GB in BF16, 35GB in INT8, and 17.5GB at 4-bit, before embeddings, quantization metadata, runtime buffers, and KV cache.

Next, verify that the selected inference runtime actually integrates the MBSA kernel. Loading Hugging Face weights does not automatically guarantee sparse execution; without matching operators or a custom kernel, the model may fall back to a conventional implementation. Production deployment at 256K will usually require tensor parallelism and KV sharding, while multi-GPU installations must evaluate the network topology used for expert parallelism rather than relying solely on single-device weight footprint.

Acceptance tests should cover 4K, 32K, 128K, and 256K, recording time to first token, time per output token, concurrent throughput, peak memory, energy per token, and long-context quality. Quality suites should include long-range retrieval, cross-segment reasoning, mathematics, code generation, function calling, terminal operation, and software-engineering trajectories. Dense GQA should be tested on the same hardware, precision, and batch settings to verify that the speed gain has not been purchased through lower recall accuracy.

For agentic use, the model should run inside a least-privilege sandbox with restricted network egress, credential mounting, and command execution. External actions should also pass structured validation and human approval. Long-context support does not remove tool-call overreach, prompt injection, or trajectory-error accumulation.


Source: Reddit r/LocalLLaMA (2026-11-17)

— END —

Tags: #Maxwell-V1.0-Flash #MoE #Block-Sparse Attention #256K Long Context #Model Release

Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.

What's your take on this story?
🚀 Explore & Submit Works

Community Comments

Loading live comments and annotations…