K2-Horizon Opens Six Models with MoVA Sparsity and Uno Diffusion Decoding
Summary:IFM released K2-Horizon, a six-model open family spanning a 0.9B dense model and a 375B-A23B MoE. The release goes beyond weights: under its stated LLM360 interpretation, it includes pretraining data, stage-by-stage checkpoints, training logs, evaluation assets, the production-scale xLLM training framework, and the RL360 reinforcement-learning stack. Four models share a 22T-token corpus and recipe, while the disclosed data pipelines generated roughly 10T synthetic pretraining tokens. The techni
K2-Horizon is not simply another single-model weight release. It is a six-model open family from IFM for general reasoning and agentic workloads. The range extends from a 0.9B dense model to a 375B-A23B MoE, where A23B denotes roughly 23B active parameters per token. More importantly, the authors say they follow the LLM360 definition of open source by releasing pretraining data, intermediate checkpoints from every training stage, logs, evaluation assets, the xLLM training framework, and the RL360 reinforcement-learning stack rather than only final checkpoints.
Release Scope and Open Assets
The six models cover both small dense configurations and large sparse architectures. K2-Horizon 36B-A4B serves as the main test case for Mixture-of-Value Attention, softplus attention gating, and Uno. Four models share the same 22T-token pretraining corpus and recipe, creating a relatively controlled setting for studying model scale and sparsity. The disclosed data pipelines also generated approximately 10T synthetic pretraining tokens, indicating that synthetic data was a substantial part of pretraining rather than a small post-training supplement.
The open assets reportedly include model weights, stage-by-stage checkpoints, pretraining data, training logs, evaluation code, and infrastructure such as xLLM and RL360. The research value lies in exposing the intermediate model-development process, allowing researchers to inspect relationships among data recipes, training trajectories, and final behavior rather than merely rerun inference from a final checkpoint.
MoVA: Expertizing the Attention Value Path
A standard attention operation can be written as:
Attention(Q, K, V) = softmax(QK^T / sqrt(d_h))V
Conventional MoE usually places experts in the feed-forward network and routes each token to a small subset of expert parameters. MoVA changes the location of sparsity: it expertizes the value projection itself. Conceptually, different experts compute different value representations:
V_e = W_e^V x
V = sum_e g_e(x)V_e
The router then combines these values before the attention operation aggregates them. Different tokens can therefore use both different FFN experts and different value subspaces, increasing conditional representational capacity while limiting per-token computation to the selected experts. The exact routing, normalization, and load-balancing mechanisms still need to be checked in the full paper.
Softplus attention gating provides a second structural change. A typical softplus gate is:
softplus(z) = log(1 + exp(z))
It maps a gate score to a smooth, nonnegative continuous weight. If top-k selection is used, only the selected experts participate in the forward pass. Compared with a hard discrete gate, a continuous gate can offer smoother optimization, but the claimed theoretical advantage may arise from smoothness, additional expressiveness, or a particular parameterization. Ablations are needed to distinguish these effects.
MoVA does not remove the quadratic sequence-length term in standard attention. Attention-score computation still grows approximately with S^2, while MoE adds routing, expert selection, and potentially All-to-All communication. Training normally requires load-balancing objectives, expert-capacity controls, and monitoring for unstable routing. Inference must also prevent hot experts from creating GPU idle time. The same-data comparison between 36B-A4B and a 32B dense model is useful controlled evidence, but the abstract does not provide task-level results, variance estimates, or separate contributions from MoVA and softplus gating.
Uno: Diffusion Decoding Without a Separate Draft Model
Uno uses lossless diffusion adapters to accelerate autoregressive decoding. The authors report up to a 2.2x speedup without deploying a separate draft model. From an engineering perspective, the adapter can be understood as proposing a block of future tokens from the current context, followed by target-model validation or parallel confirmation. A valid candidate block then advances decoding by multiple positions and shortens the serial execution chain.
The main benefit is the elimination of a separate draft model's weights, memory footprint, and operational overhead. A 2.2x result is not, however, guaranteed by the adapter architecture alone. Candidate acceptance rate, target-model verification cost, batch size, time to first token, and tail latency all affect end-to-end performance. The term lossless also requires precise interpretation: does it mean identical token sequences, preservation of the sampling distribution, or only equivalent benchmark quality? Different sampling rules or acceptance procedures can invalidate a losslessness claim unless those details are disclosed.
Performance Evidence and Evaluation Boundaries
The abstract provides three main classes of evidence:
| Evidence | What it supports | What it does not establish | |---|---|---| | 36B-A4B approaches a 32B dense model trained on the same data | Sparse value paths may retain quality while sharply reducing active computation | It does not prove task-wide equivalence or isolate the contributions of MoVA and softplus gating | | Six models lead or approach strong open models at comparable sizes on agentic and coding benchmarks | The authors report competitive results | Exact scores, baseline versions, prompts, tool environments, sampling budgets, and error intervals are missing | | Four models share a 22T-token corpus and recipe | This provides a cleaner setting for studying scale and sparsity | Synthetic-data ratios, training order, and corpus mixtures may still confound comparisons | | Uno reaches a reported 2.2x decoding speedup | This is a strong engineering signal for diffusion-based decoding | Acceptance rates, hardware, batch size, throughput, and tail latency are not disclosed |
The nominal active-parameter count of 36B-A4B is 4B, compared with approximately 32B for the dense model, so a direct division is closer to one-eighth. The reported one-seventh figure may use a different definition of effective active parameters. Engineers should verify the accounting through FLOPs, routing-success rates, and shared parameter handling before using it for capacity planning.
Agentic and coding evaluations are especially sensitive to the tool harness, pass@k policy, retry count, sampling temperature, judge model, and context construction. Publication in Hugging Face Daily Papers is a distribution channel, not independent replication or peer review. The public checkpoints, logs, and evaluation code make meaningful ablation possible, but the current abstract alone cannot establish whether the reported ranking transfers to production workloads.
Open Does Not Yet Mean Reproduced
The rarest part of this release is not a single checkpoint but the production chain connecting data generation, pretraining, checkpoint selection, and reinforcement learning. xLLM supplies dense and MoE training implementations, RL360 provides the reinforcement-learning stack, and the 22T-token corpus plus intermediate checkpoints let researchers locate changes in training behavior.
Strict reproducibility still requires verification of several items:
- The licenses applied separately to models, data, code, and logs, and whether commercial and redistribution rights are consistent;
- Whether every checkpoint, data shard, and code commit has a hash, version identifier, and machine-verifiable manifest;
- The provenance, licensing, privacy treatment, deduplication, and contamination checks for the pretraining corpus;
- The teacher models, generation settings, filtering rules, deduplication pipeline, and mixture ratios behind the synthetic tokens;
- Whether CUDA, compiler, communication-library, seed, and distributed-topology details are sufficient for near-deterministic training;
- The GPU count, token budget, energy use, and monetary cost required to reproduce training.
If these assets are downloadable under clear terms, K2-Horizon could become a valuable public platform for sparse-attention research, synthetic-data studies, and open reinforcement-learning systems.
Memory, Throughput, and Deployment Trade-offs
MoE active parameters constrain per-token computation but do not automatically reduce the resident memory occupied by all expert weights. A rough bare-weight estimate is:
| Model | Theoretical BF16 weight size | Theoretical 4-bit weight size | Deployment implication | |---|---:|---:|---| | 0.9B dense | About 1.8 GB | About 0.45 GB | Suitable for local development, edge validation, and inexpensive pipeline tests | | 36B-A4B | About 72 GB | About 18 GB | Quantization may permit a single accelerator or compact multi-accelerator setup, but KV Cache and runtime overhead remain | | 375B-A23B | About 750 GB | About 187.5 GB | Typically requires multi-GPU sharding, expert parallelism, or tiered offloading |
These are theoretical parameter-storage figures only. They exclude quantization metadata, activations, routing buffers, and the KV Cache. A standard cache has a scale roughly expressible as 2 × L × S × H_kv × d_head × bytes. GQA or MQA can reduce this substantially, but the abstract does not disclose the relevant configuration. Lower active parameters in an MoE also do not imply a smaller context cache.
Throughput and latency pull in different directions. Expert parallelism improves GEMM utilization at high batch sizes but introduces token dispatch, All-to-All communication, and load imbalance. Low-concurrency requests are more sensitive to expert selection, network latency, and offload transfers. Expert offloading can relieve memory pressure but places weight movement on the critical path of every token. Uno primarily shortens the decoding call chain; it does not eliminate KV Cache pressure at long context lengths and may not improve time to first token.
Practical Adoption Guidance
- Audit the artifacts first: Check weights, loaders, tokenizers, licenses, context limits, quantization configurations, and data manifests rather than deploying from the model card alone.
- Evaluate by size tier: Use the 0.9B model to validate tool calls, templates, and data pipelines; compare the 36B-A4B model against a dense model of similar quality; assess the 375B-A23B model only in a cluster environment. Do not compare official speedups across different hardware.
- Break down performance: Record TTFT, prefill tokens per second, decode tokens per second, p50 and p99 latency, peak memory, expert-load deviation, and KV Cache occupancy.
- Define Uno's lossless boundary: Compare token sequences, logits, task scores, and rejection rates under greedy and stochastic sampling with the base model, while logging acceptance rates and verification overhead.
- Reproduce the controlled comparisons: Use the four models sharing the 22T-token recipe to reproduce scaling curves and ablate MoVA, softplus gating, active parameters, and synthetic-data ratio separately.
- Keep production fallbacks: Agent deployments need tool-failure recovery, context truncation, approval gates for sensitive operations, and independent evaluation. Leaderboard performance is not a substitute for runtime safety controls.
K2-Horizon's most novel architectural move is to bring conditional MoE into the attention value path and combine it with diffusion adapters for faster decoding. The more consequential release may be its attempt to open the complete production chain from data to training stack. If the artifacts, licenses, and evaluations survive independent scrutiny, the family could become an important public baseline for open-model research and MoE engineering.
— END —Source: Hugging Face Daily Papers (2026-11-16)
Tags: #K2-Horizon #Mixture-of-Experts #Mixture-of-Value Attention #Diffusion Decoding #Open Training Stack
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments