ZICQ
中 Log in / Sign up
ZICQ Info Open Source AI #OVHcloud #Olala mini #MoE #Mamba-3 #Edge Inference

OVHcloud Announces Olala Mini 7A1B: A 7B MoE with 1B Active Parameters for Edge Inference

Avatar of Mr.Xu

By Mr.Xu Community Post

Published: · 2 views

中文阅读 (Chinese) English Version
Co-reading Select any sentence in the article to highlight, annotate, or copy a quote card. Join discussion ↓

Summary:OVHcloud has announced Olala mini 7A1B, a 7-billion-parameter MoE language model with roughly 1 billion active parameters per token and 256 experts per layer. Its sequence stack combines Differential-TPA with Mamba-3; the announced context length is 32K, and OVHcloud plans to publish the model on Hugging Face under Apache 2.0. OVHcloud reports 160 tokens per second on an RTX A4500 and 50 tokens per second on a CPU, while also claiming that the model can run with 8 GB of GPU memory. Those figure


OVHcloud's Olala mini 7A1B combines 7B total parameters, roughly 1B active parameters, 256 experts per layer, and a Differential-TPA–Mamba-3 hybrid sequence stack. The company plans to publish the model on Hugging Face under Apache 2.0. The available announcement is sufficient to identify the intended architecture, but not to confirm that the 8 GB memory claim and the reported throughput can be reproduced under the same precision, runtime, and prompt conditions.

Announcement scope and confirmed details

This is currently a model announcement rather than evidence of a broadly available stable release. The excerpt says the model will be available on Hugging Face and lists the following specifications:

  • 7B total parameters, with approximately 1B active parameters per token under the 7A1B naming convention and standard MoE interpretation.
  • 256 experts per MoE layer, although the Top-K routing value, number of shared experts, duplication policy, and model depth are not disclosed.
  • A sequence stack combining Differential-TPA and Mamba-3, with BF16 weights.
  • An initial context length of 32K tokens.
  • A planned Apache 2.0 release on Hugging Face.

A planned weight release is not automatically a complete open research stack. Even if Apache 2.0 is ultimately used, users will need to verify whether inference code, training code, configuration files, tokenizers, data recipes, evaluation scripts, and training logs are also published.

What the MoE design changes—and what it does not

Olala mini relies on sparse expert activation. A router selects a small subset of experts for each token, allowing the other expert weights to remain outside that token's matrix multiplications. This reduces arithmetic work per token, but adds routing, load-balancing, and expert-scheduling overhead.

Ignoring attention, state-space scanning, and routing, a rough dense-GEMM lower bound is:

FLOPs per token ≈ 2 × active parameters ≈ 2 GFLOP

At that scale, 160 tokens per second would imply at least about 320 GFLOP/s of multiply-accumulate work, while 50 tokens per second on a CPU would imply roughly 100 GFLOP/s. These are order-of-magnitude estimates, not benchmarks. Expert transfers, attention, Mamba scans, memory bandwidth, and framework scheduling can dominate the actual cost.

A 256-expert configuration is fine-grained MoE. It can improve routing granularity and distribute load, but it also introduces all-to-all communication during training and potentially expensive expert placement during inference. Different tokens can select different experts, so a long request may eventually touch most experts even if each individual token activates only 1B parameters. If experts are split between host and device memory, PCIe transfers can replace matrix multiplication as the dominant latency source.

Memory audit: the 8 GB claim needs an explanation

For 7B BF16 parameters, raw weight storage alone is:

7 × 10^9 × 2 bytes = 14 GB ≈ 13.0 GiB

That figure excludes the runtime, activations, attention KV cache, Mamba recurrent states, and temporary workspaces. Sparse routing reduces the parameters used in computation, but it does not automatically remove the storage occupied by inactive experts. Fitting the complete model into 8 GB therefore requires at least one additional mechanism, such as quantization, tiered loading, CPU offload, or host-memory paging.

A nominal 4-bit format would reduce raw 7B weights to about 3.5 GB, before metadata and runtime overhead, but the resulting quality loss would need layer-specific validation. CPU offload could keep only selected experts on the GPU, yet would increase transfer latency, especially when batch size is small. The announcement does not identify the mechanism used for the 8 GB configuration.

A Mamba-3 path can maintain a fixed-size recurrent state rather than storing an equally large KV cache for every historical token. The memory behavior of the Differential-TPA branch is unknown, however, as are the state dimensions, context scaling, and cache policy. A 32K context window therefore does not by itself establish low-memory operation.

What the hybrid sequence layer could change

Mamba-3 follows the selective state-space model approach. Its potential benefit is to carry historical information through a fixed-size recurrent state with a sequence scan that is more favorable for long contexts. Differential-TPA may provide content-dependent attention comparisons, while Mamba-3 maintains compressed sequential state. Together, they could address both exact retrieval and streaming-state requirements.

That division of labor is architecturally plausible, but it is not a confirmed description of the implementation. The announcement does not provide Differential-TPA's formula, receptive field, state dimensions, parallelism strategy, or fusion method. It is therefore impossible to determine whether the complete stack is linear in sequence length, whether the attention branch retains quadratic behavior, or how much the hybrid improves quality and latency over either component alone.

Hybrid models can also be harder to deploy than pure Transformer or pure SSM designs because their branches may require separate kernels. The reported CPU result of 50 tokens per second needs the processor model, memory configuration, thread count, instruction set, native BF16 support, and quantization status before it can be treated as a portable baseline.

Auditing the performance claims

| Claimed result | Technical implication | Evidence still needed | |---|---|---| | Runs with as little as 8 GB of GPU memory | Positions the model for consumer GPUs or hybrid offload | GPU model, quantization format, expert placement, peak memory, and PCIe paging behavior | | 160 tokens/s on an RTX A4500 | Attractive single-device decode throughput | Batch size, prompt and output lengths, runtime, compilation flags, time to first token, and numerical precision | | 50 tokens/s on a CPU | Maintains usability without a discrete GPU | CPU model, memory channels, thread settings, power draw, sustained throughput, and quantization | | 32K context | Supports longer documents and codebases | Retrieval accuracy, KV or recurrent-state footprint, and throughput degradation with context length |

The RTX A4500 has 24 GB of VRAM, so its 160-token/s result does not directly demonstrate that an 8 GB configuration sustains the same speed. These are likely separate deployment configurations. Likewise, 50 tokens per second measured with a short prompt and batch size of one may not extrapolate to long-context or multi-request serving.

The announcement also provides no quality benchmarks for perplexity, instruction following, code generation, tool use, or long-context retrieval. A 1B active-parameter budget lowers computation, but it does not establish better capability per unit of compute than a dense 3B–4B model. Fine-grained routing can also produce expert imbalance and congestion on less frequently selected experts.

Open-source status and engineering trade-offs

If Apache 2.0 ultimately covers both the weights and relevant code, the release would be favorable for commercial deployment and further development. The actual release date, data disclosure, and support across inference frameworks remain unconfirmed. Olala mini's value ultimately depends on whether three properties can coexist: low memory residency, sustainable throughput, and acceptable task quality.

  • Sparse MoE exchanges routing and scheduling complexity for lower arithmetic cost, favoring throughput-oriented workloads but not necessarily low-latency, small-batch requests.
  • Mamba-3 can limit recurrent-state growth, although an attention branch may reintroduce KV-cache costs.
  • BF16 preserves numerical range and stability, while 4-bit quantization materially reduces storage but requires calibration and fresh long-context and instruction-following evaluations.
  • CPU fallback improves deployment resilience, but a 50-token/s peak does not establish sustained concurrency, power efficiency, or service-level performance.

Deployment recommendations for developers

  1. Audit the release artifacts. Verify that the Hugging Face repository contains complete weights, configuration, tokenizer, license text, inference code, model card, and checksums, and confirm which files Apache 2.0 covers.
  2. Reproduce the official baseline. Pin the GPU or CPU, drivers, inference framework, compiler flags, batch size, prompt length, output length, and sampling settings. Record time to first token, steady-state decoding speed, peak VRAM, host RAM, and PCIe transfer volume.
  3. Sweep context length. Measure prefill time, generation throughput, and memory slope at 2K, 8K, 16K, and 32K rather than validating the 32K claim only with short prompts.
  4. Compare quantization quality. Run BF16, 8-bit, and 4-bit variants on the same prompts with deterministic decoding, covering code, mathematics, tool use, and long-document retrieval.
  5. Test expert-offload behavior. If CPU offload is involved, measure expert paging, cache hit rate, and tail latency; production deployments must also test routing imbalance and expert-capacity limits.
  6. Select using production metrics. Evaluate P50 and P95 latency, throughput per watt, concurrent request capacity, memory use, and task completion rate instead of relying on peak tokens per second.

Olala mini 7A1B presents a clear direction: use fine-grained MoE to control per-token computation and Mamba-3 to carry part of the sequential state, pushing a language model toward 8 GB-class hardware. Yet the reported speed and BF16 memory footprint are not reconciled by the available material, and the 8 GB claim especially requires quantization or offload details. The model belongs on a local-inference evaluation shortlist, but it should not replace independent stress testing until weights, quality results, and reproducible configurations are available.


Source: Reddit r/LocalLLaMA (2026-11-17)

— END —

Tags: #OVHcloud #Olala mini #MoE #Mamba-3 #Edge Inference

Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.

What's your take on this story?
🚀 Explore & Submit Works

Community Comments

Loading live comments and annotations…