ZICQ
中 Log in / Sign up
Newsroom Open Source AI #WeMM-Video #Omni-Modal Video Understanding #Hybrid Reasoning #Sparse MoE #Open-Source Multimodal Models

WeMM-Video Releases an Open-Source Omni-Modal Video Family, Led by 35B-A3B

Avatar of Mr.Xu

By Mr.Xu Compiled & Reviewed by Editorial

Published:

中文阅读 (Chinese) English Version
Co-reading Select any sentence in the article to highlight, annotate, or copy a quote card. Join discussion ↓

Summary:WeMM-Video is a bilingual Chinese-English omni-modal model family for audio-enhanced video understanding. It unifies text, image, audio, and video inputs while offering a non-thinking mode for direct answers and a thinking mode for explicit step-by-step reasoning. The 2B, 4B, 9B, and 35B-A3B variants are progressively pretrained on roughly 1.2 trillion tokens and post-trained with multi-teacher OPD, anti-repetition MrDPO, and specialist RL. The paper reports a new open-source state of the art f


Core Release

WeMM-Video uses one model family to process text, images, audio, and video, with a non-thinking instruction mode for direct answers and a thinking mode for explicit step-by-step reasoning. The family contains 2B, 4B, 9B, and 35B-A3B variants, covering lightweight local inference through server-scale deployments.

| Variant | Stated scale | Practical positioning | |---|---|---| | 2B | 2B-class | Edge devices, low-cost batch processing, and high-concurrency lightweight tasks | | 4B | 4B-class | A more balanced quality-versus-resource option | | 9B | 9B-class | Single-GPU or quantized deployment, with parameter efficiency highlighted by the paper | | 35B-A3B | 35B overall with A3B naming | Flagship serving, subject to the memory and communication cost of the complete expert set |

Under common sparse-MoE naming conventions, A3B usually means that roughly 3B parameters are activated per token while the complete model retains about 35B parameters. The exact expert count, routing granularity, and shared-expert configuration still need to be confirmed in the model card.

Architecture: From Multimodal Inputs to Dual-Mode Reasoning

The central problem is not simply concatenating video frames, audio, and text. The model has to bind three information streams along both temporal and semantic axes: visual frames establish what happened, audio contributes speech, prosody, and paralinguistic cues, and text supplies the question and reasoning constraints. The abstract does not disclose the visual and audio encoders, fusion point, temporal sampling strategy, or context length, so several implementation details remain unverified.

  • Temporal alignment: Video must be encoded as frames or spatiotemporal patches, while audio is segmented into timestamped chunks. Different sampling windows can create audiovisual misalignment even when textual reasoning is otherwise correct.
  • Cross-modal grounding: Spoken content must be associated with the correct person and visual event while retaining information such as pace, pauses, and emotion. An ASR-only preprocessing path discards much of this signal.
  • Token-budget management: Frame count, resolution, and audio-chunk length jointly determine input size. Long videos often hit the context limit before generation limits, making keyframe selection, spatiotemporal compression, and segment retrieval more important than simply extending the context window.
  • Dual-mode inference: The non-thinking path directly generates an answer and generally reduces reasoning-token overhead. The thinking path exposes intermediate steps and is more suitable for tasks requiring compositional audiovisual evidence, but increases output length, cost, and the possibility of propagating an erroneous reasoning chain. Hybrid thinking should not be equated with automatic routing because no task router is described.

General-audio, silent-video, image, and text-only evaluations check whether individual modalities regress. Genuine audiovisual reasoning requires controlled ablations to establish that the model is listening, seeing, and binding the two signals rather than relying on subtitles or language priors.

Training and Post-Training

The paper says the models were progressively pretrained on approximately 1.2 trillion multimodal tokens. Progressive training can reduce interference between newly introduced modalities and existing language capability, but the supplied material provides no stage-by-stage data mix, curriculum, modality proportions, or catastrophic-forgetting analysis.

The post-training pipeline includes multi-teacher OPD, anti-repetition MrDPO, and specialist RL. According to the abstract's framing, multi-teacher OPD combines complementary supervision, anti-repetition MrDPO discourages loops in long answers or reasoning chains, and specialist RL optimizes targeted capabilities. However, no objectives, teacher list, reward design, sampling strategy, or ablation results are provided, so the practical contribution of each component cannot be inferred from its name alone.

A 1.2-trillion-token corpus also creates two evaluation risks: benchmark decontamination and uneven representation across languages, audio conditions, and video domains. These factors directly affect performance on Chinese dialects, low-light footage, multi-speaker scenes, and low-resource audio.

Engineering Trade-offs and Memory Cost

| Design choice | Main benefit | Cost or risk | |---|---|---| | Unified multimodal model | Fewer interfaces and less error propagation between ASR, vision models, and an LLM | Weak data in any modality can drag on the shared backbone, and upgrades become more coupled | | Non-thinking and thinking modes | Trade response cost against deeper reasoning | Capability and output style may differ across modes | | 35B-A3B sparse architecture | Per-token computation resembles that of a smaller model | All expert weights still need to reside in memory or be offloaded; expert parallelism adds All-to-All communication | | 1.2-trillion-token pretraining | Broader linguistic, visual, and acoustic coverage | Higher training cost and greater licensing, governance, and decontamination burden | | Direct long-video input | Preserves global context | Frames, audio chunks, and KV Cache grow rapidly, increasing time to first token |

Weight memory can be approximated as M_weight ≈ P_total × b, where P_total is total parameter count and b is bytes per parameter. In BF16, raw weights for a 9B model are about 18GB and for a 35B model about 70GB. A 4-bit estimate is roughly 9GB and 17.5GB, respectively, before quantization metadata, encoders, activations, and runtime overhead. For a MoE model, fewer active parameters mainly reduce matrix computation; they do not automatically reduce weight-storage requirements.

For grouped-query attention, per-sequence KV Cache can be approximated as M_KV ≈ 2 × L × S × H_kv × d_head × b, where L is the number of layers and S is the total token count. Raising video resolution, frame rate, or audio segmentation increases S quickly. Production inference should therefore evaluate Paged KV Cache, chunked prefill, and visual-token compression rather than looking only at parameter count.

Performance Claims and Evaluation Gaps

The paper reports that WeMM-Video-35B-A3B establishes a new open-source state of the art for audiovisual understanding and reasoning, substantially outperforming Qwen3.5-Omni-Flash and Seed2.0-Mini-0428 while narrowing the gap to Gemini-3.0-Flash. It also says the 9B model surpasses Qwen3-Omni-30B-A3B-Instruct. The 9B-versus-30B-A3B result is particularly interesting, but it cannot be explained through total parameters alone: the larger model may also activate only about 3B parameters, while differences in architecture, memory use, visual encoders, and serving stacks can dominate deployment cost.

No absolute scores, run-to-run variance, confidence intervals, benchmark versions, prompt templates, decoding parameters, thinking budgets, or judge models are provided. Closed baselines are especially difficult to reproduce if they are accessed through changing APIs or different inference allowances.

A stronger reproduction should lock frame rate, resolution, audio sample rate, maximum video duration, and context window, then add four ablations: audio only, video only, speech with subtitles removed, and temporally shuffled audiovisual pairs. Temporal localization, weak-alignment negatives, long-video tasks, and multi-speaker dialogues would further reduce the risk of solutions based on subtitles, language priors, or isolated frames.

Developer Adoption and Deployment

  1. Select by memory and serving constraints, not leaderboard position: Test 2B, 4B, and quantized 9B models on a single workstation or GPU first. A 35B-A3B deployment must reserve space for all expert weights, the KV Cache, and multimodal buffers. CPU or NVMe expert offloading may erase the latency benefit of sparse activation.
  2. Lock audiovisual preprocessing: Standardize timestamps, frame rate, audio sample rate, and channel handling. ASR-only pipelines should be treated as a separate system, because they do not test native audio understanding.
  3. Verify runtime compatibility first: Confirm support for the exact architecture, processor, multimodal templates, Flash Attention, quantization, and tensor parallelism in Transformers, vLLM, or SGLang. For A3B, also test expert-parallel topology and All-to-All bandwidth requirements.
  4. Measure quality and cost together: Track task accuracy, audiovisual grounding errors, time to first token, end-to-end latency, throughput, peak memory, and cost per task. Run all baselines on matching hardware, precision, and decoding settings.
  5. Use an external mode router: Send routine questions to non-thinking mode and reserve thinking mode for difficult audiovisual reasoning. Enforce maximum video duration, segment retrieval, timestamp validation, and low-confidence fallback to limit the impact of long-context localization errors.
  6. Check licenses and reproducibility before deployment: Public weights do not remove ambiguity over usage rights. Verify the licenses of model weights, training code, evaluation scripts, and datasets, and confirm that the reproduction pipeline can rebuild the principal reported results.

Engineering Assessment

WeMM-Video's main contribution is an open model family spanning several deployment sizes, a dual reasoning interface, and an evaluation and reproduction pipeline rather than a checkpoint-only release. Its reported open-source SOTA and the claimed efficiency of the 9B model merit focused replication. At present, however, absolute results, internal architecture, license terms, and production performance remain unavailable; the defensible conclusion is that the project offers a promising new open baseline for audiovisual understanding, not a fully validated absolute leader.


Source: Hugging Face Daily Papers (2026-11-19)

— END —

Tags: #WeMM-Video #Omni-Modal Video Understanding #Hybrid Reasoning #Sparse MoE #Open-Source Multimodal Models

Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.

What's your take on this story?
🚀 Explore & Submit Works

Community Comments

Loading live comments and annotations…