Boson AI Launches 45B Higgs Realtime Duplex for Simultaneous Voice, Vision, Reasoning, and Tool Use
By Mr.Xu Community Post
Published:
Summary:Boson AI has unveiled Higgs Realtime Duplex, a 45B-parameter mixture-of-experts model that unifies audio, vision, text, reasoning, and tool execution into 160-millisecond streams. The model can continue reasoning and coordinating tools while it speaks, respond before staged tool results are complete, and delegate difficult subtasks to a larger backend model with shared context. The system supports zero-shot voice and avatar personalization, uses multi-teacher on-policy distillation and GRPO-bas
Core Release: A 45B MoE for Multitasking Real-Time Interaction
Boson AI has released Higgs Realtime Duplex, a 45B-parameter mixture-of-experts model designed for full-duplex multimodal interaction. Instead of following a strictly serial pattern—listen until the user finishes, wait for a turn boundary, reason, and then speak—the model processes audio, speech output, video, avatar output, text, reasoning, and tool execution concurrently.
Every signal is segmented into 160-millisecond chunks. Audio input consumes one token per chunk, while audio output consumes two. A camera frame of roughly 12 tokens enters every six chunks, and avatar generation produces two latents per chunk. Text runs alongside these streams. Silence is represented explicitly as chunks rather than as an absence of input, allowing the model to decide when to backchannel, interrupt, or remain quiet.
This is not a conventional voice model with a dialogue controller attached. Interaction timing itself is trained as part of the model. The weights are not open, and the Duplex API is currently served through a waitlist, although the earlier turn-based Higgs Realtime model is available.
How the Unified Stream Enables Full-Duplex Behavior
Real-time interaction depends on more than low time-to-first-token. The model must continue consuming input, reasoning, and coordinating tools while it is generating speech. Higgs Realtime Duplex represents reasoning, tool calls, and delegation as spans in the same event stream, with a span capable of spanning many 160-millisecond chunks.
Tool results can arrive in stages, allowing the model to begin speaking before every result is available. For more difficult tasks, the 45B foreground model delegates to a larger backend model with access to the shared context. This differs from architectures that move every reasoning and tool operation into a background model: the latter can introduce response delays exceeding 10 seconds and make cost attribution harder.
Boson estimates that audio and video consume approximately 2.6K tokens per minute, rising to roughly 2.6K—5K tokens per minute when reasoning and tool traces are included. The last five minutes, together with compacted text, may occupy 13K—25K tokens, while older material is retained as text. The central engineering question is therefore not simply token consumption, but which details must remain available at fine temporal resolution.
Voice and Avatar Personalization
Deployment requires no fine-tuning. A short voice reference clip, one avatar image, and a textual behavior prompt are sufficient. Audio is produced by a causal flow-matching decoder that emits two frames per chunk. The avatar is generated by a causal DiT conditioned on the reference image and the previous two latents.
This lowers integration friction, but it does not make real-time generation computationally trivial. The system must continuously execute flow-matching and DiT inference while coordinating the MoE, visual encoding, and tool execution within a tight latency budget. Long-term identity consistency, reference-noise amplification, and drift under unusual lighting still require production testing.
Training: Reinforcement-Learning Teachers, Distillation, and Simulated Self-Improvement
The model begins with a text-pretrained LLM and continues pretraining on approximately 2T tokens of text, audio, and video using an in-house H100 cluster and a Megatron-based stack. Post-training uses multiple specialist teachers trained with reinforcement learning, including GRPO, for dialogue, interaction timing, delegation, and task completion. Their on-policy traces are distilled into a single student.
The student then enters simulated environments with realistic timing, interruptions, overlapping speech, and background noise for self-improvement. This makes interaction policy trainable rather than hard-coded. The limitation is distribution mismatch: simulated conversations do not reproduce every aspect of real meetings, customer support, or robot interaction, and distillation cannot reliably exceed the capabilities of its teachers.
Benchmarks: Competitive Overall, Weaker at Turn-Taking
Boson compares Higgs Realtime Duplex with the best published or reproduced configurations from other laboratories and says it reproduced more than half of the competing results. The model reaches 98.9 on Big Bench Audio, above TML Interaction Small at 96.5 and OpenAI's best configuration at 97. On ProactiveVideoQA, it records a PAUC score of 31.9, close to TML's 33.5 and above OpenAI's 25.0.
The model is not the leader across all seven listed benchmarks. OpenAI's best configuration leads five of them by approximately one to five points. Higgs Realtime Duplex scores 75.5 on Full-Duplex-Bench v1.5, below TML's 77.8 and OpenAI's 80.1. Its 69.8 on τ-Voice also trails OpenAI's 74.5. Boson explicitly identifies turn-taking as its largest remaining gap.
The comparison requires caution. Reasoning budgets, tool configurations, and inference costs are not fully aligned. Some competitor results were reproduced by Boson, and aggregate benchmark scores do not expose streaming first-response latency, jitter, tool-failure rates, or cost per minute. Production validation should additionally measure time to the first perceivable response, p95 output jitter, false and missed interruptions, and answer correctness when tool results are incomplete.
Engineering Trade-offs
Boson deliberately uses a medium-sized 45B foreground model, with a stated target of keeping one hour of model time below roughly one-tenth of an hour of human time. A smaller foreground model reduces recurring inference cost, while selective delegation to a larger backend model reserves expensive reasoning for difficult tasks.
This depends on accurate routing. Too little delegation can slow responses; too much erodes the intended cost advantage. Multimodal token usage also grows with camera sampling frequency and tool traces. Deployments need explicit limits on video frequency and fine-grained history.
Staged tool results create another risk: the foreground model may speak before information is complete, requiring policies that distinguish provisional progress from final claims. External tool and backend latency may also fall out of phase with the 160-millisecond cadence. Closed weights and waitlist-only access currently limit independent benchmarking and increase vendor-lock-in concerns.
Deployment Guidance
Developers can use the available turn-based Higgs Realtime to validate dialogue and tool workflows while waiting for Duplex access. A production integration should expose audio, video, text, tool, and reasoning events through separate channels, with application state covering listening, speaking, waiting for tools, and permission to backchannel. Business logic should not wait for an entire completed response.
Priority metrics should include first perceptible feedback latency, p95 streaming jitter, false and missed interruption rates, and correctness while tool results are pending. Cost accounting should separate foreground tokens, visual frames, reasoning traces, and delegated backend tokens so that long sessions do not hide unbounded consumption.
Higgs Realtime Duplex's main advance is the conversion of duplex interaction, background reasoning, and tool execution into a unified event stream. Its benchmark results also demonstrate that performing many activities concurrently does not automatically solve the harder problem of deciding when to speak, interrupt, or remain silent. That depends on training data, simulation coverage, and real-time evaluation discipline.
— END —Source: Reddit r/MachineLearning (2026-11-06)
Tags: #Full-Duplex Voice Model #Mixture-of-Experts #Realtime Multimodal AI #Tool Use #Streaming Inference
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments