Fish Audio Releases Drama 3: Open-Source Speech Generation Model for Multi-Character Scenes and Real-Time Audio Editing
Summary:Fish Audio has released Drama 3, an open-source speech generation model built on a 35B-parameter mixture-of-experts (MoE) backbone with 3B parameters activated per token. Drama 3 supports multi-character scene dubbing, real-time audio editing, and long-form audio synthesis, along with multi-turn revision and streaming text input. Its key innovations include a hybrid linear-attention backbone and a novel codebook multi-token-prediction audio decoder, achieving an inference speed of 271 frames per
Technical Architecture and Core Mechanisms
Fish Audio Drama 3 is built on a 35B-parameter mixture-of-experts (MoE) backbone, with 3B parameters activated per token. Its core mechanisms include:
- Hybrid Linear Attention Mechanism: This mechanism combines the computational efficiency of linear attention with the expressive power of multi-head attention, significantly reducing memory overhead and computational complexity.
- Codebook Multi-Token-Prediction Audio Decoder: The decoder predicts the acoustic codebooks of each audio frame and performs attention over the entire sequence in dedicated layers, enabling more accurate audio generation.
- Multi-Turn Revision and Real-Time Editing: Drama 3 supports real-time revision of audio content in multi-character scenes and can adjust previous audio outputs based on subsequent dialogue.
- Long-Form Audio Synthesis and Streaming Input: The model can handle long text inputs and generate coherent audio streams, supporting real-time interaction scenarios.
Engineering Trade-offs and Performance
Drama 3 demonstrates impressive inference speed, processing 271 frames per second on a single H200 GPU, compared to 177 frames per second for Fish Audio S2. Despite having more than seven times as many parameters as S2, the optimized architecture and efficient decoder design mitigate this gap.
However, the training cost of Drama 3 is high, requiring substantial computational resources and time. Additionally, its multi-character scene handling capability may still face challenges in complex scenarios, such as distinguishing between character voices and expressing emotions.
Developer Implementation and Deployment Recommendations
- Model Deployment: Developers can utilize the pre-trained model weights provided on GitHub and Hugging Face and deploy the model via SGLang-Omni for inference.
- Performance Optimization: It is recommended to employ quantization techniques or model pruning in resource-constrained environments to reduce memory usage and inference latency.
- Application Scenarios: Drama 3 is suitable for multi-character dubbing, virtual assistants, and audiobook production. Developers can combine it with other tools (such as speech recognition and natural language processing) to build more complex voice interaction systems.
Future Outlook
The release of Drama 3 marks a significant breakthrough in speech generation technology for multi-character scenes and real-time editing. As the model architecture is further optimized and training data is expanded, Drama 3 is expected to find applications in more domains, such as healthcare, education, and entertainment.
— END —Source: Hugging Face Daily Papers (2026-10-13)
Tags: #Fish Audio #Open-Source Model #Speech Generation #Multi-Character Scenes #Real-Time Audio Editing
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments