Open-XTTS-Trainer Released: Enabling Multilingual Expansion and Full XTTS v2 Pipeline Training

Summary:Amit Bhakt has open-sourced Open-XTTS-Trainer, a modular training suite under the MPL-2.0 license designed to address the training challenges of the XTTS v2 model following Coqui AI's shutdown. The tool supports the full training pipeline for the DVAE audio tokenizer, GPT text extender, and Latent HiFi-GAN vocoder, while resolving issues with multilingual script support. Key features include compatibility with modern PyTorch versions, GPU training convergence verification, and a one-click Colab

---

Background and Motivation

Following the shutdown of Coqui AI, the XTTS v2 model lost official training code support, particularly for the DVAE audio tokenizer and Latent HiFi-GAN vocoder. Additionally, XTTS v2 faced issues with handling new language scripts (e.g., Arabic, Hindi) due to its fixed vocabulary and hardcoded design, leading to out-of-vocabulary errors or matrix dimension crashes. Amit Bhakt's Open-XTTS-Trainer aims to fill this gap by providing a complete training toolchain for developers.

Key Features and Technical Details

1. Modular Design

   • Stage 1 (DVAE Audio Tokenizer): Offers a full training and fine-tuning loop, including Gumbel-Softmax codebook annealing and mel-spectrogram reconstruction loss.
   • Stage 2 (GPT Text Extender): Scans new language corpora, identifies unique Unicode characters, and dynamically adjusts the GPT embedding matrix while preserving existing weights.
   • Stage 3 (Latent HiFi-GAN Vocoder): Employs Multi-Period (MPD) and Multi-Scale (MSD) discriminators and feature-matching loss for latent-conditioned vocoder training.

2. Compatibility Improvements

   • Addresses dependency rot by supporting seamless operation on modern PyTorch (3.10-3.13) versions without relying on torchcodec.

3. Performance Verification

   • Tested on an NVIDIA T4 GPU, the HiFi-GAN mel reconstruction loss decreased by 67% over 10 epochs (from 190.83 to 63.49).
   • The discriminator equilibrium stabilized at 3.62 with no mode collapse.
   • The DVAE mel error dropped by 67% (from 8.14 to 2.71).

Engineering Trade-offs and Challenges

• Multilingual Support: While XTTS v2 was initially designed for 17 languages, Open-XTTS-Trainer addresses the support for new Unicode scripts by dynamically extending the vocabulary and embedding matrix. However, this introduces additional training time and computational resource overhead.
• Dependency Management: Although compatibility with modern PyTorch versions enhances usability, it requires re-implementing certain functionalities to replace the deprecated torchcodec module.

Developer Deployment Recommendations

• Quick Start: Use the provided one-click Colab demo to quickly test and experiment with Open-XTTS-Trainer.
• Custom Extensions: Developers are encouraged to further optimize the architecture parameters of DVAE and HiFi-GAN to suit specific application scenarios.
• Community Engagement: Feedback, PRs, and participation in new language experiments are welcome through GitHub and Hugging Face platforms to collectively advance the project.

Conclusion

The release of Open-XTTS-Trainer fills the training and fine-tuning gap for the XTTS v2 model, providing a powerful tool for multilingual TTS applications. Its open-source nature and modular design make it a valuable resource for researchers and developers.

---

Source: Reddit r/MachineLearning (2026-10-22)

— END —

Tags: #Open-XTTS-Trainer Follow #XTTS v2 Follow #Multilingual TTS Follow #Open Source Model Follow #PyTorch Follow