Whisper
Whisper is OpenAI's open-source automatic speech recognition (ASR) model released in 2022. With its multilingual robustness and open-source availability, it quickly became the industry ASR de facto standard.
Model specifications
| Size | Params | VRAM | Relative speed |
|---|---|---|---|
| tiny | 39M | ~1 GB | ~32x |
| base | 74M | ~1 GB | ~16x |
| small | 244M | ~2 GB | ~6x |
| medium | 769M | ~5 GB | ~2x |
| large-v3 | 1.5B | ~10 GB | 1x |
Core capabilities
- Multilingual recognition: 99 languages, WER near human level on major languages.
- Translation: directly translate foreign speech to English text.
- Timestamps: word-level / segment-level timestamp output.
- Long audio: chunked processing supports arbitrary length.
- Robustness: much better than traditional Kaldi systems on accents, noise, phone audio.
One-line startup
pip install openai-whisper
whisper audio.mp3 --language en --model medium
Chinese experience
- Chinese recognition accuracy is very high (medium+ is good enough).
- Numbers / proper nouns occasionally wrong, needs post-processing correction.
- Speaker diarization needs additional tools (pyannote.audio).
Deployment
- Local CLI: the
whispercommand above, simplest. - Python library:
whisper.load_model("large-v3"), embed in apps. - whisper.cpp: C++ rewrite, runs on CPU / Apple Silicon / Android.
- faster-whisper: CTranslate2 backend, 4-5x faster, half the memory.
- whisper-jax: JAX backend, TPU acceleration.
Limitations
- No built-in speaker diarization.
- Real-time streaming needs custom wrapper (
whisper_streamingetc). - Long silence / song change scenarios may produce hallucinated text.