Groq is an AI inference hardware company and cloud serving platform best known for its LPU (Language Processing Unit) architecture. By designing silicon and compilers specifically around deterministic sequential execution and high-bandwidth on-chip SRAM, Groq delivers exceptional tokens-per-second streaming throughput for real-time voice assistants, live agents, and System 1 fast-decision pipelines.
Why LPUs Excel at Autoregressive Decoding
During small-batch LLM token generation, standard GPUs typically become memory-bandwidth-bound rather than compute-bound: generating each token requires streaming huge weight matrices repeatedly from external HBM memory into Tensor Cores.
Groq's LPU trades memory density for deterministic bandwidth:
- All on-chip SRAM storage instead of external HBM: On-chip SRAM offers dramatically higher bandwidth and far lower read latency than off-die HBM;
- Compiler-orchestrated deterministic execution: Unlike GPUs with hardware warp schedulers and cache hierarchies that can create tail-latency jitter, Groq compilers statically schedule every data movement and vector op down to exact clock cycles.
Architectural Trade-Off: The SRAM Capacity Ceiling
The inverse cost of on-chip SRAM speed is limited capacity per chip (hundreds of megabytes rather than 80–192 GB of HBM per GPU):
- Serving large 70B+ open-weight models requires interconnecting numerous LPU chips across nodes;
- GPUs regain total throughput efficiency when massive batch sizes saturate Tensor Core compute density, whereas LPUs dominate when minimal single-stream latency and instant token generation drive user experience.
Primary Workloads on GroqCloud
GroqCloud serves curated open-weight text models (Llama, Qwen, DeepSeek variants) and high-speed Whisper audio transcription via an OpenAI-compatible API:
- Voice agents & live subtitles: Keeping Whisper transcription plus short LLM turns well within conversational pause tolerances;
- Real-time guardrails, routing, and speculative/draft verification: Running sub-second structured classification before heavier reasoning paths trigger.
Further Reading
- GroqCloud Developer Documentation: https://console.groq.com/docs
- Groq Architecture & Website: https://groq.com/