DeepSeek
DeepSeek is a foundation AI research lab and tech company based in Hangzhou, China. Between 2024 and 2025, DeepSeek disrupted the global artificial intelligence landscape with the releases of DeepSeek-V3 and DeepSeek-R1 — matching or exceeding proprietary frontier models (GPT-4o, Claude 3.5 Sonnet, and OpenAI o1) at a fraction of the computational training budget.
Core Models & Product Family
- DeepSeek-V3: A 671B total parameter Mixture of Experts (MoE) foundation model activating 37B parameters per token, ranking at the top tier of open-source general language understanding and coding.
- DeepSeek-R1: A state-of-the-art Reasoning Model trained with large-scale reinforcement learning, demonstrating elite performance on mathematics (AIME, MATH-500) and competitive programming.
- DeepSeek-R1-Distill Series: Open-weight models distilled from R1 into lightweight form factors (1.5B to 70B, based on Qwen and Llama architectures), democratizing reasoning capabilities on local and edge hardware.
Key Architectural Innovations
1. Multi-Head Latent Attention (MLA)
Standard Multi-Head Attention (MHA) and Grouped-Query Attention (GQA) store high-dimensional Key and Value projections for all context positions in GPU memory during generation, causing memory constraints via the KV Cache.
MLA introduces low-rank joint compression:
- Projects Keys and Values into a single compact latent vector ($c_t^{KV}$);
- During autoregressive inference, only this small latent vector needs to be cached;
- Computes attention by dynamically up-projecting the latent vectors and coupling with decoupled RoPE position components.
Engineering Impact: MLA slashes KV Cache memory consumption to roughly 1/6th of standard MHA, far surpassing GQA while retaining full representational capacity. This enables high-concurrency long-context serving on standard 8x GPU nodes.
2. Fine-Grained DeepSeekMoE Architecture
Instead of activating a small number of coarse experts (like Mixtral 8x7B), DeepSeek subdivides parameters into finer expert grains:
- Segments layers into many smaller experts (e.g., 64 routed experts, activating 8 per token);
- Dedicates fixed Shared Experts that are always active to capture general foundational representations. This design reduces expert redundancy, mitigates load imbalance, and maximizes parameter utilization efficiency.
3. Multi-Token Prediction (MTP)
Sequentially adds speculative prediction heads atop the model to forecast consecutive tokens ($t+1, t+2$):
- Provides denser gradient feedback throughout pre-training;
- Serves as an integrated validator for speculative decoding during inference, speeding up throughput without auxiliary models.
Infrastructure & Communication Co-Design
DeepSeek's low cost is achieved through rigorous end-to-end hardware-software co-design:
- DualPipe Bidirectional Pipeline Parallelism: MoE cross-node dispatch entails heavy All-to-All communication overhead. DualPipe interleaves and overlaps forward and backward execution to hide inter-node transfer delays, reducing GPU idle bubbles to near zero.
- Custom DeepEP Communication Library: Highly tuned low-latency communication kernels specifically built for expert routing across NVLink node topologies and InfiniBand/RoCE clusters.
- End-to-End FP8 Mixed-Precision Training: Pioneered stable, large-scale FP8 training across thousands of GPUs using fine-grained block scaling (128-element tiles) and delayed scaling factor caching, doubling computing throughput and cutting memory bandwidth demands.
Reinforcement Learning Paradigm of R1
With DeepSeek-R1-Zero, the team demonstrated that complex reasoning abilities can emerge purely through reinforcement learning with verifiable rewards (RLVR), without requiring initial human-annotated Chain-of-Thought (CoT). The model organically developed self-reflection, alternative search exploration, and error correction. With a minimal cold-start dataset and multi-stage rejection sampling, DeepSeek-R1 established a major milestone for open science.
Further Reading
- DeepSeek-V3 Technical Report — Full architecture and training infrastructure
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning — R1 RL training methodology
- DeepSeek-MoE: Towards Ultimate Expertise in Mixture-of-Experts Language Models — Fine-grained MoE paper