FlashAttention
FlashAttention is an exact, IO-aware attention algorithm engineered around GPU memory hierarchies. Maintaining strict numerical equivalence without precision loss, FlashAttention leverages SRAM tiling, online softmax normalization, and backward pass recomputation to bypass the memory bandwidth wall of reading and writing intermediate $O(N^2)$ attention matrices. It delivers a $2\times \sim 5\times$ speedup in Large Language Model (LLM) training and inference while reducing memory consumption to linear $O(N)$.
Introduced by Tri Dao and colleagues in 2022, FlashAttention has become the industry standard for modern Transformer architectures.
Hardware Root Cause: The GPU Memory Bandwidth Wall
Within modern GPU architectures (e.g., NVIDIA A100 / H100), memory structures exhibit severe disparities between bandwidth and capacity:
┌────────────────────────────────────────────────────────┐
│ On-Chip SRAM (Shared Memory / L1 Cache) │
│ Extremely Small (~228 KB / SM), Ultra-Fast (~19 TB/s) │
└───────────────────────────▲────────────────────────────┘
│ Frequent round-trips cause memory bottleneck
┌───────────────────────────▼────────────────────────────┐
│ High Bandwidth Memory HBM (Main GPU DRAM) │
│ Massive Capacity (80 GB ~ 140 GB), Lower Bandwidth (~3.35 TB/s)
└────────────────────────────────────────────────────────┘
The $O(N^2)$ Memory Bottleneck of Standard Attention
Standard PyTorch attention executes through three decoupled CUDA kernels:
$$S = \frac{QK^T}{\sqrt{d_k}} \in \mathbb{R}^{N \times N} \quad (\text{Materialized and written to HBM})$$
$$P = \text{softmax}(S) \in \mathbb{R}^{N \times N} \quad (\text{Read from HBM, normalized, written back to HBM})$$
$$O = PV \in \mathbb{R}^{N \times d} \quad (\text{Read } P \text{ and } V \text{ from HBM, output written to HBM})$$
- When sequence lengths expand to $N=8,192$ or $64,000$, materializing intermediate matrix $P \in \mathbb{R}^{N \times N}$ scales quadratically (a single attention head at $N=64\text{K}$ consumes ~8GB of VRAM);
- GPU Tensor Cores sit underutilized, bottlenecked by memory traffic moving data across the HBM bus (Memory-Bound).
Core Algorithmic Innovations
FlashAttention restructures the attention computational graph:
FlashAttention: Complete Pipeline Enclosed in On-Chip SRAM
┌─────────────────────────────────────────────────────────────────────────┐
│ Stream sub-blocks Q_i, K_j, V_j from HBM into SRAM (~100 to 200 KB) │
│ │
│ [SRAM Inner Pipeline] │
│ 1. Compute local matrix block product S_ij = Q_i * (K_j)^T │
│ 2. Online Softmax update: m_new = max(m_old, m_ij), update scale l │
│ 3. Rescale and accumulate to O_i: O_i = diag(e^(m_old - m_new)) * O_i..│
│ │
│ After iteration completes: Write final O_i (shape N x d) back to HBM │
└─────────────────────────────────────────────────────────────────────────┘
(Entire computation executes within SRAM; HBM I/O drops from O(N^2) to O(N))
1. Tiling
FlashAttention divides sequence matrices $Q, K, V$ into blocks sized to fit within GPU on-chip SRAM. By orchestrating nested loops over blocks, all matrix multiplications and intermediate softmax updates occur directly inside fast shared memory. Only the finalized $N \times d$ output is written back to HBM, eliminating the physical need to store the $N \times N$ attention matrix.
2. Online Softmax Normalization
Standard softmax requires finding the row-wise maximum $m = \max(x)$ to prevent exponential numerical overflow, followed by computing the denominator sum $\ell = \sum e^{x_i - m}$. Tiling initially seems infeasible because the global maximum is unknown until the entire row is observed.
FlashAttention utilizes an online incremental softmax recurrence relation:
Let $m^{(1)}$ be the maximum of the prior block, $\ell^{(1)}$ the accumulated denominator sum, and $O^{(1)}$ the un-normalized output accumulation. When a new block arrives with local maximum $m^{(2)}$ and local sum $\ell^{(2)}$, the global statistics update dynamically:
$$m^{\text{new}} = \max\left(m^{(1)}, m^{(2)}\right)$$
The global denominator sum scales through an exponential normalization factor:
$$\ell^{\text{new}} = e^{m^{(1)} - m^{\text{new}}} \ell^{(1)} + e^{m^{(2)} - m^{\text{new}}} \ell^{(2)}$$
The intermediate output accumulates through dynamic reweighting:
$$O^{\text{new}} = \text{diag}\left(e^{m^{(1)} - m^{\text{new}}}\right) O^{(1)} + e^{m^{(2)} - m^{\text{new}}} P^{(2)} V^{(2)}$$
Once block traversal completes, a final scaling $O^{\text{final}} = \text{diag}\left(\frac{1}{\ell^{\text{final}}}\right) O^{\text{new}}$ yields the exact mathematical output of standard attention without saving intermediate states.
3. Recomputation in the Backward Pass
In standard backpropagation, the optimizer retains the $N \times N$ attention matrix $P$ to compute gradients with respect to $Q, K, V$. FlashAttention retains only compact row summary statistics in HBM (the max $m$ and sum $\ell$, occupying linear $O(N)$ space). During the backward pass, it rapidly recomputes attention blocks on-the-fly inside SRAM. It trades inexpensive GPU FLOPs for scarce memory bandwidth and capacity.
Version Evolution: v1 to v3
| Version | Key Breakthrough | Architecture Target | FLOPs Utilization (A100/H100) |
|---|---|---|---|
| FlashAttention-1 (2022) | Established Tiling, Online Softmax, and Recomputation | NVIDIA Ampere (A100) | $\sim 30% - 45%$ Theoretical Peak |
| FlashAttention-2 (2023) | Outer loop over $Q$ blocks, inner loop over $K, V$; eliminated thread block synchronization | Ampere / Ada Lovelace | $\sim 50% - 73%$ Theoretical Peak |
| FlashAttention-3 (2024) | Hopper optimizations: Asynchronous TMA copies, FP8 low-precision, Warp-Group instruction overlapping | NVIDIA Hopper (H100/H800) | $\sim 75% - 85%$ (Exceeding 1.2 PFLOPs/s) |
FlashAttention-3 exploits architectural primitives on Hopper GPUs: it uses the Tensor Memory Accelerator (TMA) for asynchronous global-to-shared memory transfers and overlaps GEMM tensor core executions with softmax calculations across specialized warp groups.
Production Deployment and Frameworks
FlashAttention is natively integrated across production ecosystems:
- PyTorch 2.x SDPA: Built-in
torch.nn.functional.scaled_dot_product_attentionautomatically routes execution to FlashAttention when hardware constraints allow. - Hugging Face Transformers: Enabled simply by specifying the implementation flag:
model = AutoModelForCausalLM.from_pretrained( "meta-llama/Llama-3-70B", torch_dtype=torch.bfloat16, attn_implementation="flash_attention_2" ) - Inference Engines: vLLM, TensorRT-LLM, and SGLang deploy FlashAttention as the default kernel for prefill attention computation.
Hardware Prerequisites
- Requires GPU architectures with Compute Capability $\ge 8.0$ (NVIDIA A100, RTX 3090/4090, H100);
- Older architectures (such as Volta V100, SM 7.0) must fall back to Memory-Efficient Attention kernels (xFormers).
Sources and Key Literature
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (Dao et al., NeurIPS 2022) — Original foundational paper.
- FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning (Dao, ICLR 2024) — Advanced parallelization and work partitioning.
- FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-Precision (Shah et al., 2024) — Architecture paper covering Hopper H100 and FP8 optimizations.
- Dao AILab FlashAttention GitHub Repository — Official repository and CUDA kernel implementations.