Skip to main content
ZICQ

Wiki Concepts

FlashAttention

Concepts
Aliases: Flash Attention FlashAttention-2 FlashAttention-3 ·2026-10-10

FlashAttention

FlashAttention is an exact, IO-aware attention algorithm engineered around GPU memory hierarchies. Maintaining strict numerical equivalence without precision loss, FlashAttention leverages SRAM tiling, online softmax normalization, and backward pass recomputation to bypass the memory bandwidth wall of reading and writing intermediate $O(N^2)$ attention matrices. It delivers a $2\times \sim 5\times$ speedup in Large Language Model (LLM) training and inference while reducing memory consumption to linear $O(N)$.

Introduced by Tri Dao and colleagues in 2022, FlashAttention has become the industry standard for modern Transformer architectures.


Hardware Root Cause: The GPU Memory Bandwidth Wall

Within modern GPU architectures (e.g., NVIDIA A100 / H100), memory structures exhibit severe disparities between bandwidth and capacity:

┌────────────────────────────────────────────────────────┐
│ On-Chip SRAM (Shared Memory / L1 Cache)                │
│ Extremely Small (~228 KB / SM), Ultra-Fast (~19 TB/s)  │
└───────────────────────────▲────────────────────────────┘
                            │  Frequent round-trips cause memory bottleneck
┌───────────────────────────▼────────────────────────────┐
│ High Bandwidth Memory HBM (Main GPU DRAM)              │
│ Massive Capacity (80 GB ~ 140 GB), Lower Bandwidth (~3.35 TB/s)
└────────────────────────────────────────────────────────┘

The $O(N^2)$ Memory Bottleneck of Standard Attention

Standard PyTorch attention executes through three decoupled CUDA kernels:

$$S = \frac{QK^T}{\sqrt{d_k}} \in \mathbb{R}^{N \times N} \quad (\text{Materialized and written to HBM})$$

$$P = \text{softmax}(S) \in \mathbb{R}^{N \times N} \quad (\text{Read from HBM, normalized, written back to HBM})$$

$$O = PV \in \mathbb{R}^{N \times d} \quad (\text{Read } P \text{ and } V \text{ from HBM, output written to HBM})$$

  • When sequence lengths expand to $N=8,192$ or $64,000$, materializing intermediate matrix $P \in \mathbb{R}^{N \times N}$ scales quadratically (a single attention head at $N=64\text{K}$ consumes ~8GB of VRAM);
  • GPU Tensor Cores sit underutilized, bottlenecked by memory traffic moving data across the HBM bus (Memory-Bound).

Core Algorithmic Innovations

FlashAttention restructures the attention computational graph:

               FlashAttention: Complete Pipeline Enclosed in On-Chip SRAM
┌─────────────────────────────────────────────────────────────────────────┐
│ Stream sub-blocks Q_i, K_j, V_j from HBM into SRAM (~100 to 200 KB)     │
│                                                                         │
│  [SRAM Inner Pipeline]                                                  │
│  1. Compute local matrix block product S_ij = Q_i * (K_j)^T             │
│  2. Online Softmax update: m_new = max(m_old, m_ij), update scale l     │
│  3. Rescale and accumulate to O_i: O_i = diag(e^(m_old - m_new)) * O_i..│
│                                                                         │
│ After iteration completes: Write final O_i (shape N x d) back to HBM    │
└─────────────────────────────────────────────────────────────────────────┘
(Entire computation executes within SRAM; HBM I/O drops from O(N^2) to O(N))

1. Tiling

FlashAttention divides sequence matrices $Q, K, V$ into blocks sized to fit within GPU on-chip SRAM. By orchestrating nested loops over blocks, all matrix multiplications and intermediate softmax updates occur directly inside fast shared memory. Only the finalized $N \times d$ output is written back to HBM, eliminating the physical need to store the $N \times N$ attention matrix.

2. Online Softmax Normalization

Standard softmax requires finding the row-wise maximum $m = \max(x)$ to prevent exponential numerical overflow, followed by computing the denominator sum $\ell = \sum e^{x_i - m}$. Tiling initially seems infeasible because the global maximum is unknown until the entire row is observed.

FlashAttention utilizes an online incremental softmax recurrence relation:

Let $m^{(1)}$ be the maximum of the prior block, $\ell^{(1)}$ the accumulated denominator sum, and $O^{(1)}$ the un-normalized output accumulation. When a new block arrives with local maximum $m^{(2)}$ and local sum $\ell^{(2)}$, the global statistics update dynamically:

$$m^{\text{new}} = \max\left(m^{(1)}, m^{(2)}\right)$$

The global denominator sum scales through an exponential normalization factor:

$$\ell^{\text{new}} = e^{m^{(1)} - m^{\text{new}}} \ell^{(1)} + e^{m^{(2)} - m^{\text{new}}} \ell^{(2)}$$

The intermediate output accumulates through dynamic reweighting:

$$O^{\text{new}} = \text{diag}\left(e^{m^{(1)} - m^{\text{new}}}\right) O^{(1)} + e^{m^{(2)} - m^{\text{new}}} P^{(2)} V^{(2)}$$

Once block traversal completes, a final scaling $O^{\text{final}} = \text{diag}\left(\frac{1}{\ell^{\text{final}}}\right) O^{\text{new}}$ yields the exact mathematical output of standard attention without saving intermediate states.

3. Recomputation in the Backward Pass

In standard backpropagation, the optimizer retains the $N \times N$ attention matrix $P$ to compute gradients with respect to $Q, K, V$. FlashAttention retains only compact row summary statistics in HBM (the max $m$ and sum $\ell$, occupying linear $O(N)$ space). During the backward pass, it rapidly recomputes attention blocks on-the-fly inside SRAM. It trades inexpensive GPU FLOPs for scarce memory bandwidth and capacity.


Version Evolution: v1 to v3

Version Key Breakthrough Architecture Target FLOPs Utilization (A100/H100)
FlashAttention-1 (2022) Established Tiling, Online Softmax, and Recomputation NVIDIA Ampere (A100) $\sim 30% - 45%$ Theoretical Peak
FlashAttention-2 (2023) Outer loop over $Q$ blocks, inner loop over $K, V$; eliminated thread block synchronization Ampere / Ada Lovelace $\sim 50% - 73%$ Theoretical Peak
FlashAttention-3 (2024) Hopper optimizations: Asynchronous TMA copies, FP8 low-precision, Warp-Group instruction overlapping NVIDIA Hopper (H100/H800) $\sim 75% - 85%$ (Exceeding 1.2 PFLOPs/s)

FlashAttention-3 exploits architectural primitives on Hopper GPUs: it uses the Tensor Memory Accelerator (TMA) for asynchronous global-to-shared memory transfers and overlaps GEMM tensor core executions with softmax calculations across specialized warp groups.


Production Deployment and Frameworks

FlashAttention is natively integrated across production ecosystems:

  1. PyTorch 2.x SDPA: Built-in torch.nn.functional.scaled_dot_product_attention automatically routes execution to FlashAttention when hardware constraints allow.
  2. Hugging Face Transformers: Enabled simply by specifying the implementation flag:
    model = AutoModelForCausalLM.from_pretrained(
        "meta-llama/Llama-3-70B",
        torch_dtype=torch.bfloat16,
        attn_implementation="flash_attention_2"
    )
    
  3. Inference Engines: vLLM, TensorRT-LLM, and SGLang deploy FlashAttention as the default kernel for prefill attention computation.

Hardware Prerequisites

  • Requires GPU architectures with Compute Capability $\ge 8.0$ (NVIDIA A100, RTX 3090/4090, H100);
  • Older architectures (such as Volta V100, SM 7.0) must fall back to Memory-Efficient Attention kernels (xFormers).

Sources and Key Literature