Skip to main content
ZICQ

Wiki AI Concepts

Transformer Architecture

AI Concepts
Aliases: transformer self-attention model ·2026-10-10

Transformer Architecture

The Transformer is a landmark neural network architecture introduced by Google in the 2017 paper Attention Is All You Need. By replacing recurrent neural networks (RNNs) and convolutions with self-attention, it fundamentally redefined sequence modeling and serves as the cornerstone of modern large language models (LLMs), multimodal foundation models, and AI systems.

Core Mechanism: Scaled Dot-Product & Multi-Head Attention

The foundation of the Transformer is the self-attention mechanism. Given an input sequence mapped to Queries ($Q$), Keys ($K$), and Values ($V$), the scaled dot-product attention is formulated as:

$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$

Why scale by $\sqrt{d_k}$?

If the elements of $Q$ and $K$ are independent random variables with zero mean and unit variance, their dot product has a variance of $d_k$. For large dimensions $d_k$, the dot-product values grow large, pushing the Softmax function into regions with near-zero gradients (saturation). The scale factor $\frac{1}{\sqrt{d_k}}$ normalizes the variance back to 1, ensuring stable gradient flow during early training.

Multi-Head Attention (MHA)

Rather than performing a single attention function, Multi-Head Attention linearly projects $Q, K, V$ into $h$ different lower-dimensional representation subspaces:

$$\text{MHA}(Q, K, V) = \text{Concat}(\text{head}_1, \dots, \text{head}_h)W^O$$

This mechanism allows the model to jointly attend to information from different representation subspaces at different sequence positions (e.g., syntactical relationships vs. coreference resolution).

Evolution of Positional Encodings

Because self-attention is intrinsically permutation-invariant over token positions, order information must be injected:

  1. Sinusoidal Positional Encoding: Fixed sine and cosine functions of varying frequencies directly added to embeddings (original paper).
  2. Learned Positional Embeddings: Trainable vectors per position (e.g., BERT, GPT-2), but struggling to extrapolate beyond context length seen in training.
  3. Rotary Position Embedding (RoPE): The de facto standard in modern LLMs (LLaMA, DeepSeek, Qwen). RoPE rotates 2D feature chunks in the complex plane, so the dot product naturally encapsulates relative token distance, enabling strong context extension.
  4. ALiBi (Attention with Linear Biases): Injects linear distance penalties directly into attention scores, offering robust zero-shot length extrapolation.

Computational Complexity & Engineering Bottlenecks

Standard self-attention requires $O(N^2)$ compute and memory with respect to sequence length $N$. In industrial deployment, this poses two critical challenges:

  • Autoregressive Inference & Memory Limits: During generation, recomputing Key/Value tensors for past tokens is avoided via KV Cache. However, large context windows and high concurrency cause KV Cache memory to rapidly dominate GPU memory.
  • Hardware-Aware IO Optimization: Conventional attention implementations are memory-bandwidth-bound due to frequent reads/writes to GPU High Bandwidth Memory (HBM). FlashAttention restructures computation via SRAM tiling and online Softmax, delivering substantial speedups without compromising numerical precision.

Architectural Convergence of Modern LLMs

Over years of empirical refinement, frontier open-source and proprietary LLMs have largely converged on:

  • Topology: Decoder-only autoregressive causal architectures.
  • Normalization: Pre-LayerNorm topology with Root Mean Square Normalization (RMSNorm) instead of standard LayerNorm.
  • Activation Functions: Gated Linear Units such as SwiGLU replacing traditional ReLU/GELU.
  • Sparse Scaling: Routing inputs via Mixture of Experts (MoE) to expand total parameter capacity while maintaining bounded active compute per token.

Further Reading