Skip to main content
ZICQ

Wiki Technical Terms

Low-Rank Adaptation (LoRA)

Technical Terms
Aliases: Low-Rank Adaptation LoRA PEFT ·2026-10-10

Low-Rank Adaptation (LoRA)

LoRA (Low-Rank Adaptation) is a foundational Parameter-Efficient Fine-Tuning (PEFT) methodology introduced by Microsoft in 2021. By freezing the pre-trained model weights and injecting trainable rank decomposition matrices into Transformer layers, LoRA enables models to match or exceed full-parameter fine-tuning quality on downstream tasks while updating only 0.1% to 1% of the network parameters.

Theoretical Foundation: The Intrinsic Rank Hypothesis

Deep neural networks are heavily over-parameterized. Theoretical studies establish that when adapting foundation models to specific tasks, weight update trajectories $\Delta W$ predominantly reside within a low-dimensional intrinsic rank subspace. Consequently, optimizing every dense parameter matrix during domain adaptation is mathematically redundant.

Mathematical Formulation & Forward Pass

Given a frozen pre-trained dense projection matrix $W_0 \in \mathbb{R}^{d \times k}$, LoRA decomposes the parameter update $\Delta W$ into the product of two low-rank matrices:

$$\Delta W = \frac{\alpha}{r} (B \cdot A)$$

where:

  • $A \in \mathbb{R}^{r \times k}$: Down-projection matrix initialized via random Gaussian distributions with zero mean;
  • $B \in \mathbb{R}^{d \times r}$: Up-projection matrix initialized to zeros;
  • $r \ll \min(d, k)$: The low rank, typically set to 4, 8, 16, 32, or 64;
  • $\alpha$: A scaling hyperparameter governing the magnitude of adapter updates.

During the forward computation, an input activation $x$ passes through both branches concurrently:

$$h = W_0 x + \Delta W x = W_0 x + \frac{\alpha}{r} B A x$$

Because $B = 0$ at initialization, $\Delta W = 0$ initially, ensuring the adapted model produces identical outputs to the baseline pre-trained checkpoint at step zero without training instability.

Hyperparameter Engineering & Practical Guidelines

  1. Rank ($r$) Selection:
    • For instruction tuning, classification, and structured extraction, $r = 8$ to $16$ is standard;
    • For mathematical reasoning, multi-turn coding, or domain transformation, increasing $r$ to $32$ or $64$ provides superior representational capacity.
  2. Scaling Factor ($\alpha$):
    • Conventionally set to $\alpha = 2r$ or $\alpha = r$. Maintaining a consistent ratio stabilizes optimization dynamics when scaling the rank.
  3. Target Layer Coverage:
    • Early implementations targeted only attention projections ($W_q, W_v$);
    • Modern empirical studies demonstrate that applying LoRA across all linear layers (Attention $W_q, W_k, W_v, W_o$ plus FFN $W_{\text{gate}}, W_{\text{up}}, W_{\text{down}}$) consistently yields results competitive with full-parameter training.

Deployment Advantages: Zero Latency & Multi-Tenancy

  • Zero Added Inference Overhead: Upon training completion, the adapter weights can be permanently collapsed into the base weights via standard matrix addition:

    $$W_{\text{deploy}} = W_0 + \frac{\alpha}{r} BA$$

    Deploying merged weights introduces zero additional latency or architectural modifications.

  • Dynamic Multi-LoRA Serving: In enterprise serving clusters, a single foundational model (e.g., LLaMA-70B) can remain memory-resident while lightweight task-specific LoRA checkpoints (megabytes in size) are dynamically swapped per request.

Advanced Variants

  • QLoRA: Compresses base weights into 4-bit NormalFloat (NF4) with Double Quantization and Paged Optimizers, enabling fine-tuning of 70B models on a single 24GB consumer GPU (e.g., RTX 3090/4090).
  • DoRA (Weight-Decomposed Low-Rank Adaptation): Decomposes weights into magnitude vectors and directional matrices, tuning directions with low-rank adapters to bridge the capability gap with full fine-tuning on reasoning benchmarks.

Further Reading