Document Chunking
Document Chunking represents the foundational data engineering pipeline in Retrieval-Augmented Generation (RAG). Its core mission is segmenting lengthy, heterogeneous raw unstructured documents (PDFs, Markdown files, Word documents, codebase repositories, or technical manuals) into standalone, semantically coherent, and easily retrievable text passages (Chunks).
Chunking quality directly governs the semantic purity of downstream embeddings, the recall accuracy of vector databases, and the signal density of the final context window.
1. The Core Trade-off: The Granularity Dilemma
Document chunking faces an inescapable engineering dilemma:
Small Chunks (100 ~ 256 Tokens) Large Chunks (512 ~ 1500 Tokens)
◄─────────────────────────────────────────────────────────────────────────────►
• Pros: High semantic density; sharp vectors; precise retrieval • Pros: Rich contextual scope; complete causal logic; coherent generation
• Cons: Context starvation; referential ambiguity ("it", "they") • Cons: Semantic dilution; needle lost in haystack; context overflow
- Oversized Fragmentation: Sentences are severed arbitrarily, creating severe referential ambiguity (e.g., "This method was executed in step two"), leaving the embedding model incapable of identifying the subject entity.
- Undersized Fragmentation: Embedding models are forced to compress multiple disjoint topics into a single high-dimensional vector. Fine-grained factual nuances are diluted in vector space, failing to trigger matches for targeted queries.
2. Five Mainstream Chunking Strategies
| Strategy | Operational Mechanism | Best Suited For | Engineering Risks |
|---|---|---|---|
| Fixed-Size with Overlap | Cuts text strictly at fixed token or character lengths, preserving a set sliding overlap (typically 10%–20%) | Unstructured prose, uniform narrative flows | Frequently severs sentences mid-clause, disrupting grammatical and logical continuity |
| Recursive Character Splitting | Evaluates a prioritized list of separators (e.g., ["\n\n", "\n", " ", ""]), breaking recursively at natural paragraph boundaries |
General prose documents (Default in LangChain / LlamaIndex) | Lacks semantic awareness of complex structured elements (tables, nested lists) |
| Structure-Aware AST Splitting | Traverses Markdown ASTs, HTML DOM trees, or Code ASTs (Tree-sitter), splitting strictly on section headers, functions, or classes | API references, developer docs, codebases, legal contracts | Fragile parser logic when encountering malformed syntax |
| Semantic Chunking | Measures cosine similarity across consecutive sentences; splits where similarity plunges below an adaptive threshold (e.g., percentile standard deviation) | Essays and long analytical reports with organic topic shifts | High pre-processing compute overhead (requires sentence-level embedding calls) |
| Propositional / Agentic Chunking | Deploys small LLMs to decompose passages into self-contained factual propositions, resolving pronouns and filling contextual context | High-stakes knowledge bases (financial reports, clinical protocols) | High LLM API cost and offline indexing latency |
3. Advanced Paradigm: Small-to-Big Retrieval
To resolve the contradiction between requiring small chunks for search and large chunks for generation, production systems deploy the Small-to-Big retrieval architecture:
┌─────────────────────────────────────────────────────────────────┐
│ Parent Document Chunk (1024 Tokens) │
│ ┌───────────────────────┐ ┌───────────────────────┐ │
│ │ Child 1 (256 Tokens) │ │ Child 2 (256 Tokens) │ ... │
│ │ (Indexed in DB) │ │ (Indexed in DB) │ │
│ └───────────────────────┘ └───────────────────────┘ │
└─────────────────────────────────────────────────────────────────┘
│ (Hit Child 1) │ (Lookup Parent ID)
▼ ▼
High Cosine Similarity Hit Full Parent Context Injected into LLM
- Parent Document Retrieval: Documents are partitioned into large parent chunks (e.g., 1000 tokens for complete sections), which are subsequently divided into small child chunks (e.g., 200 tokens). Only child chunks are embedded and indexed. When a child chunk matches a query, the system retrieves its associated parent chunk via metadata foreign keys to inject into the LLM prompt.
- Sentence Window Retrieval: Single sentences (50–100 tokens) serve as the primary vector unit. Upon a retrieval hit, the system dynamically expands the window by retrieving ±K neighboring sentences (e.g., 3 preceding and 3 succeeding sentences) from the physical document.
- Contextual Retrieval: Pioneered by Anthropic: before calculating embeddings, an LLM generates a 50–100 token situational summary (e.g., "This document is the Q2 2026 earnings report; this section discusses cloud infrastructure margin expansion") prepended to the head of each chunk. This eliminates referential isolation across documents and drastically improves search precision.
4. Metadata Preservation & Structured Content Guidelines
Chunking extends far beyond cutting raw text; it demands robust metadata lineage:
- Breadcrumb Hierarchy:
Every chunk must preserve structural lineage in metadata, such as:
{"hierarchy": "ZICQ Developer Guide > Database Architecture > PostgreSQL > pgvector Index Tuning"}. Prepending this path to the chunk text prior to embedding provides indispensable semantic category anchors. - Preserving Tables & Structured Artifacts:
- Never slice Markdown or HTML tables horizontally. Breaking table rows separates data from header schemas, blinding the downstream model.
- Tables should either be indexed as atomic units or flattened row-by-row into key-value factual assertions (e.g.,
Year: 2026, Revenue: 10B) before vectorization.
- Deduplication & Reranker Synergy: When using 20% overlap, adjacent chunks frequently appear simultaneously in top-K vector matches. Downstream pipelines must integrate a Reranker model or diversity filtering (such as Maximal Marginal Relevance, MMR) to filter redundant slices from the prompt.
Further Reading
- Introducing Contextual Retrieval (Anthropic) — Context-augmented chunking combined with hybrid search
- LlamaIndex Node Parsers & Text Splitters — Document splitting and hierarchical node parsing reference