Glossary
Chunking
Chunking divides source material into units that can be indexed, retrieved, and cited.
Context engineering
Chunking divides a larger source into smaller units for search or model input. A chunk can be a sentence, paragraph, section, page, or overlap of adjacent spans. Its boundaries affect what a retrieval system finds and what surrounding context a reader sees. An isolated sentence may lose a condition in the preceding paragraph; an oversized section may bury the relevant sentence among unrelated material.
A practical chunk retains its source identifier, position, and a way to recover neighbouring text. Boundaries should respect document structure where possible, especially headings, tables, footnotes, and definitions. Overlap can preserve continuity but may also create near-duplicate search hits that crowd out other evidence. Evaluation should use questions whose answers depend on qualifications and cross-paragraph context, not only straightforward keyword matches.
For defensible values, chunking is an evidence-handling choice rather than a harmless preprocessing step. It can separate a claim from the scope that limits it, or sever a table value from its column heading. Evidence spans and retained source copies let a reviewer inspect the original setting of a retrieved passage. Reranking can improve selection among chunks, but it cannot restore context discarded before retrieval.
Example
A paragraph on Halden Therapeutics AG (fictional) says a licence is valid only for one site; a chunk should keep that condition with the status claim.
Approaches to splitting
The simplest approach splits text into fixed-size pieces, by characters, words or tokens, with a small overlap so that a sentence at a boundary appears in both pieces. The dense retrieval work of Karpukhin and colleagues split Wikipedia articles into disjoint passages of one hundred words, which became a common baseline. Structure-aware approaches split at headings, paragraphs, list items or table boundaries, so that each chunk is a unit a human would recognise. Semantic approaches split where the topic changes, and hierarchical approaches keep both small chunks for retrieval and the larger parent section for display. The surveys of retrieval-augmented generation, such as Gao and colleagues, discuss chunk size and its effect on retrieval and on generation.
The trade-off is stable across methods. Small chunks match queries precisely but may lack the context to be understood; large chunks carry context but dilute the match and consume the context window. Documents with tables, footnotes and cross-references need special handling, because the meaning of a number may depend on a header several lines up, and the qualification of a statement may be in a note.
Chunks as units of evidence
A chunk is more than a retrieval unit; it is what a citation will point to. If chunks cut sentences in half, a cited chunk may not contain a complete claim. If they merge unrelated sections, a citation to the chunk does not identify the passage that supports the statement. Each chunk therefore needs stable identity: the document identifier and version, the position in the document, and a way to recover the exact text, ideally using the selectors of the W3C Web Annotation model described under evidence span. When a document is updated, the chunking should be rerun and the old chunks retained or versioned, so that earlier citations remain resolvable.
Duplicates matter. Documents repeated across a corpus produce near-identical chunks that crowd out other evidence in the top results and give an illusion of agreement, a retrieval-side version of source dependence. Deduplicate before indexing, or group duplicates at query time and present them as a single source. The article on RAG evaluation leakage audits shows how repeated or overlapping content can also contaminate evaluation.
A worked example
A supply agreement for Halden Therapeutics AG (fictional) has a clause on termination for convenience that runs across a page break, followed by a table of notice periods by contract value. A fixed 200-word split cuts the clause between its condition and its consequence, and puts the table headers in a different chunk from the table rows. A query about the notice period retrieves the table rows without their headers and the model reads the numbers as months when the headers say days. A structure-aware chunker keeps the clause whole, keeps each table with its header row and caption, and attaches the section title to each chunk. The retrieval test on twenty known questions rises from 11 correct to 17 correct. The figures are illustrative; the failure pattern is typical.
Checks a reviewer can run
Read a sample of chunks as a human would and ask whether each stands alone. Check that tables keep their headers. Retrieve with known questions and look at whether the needed sentence is in a single chunk. Check that each chunk can be traced to a document version and position. Count duplicates in the top results of typical queries.
Related terms and boundaries
Chunking determines what vector search can retrieve and what fits in the context window. It feeds retrieval-augmented generation, constrains reranking, and sets the granularity of an evidence span.
Common misreadings
- Separating a statement from its exception or date.
- Dropping table headers when indexing cells.
- Mistaking overlap-generated duplicate hits for independent evidence.
Appears in
- Long context vs retrieval
Long context places more material in one model input; retrieval selects a smaller set of material for each question.
References
- Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, NeurIPS (2020).
- Karpukhin et al. (2020), Dense Passage Retrieval for Open-Domain Question Answering, EMNLP.
- Gao et al. (2023), Retrieval-Augmented Generation for Large Language Models: A Survey.
- Barnett et al. (2024), Seven Failure Points When Engineering a Retrieval Augmented Generation System.
- W3C (2017), Web Annotation Data Model.
Found an error on this page? See corrections for what counts as one, how to report it and the dated log.
Enquiry
Where does "Chunking" bear on a decision of yours?
This entry sits under context engineering. If a decision in your organization turns on it, send the entity concerned, the sources you fuse to describe it and who may read which of them. We reply with what a first record would cover.
Or write to [email protected] in your own words.