Glossary
Context window
A context window is the bounded input span a model can process in a single inference interaction.
Context engineering
A context window is the bounded sequence of tokens available to a model for a given interaction. It may contain instructions, conversation history, retrieved documents, and the user's current request. The nominal capacity describes what can fit, not what the model will use accurately. Long inputs can dilute relevant evidence or place it where the model is less likely to attend to it, even when the entire text fits within the stated limit.
Good use of the window prioritizes task-relevant material, preserves source boundaries, and makes omissions deliberate. Chunking and reranking help select passages, but can also remove qualifications needed for interpretation. Position and formatting should be tested with the actual questions the system must answer. A reviewer should know when the available evidence was truncated, especially if a missing passage might reverse the conclusion or reveal a conflict.
An organization cannot defend an answer solely by saying the document was within the model's context window. It must show that the supporting part was present, applicable, and reflected in the response. Research on long-context use shows that location within a long input can affect performance. Context engineering therefore concerns both selection and presentation, while grounding and faithfulness assess the claim that emerges from that supplied context.
Example
A long dossier about Halden Therapeutics AG (fictional) fits in the window, but the relevant exception is buried between unrelated appendices and must be checked explicitly.
Why the window is bounded and what that costs
The transformer architecture introduced in Attention Is All You Need lets every token attend to every other token in the input, and the cost of that attention grows with the square of the sequence length in the basic form. This is one reason models have a maximum number of tokens that they can process in a single call, and why larger windows are more expensive to serve. Windows have grown a great deal, and techniques have been developed to extend them, but the limit remains a design constraint: the window must hold the instructions, the question, the retrieved material, any conversation history and the space for the answer.
A larger window does not by itself guarantee that all of its contents are used well. The study Lost in the Middle found that performance was highest when relevant information was at the beginning or end of the input and degraded when it sat in the middle, even for models built for long contexts. The comparison of retrieval and long context by Xu and colleagues found that retrieval can help even models with very long windows, and that the best configuration depends on the task. Treat the stated window as an upper bound on what can be provided, not as a measure of what will be reliably used.
Working within the window
Strategies include selecting only the most relevant passages through vector search and reranking; ordering them so the most important appear where they are most reliably used; summarising older conversation turns; and dropping material that is not needed for the current step. Each has a cost to evidence. A summary replaces words with a model's account of them, and the qualifiers may be lost, as the article on LLM context compression and evidence loss tests. Where an answer must cite its sources, the compressed context should retain source identifiers and exact quotations for anything that may be cited.
Measure rather than assume. Build test questions with a known supporting passage, vary its position and the amount of surrounding material, and record accuracy. The article on long-context robustness describes such a test. Record the window use for every call: how many tokens of each kind were included, and how many were dropped, so that truncation does not occur silently.
A worked example
A question about the obligations in a long agreement with Halden Therapeutics AG (fictional) requires clauses from three different sections and a schedule of amendments. The agreement runs to 150 pages, which fits within the model's window, but the pipeline retrieves ten passages of about 500 tokens each and places them in score order, with the amendment schedule ranked seventh. Tests with similar questions show that answers are less accurate when a needed passage is in the middle of the list. The team changes the order to place the amendment schedule first and the remaining passages by document order, and adds a prompt line that states how many passages follow. The same test set then shows improved accuracy. The window was never the limit; the arrangement within it was.
Checks a reviewer can run
Ask what the system does when the material exceeds the window: truncate, summarise or refuse, and whether it records the choice. Move a known passage through positions in the prompt and see whether the answer changes. Check that any compression keeps identifiers and exact quotes for cited material. Confirm that the instructions are not displaced by long inputs.
Related terms and boundaries
The window is the budget within which context engineering works. Its limits are why chunking, reranking and selection exist, and its uneven use is a source of lapses in faithfulness.
Common misreadings
- Treating nominal token capacity as a guarantee of accurate use.
- Truncating a qualification without recording the omission.
- Assuming a cited document was fully read because it was supplied.
Related terms
Appears in
- Long context vs retrieval
Long context places more material in one model input; retrieval selects a smaller set of material for each question.
References
- Liu et al., Lost in the Middle: How Language Models Use Long Contexts, Transactions of the Association for Computational Linguistics (2024).
- Vaswani et al. (2017), Attention Is All You Need, NeurIPS.
- Liu et al. (2023), Lost in the Middle: How Language Models Use Long Contexts, TACL.
- Xu et al. (2023), Retrieval meets Long Context Large Language Models.
- Anthropic Engineering, Effective context engineering for AI agents.
Found an error on this page? See corrections for what counts as one, how to report it and the dated log.
Enquiry
Where does "Context window" bear on a decision of yours?
This entry sits under context engineering. If a decision in your organization turns on it, send the entity concerned, the sources you fuse to describe it and who may read which of them. We reply with what a first record would cover.
Or write to [email protected] in your own words.