Comparisons
Long context vs retrieval
Long context places more material in one model input; retrieval selects a smaller set of material for each question.
The distinction
A long context window lets a model receive substantial source material together. This can help when the task depends on relationships across many parts of a bounded document set, and it reduces the need to design a separate search step for a small corpus. Capacity is not the same as reliable use of every included fact: position, distraction, conflicting passages, and output limits can still affect the answer. The supplied material also needs version control if the result must be reproduced later.
Retrieval chooses a subset of a corpus for a particular question. It is suited to collections too large or dynamic to include in full, and it can provide explicit locators for passages shown to the model. The retrieval step can miss decisive evidence, return redundant copies, or rank a convenient but outdated passage highly. Evaluation therefore needs to inspect both selection and final answer. A long context window and retrieval are compatible: retrieved passages can be accompanied by adjacent context when a narrow chunk would distort meaning.
By dimension
| Dimension | Long context | Retrieval |
|---|---|---|
| Selection | Much or all of a bounded set is supplied. | A query-specific subset is supplied. |
| Corpus size | Constrained by context capacity and cost. | Can search a corpus larger than one context. |
| Missed evidence | Included evidence may still be overlooked by the model. | Relevant evidence may never be retrieved. |
| Source updates | The input bundle must be refreshed and versioned. | The index and source copies must be refreshed and versioned. |
| Context | Nearby passages can remain together. | Chunking may separate qualifications from a claim. |
| Attribution | Citations need locators within the supplied material. | Retrieved passages can carry source locators. |
| Evaluation | Test whether the model uses relevant included evidence. | Test retrieval coverage and answer support separately. |
Choose Long context when
- The relevant corpus is bounded and fits within a controlled input.
- Cross-document reading needs broad surrounding context.
- The source bundle can be retained for replay.
Choose Retrieval when
- The corpus is large or changes frequently.
- Questions concern a small subset of material.
- Passage-level retrieval and attribution are required.
Using both
Retrieve likely evidence from the larger corpus, then include enough neighboring text in a long context to test qualifications and conflicts.
A worked example
A fictional legal team wants to know whether any of forty supply agreements contains a clause that restricts assignment. With a long context, the team supplies all forty agreements to a model in one request and asks the question. Retrieval would instead find the clauses that look relevant and show those.
Both can miss. The long-context run may overlook a clause buried in the middle of one agreement, and the retrieval run may not retrieve a clause worded unusually. The team tests both on a handful of agreements where the answer is known, including one with a hidden clause, before trusting either on the forty. The final answer lists each agreement, the clause found or the statement that none was found, and the passage with its location. The method matters less than the evidence that comes with the answer.
Where each one fails
Long context fails by position, distraction and cost. Studies of how models use long inputs have found that performance can be best when relevant information is at the beginning or end and worse in the middle, and that adding irrelevant passages can degrade answers. A full context also makes reproduction harder unless the exact contents are stored, and every request pays for all of it.
Retrieval fails by omission and by ranking. If the decisive passage is not retrieved, the model cannot use it, and nothing in the answer reveals the absence. Chunking can split a sentence from its qualification. Rankers favour passages that resemble the query, which is not the same as passages that settle it. Surveys of retrieval-augmented generation treat retrieval, augmentation and generation as separate stages, each needing its own evaluation.
Questions to settle before choosing
How large is the corpus, and does it change? Is the question about a few documents or about a needle in many? Must the answer list every instance, or does one suffice? Can the inputs be stored for replay? What does an omission cost? For exhaustive questions, such as find every clause, neither approach is safe without a completeness check, such as a deterministic scan or a second independent pass. For questions where one good passage is enough, retrieval with a modest context window is often adequate.
How the two meet in a review
Treat position and selection as things to measure. A test that moves the same evidence to different positions in the input shows whether the answer is stable. A test that removes the decisive passage from the retrieved set shows whether the system says it cannot answer. Either result belongs in the evaluation record, together with the model version and the exact inputs. A system that is evaluated only on whether the final answer looks right can pass while failing both tests.
References
- Liu et al. (2024). Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics 12.
- Gao et al. (2023). Retrieval-Augmented Generation for Large Language Models: A Survey. arXiv:2312.10997.
- Liu et al. (2023). Lost in the Middle: How Language Models Use Long Contexts. arXiv:2307.03172.
- Es et al. (2023). Ragas: Automated Evaluation of Retrieval Augmented Generation. arXiv:2309.15217.
Found an error on this page? See corrections for what counts as one, how to report it and the dated log.
Enquiry
Which do you run today: Long context or Retrieval?
Describe how your organization handles this now and one entity where it falls short. We reply with where a first record would sit alongside it and what it would need from you.
Or write to [email protected] in your own words.