LLM Context Windows Explained

A large language model context window is the amount of text, measured in tokens rather than words, that the model can consider at once, including the prompt itself plus any documents retrieved to inform its answer. Understanding this limit, and its trade-offs, matters directly for cost and answer quality, not just for theoretical capacity.

A model's context window is the amount of text it can consider at once, including both the prompt and any retrieved documents, measured in tokens rather than words.

A larger context window sounds like an unambiguous benefit, but stuffing too much irrelevant information into it can actually reduce answer quality and increases cost, since every token is processed and paid for.

We design retrieval systems to fetch only the most relevant passages rather than maximising context usage, which tends to produce both cheaper and more accurate results.

A token is roughly three-quarters of a word in English, so a context window advertised as a large number of tokens still has real, practical limits once a system prompt, conversation history, and retrieved documents are all added together. Exceeding the window forces older content to be dropped or summarized, which can silently remove information a user still expects the model to remember.

Models also tend to pay less attention to information buried in the middle of a very long context compared to content near the beginning or end, an effect sometimes called "lost in the middle." This is one practical reason why simply pasting an entire document library into a prompt performs worse than a well-tuned retrieval system that surfaces only the passages actually relevant to a specific question.

We measure retrieval quality directly, using relevance scoring on retrieved passages before they ever reach the model, rather than assuming that more retrieved content automatically produces a better answer. In our experience, a tightly curated context of a few highly relevant passages consistently outperforms a context window stuffed to capacity, while also costing noticeably less per query.