LLM Context Windows Explained

A model's context window is the amount of text it can consider at once, including both the prompt and any retrieved documents, measured in tokens rather than words.

A larger context window sounds like an unambiguous benefit, but stuffing too much irrelevant information into it can actually reduce answer quality and increases cost, since every token is processed and paid for.

We design retrieval systems to fetch only the most relevant passages rather than maximising context usage, which tends to produce both cheaper and more accurate results.