Context Window
The maximum amount of text a language model can consider at once — the budget every AI feature is designed inside.
A context window is the total tokens a model can hold in one request: system instructions, conversation history, retrieved documents and the response, all counted together. Exceed it and something has to be dropped. Which something, and whether the user is told, is a product decision that most AI features make silently and badly.
Bigger windows have not removed the constraint, they have moved it. Cost scales with tokens processed, latency grows with them, and models attend unevenly across a long context — material in the middle is reliably used less than material at either end. So the design question is not “how much fits” but “what earns its place”: which retrieved passages, how much history, and what gets summarised rather than carried verbatim.
The user needs to know what the model can see
“Summarise this document” silently truncating at page 40 produces a confident summary of part of a document. Showing what was included is the difference between a tool and a guess.
In practice
A support assistant carried full conversation history until it hit the limit, then silently dropped the oldest turns. Agents noticed it “forgetting” the customer's original problem mid-conversation. Replacing truncation with a rolling summary of earlier turns, and showing a marker where summarisation began, ended the complaints.
Where teams get it wrong
- Silently truncating input, so the model answers confidently about part of a document.
- Dropping the oldest conversation turns, which usually contain the actual problem.
- Filling the window because it is available, paying in latency and cost for material the model barely uses.
- Ignoring that mid-context material is attended to less reliably than the start or end.
- No indication to the user of what the model could actually see.
( Learn more )
Related terms
All terms- AI Product DesignDesigning products whose behaviour is probabilistic — where the system can be confidently wrong and the interface has to account for it.
- Prompt DesignDesigning what a product asks a model, and what it asks the user — the interface layer around a prompt, not just the wording of one.
- Conversational UIAn interface where the primary interaction is dialogue — and a pattern applied far more often than it is warranted.
- Generative UIInterfaces assembled at runtime by a model, rather than laid out in advance by a designer.