Tokens, Context and Why Conversations Have Limits

You can have a brilliant conversation with an AI assistant, return to it much later and discover that it has missed something you said earlier. That feels odd when the conversation is still visible on your screen. The explanation starts with two ideas: tokens and context.

Think of a conversation as more than a series of messages. To respond, a model needs an input containing your current question and enough background to make sense of it. That input takes up space. The available space is called the context window, and its size is measured in tokens, not ordinary pages.

A token is a piece of text that a model processes. Some common words may be single tokens; others are split into smaller pieces. Punctuation, numbers and unfamiliar terms also contribute. That means a 1,000-word document doesn't occupy exactly 1,000 tokens. The number depends on how the model divides the text.

Imagine you're asking AI to help plan a renovation. You begin by describing the room, your budget and what you dislike about the current layout. Over several days you explore colours, materials and three different schemes. By the time you ask for a final shopping list, the conversation contains the current choices, several abandoned ideas and many earlier details.

The model doesn't necessarily receive every earlier message forever. Depending on the application and model, older material might be omitted, summarised or otherwise managed to fit available context. The chat interface may still show the entire conversation history even when a particular answer was generated using only a selected part of it.

That's why a short project recap can be so helpful. You might say, 'We're keeping the existing cabinets, the budget is fixed and we've decided against tiles. Please use those decisions when suggesting the next step.' The summary isn't an admission that you're bad at prompting. It's a way of making the current, important information visible.

Context is also shared with other material. If you upload a long report, provide several webpages and ask for a comparison, the assistant needs to work with some representation of those sources as well as your instructions. Some products process documents through search or retrieval instead of loading every word at once. The way the app manages that material affects which details are available for a given response.

More context can be useful, but more isn't automatically better. Imagine sending an assistant every version of your renovation plan without identifying which decisions are final. It could accidentally mix a discarded layout with the current budget. A clean summary of current requirements may produce a more coherent result than an enormous unfiltered history.

There's another subtle point: context is not the same thing as permanent learning. When you correct a model in conversation, it can often use that correction in subsequent replies in the same session. That does not normally mean its underlying trained parameters have been rewritten to remember your correction forever.

To make this practical, try keeping three short notes during a long AI project: the goal, the decisions already made and the questions still open. Paste or reference those when starting a new conversation. You'll spend less time wondering which version the assistant has in mind.

Consider what happens when you change your mind halfway through a long project. You originally wanted a bright colour scheme for the kitchen, then decided on something more restrained. Both decisions remain in the conversation. If you later ask for a final plan, the assistant needs to distinguish the current choice from the abandoned one. A clean decision log makes this easy.

You can even write the recap as three short sections: what we're building, what we've decided and what we're still exploring. Ask the assistant to restate those constraints before generating something expensive or detailed. If it repeats an old decision, you can correct it before the error spreads into a full proposal.

Context management also matters when working with multiple documents. Imagine two reports with similar titles, one superseded by the other. Tell the assistant which document has priority and why. That little act of organisation often does more for the result than supplying another hundred pages of background.

Tokens and context can sound like technical infrastructure, but they explain an everyday experience: why the assistant sometimes needs reminding, why long documents can be challenging and why clear, relevant background helps. Once you know the limits, you can work with them deliberately rather than expecting perfect recall.