How AI Chatbots Work: Tokens, Context, and Forgetting

A chatbot does not read your words the way you wrote them, and it does not remember a conversation the way a person does. It reads small chunks called tokens, holds a limited number in a context window, and builds each answer one token at a time. Most odd behavior in long chats, from dropped instructions to a missed fact in a document, follows from those three facts.

Tokens: the pieces a model actually reads

Before a model processes a prompt, a piece of software called a tokenizer splits the text into tokens. OpenAI describes a token as the unit its models use to process text. A token can represent a character, part of a word, a whole word, or punctuation, and spaces affect how the text is divided.

That split is not the same as a word count. Common short words often fit in one token. Longer or rarer words break into several. In an explainer for OpenAI models, “apple” is one token, while “hamburger” is three: “ham”, “bur”, and “ger”. Capitalization and spacing matter too. OpenAI notes that “red”, “Red”, and a version with a leading space can be represented differently. Google gives the same picture for Gemini models: a token can be a single character such as “z” or a whole word such as “cat”, and the full set of tokens a model can use is its vocabulary.

For rough planning in English, the two companies give similar rules of thumb:

• OpenAI: 1 token is about 4 characters, about three-quarters of a word, and 100 tokens are about 75 words.
• Google, for Gemini models: 1 token is about 4 characters, and 100 tokens is about 60 to 80 English words.

Both companies stress that these are estimates. The same text can produce different counts in different models, and other languages differ. Tokens are not only text. Google notes that all Gemini input and output is tokenized, including images at 258 tokens for an image up to 384 pixels in both dimensions (larger images are tiled at 258 tokens per 768 by 768 tile), video at 263 tokens per second, and audio at 32 tokens per second. A large file can therefore use far more of a window than the visible words suggest.

One token at a time: how an answer is built

A chatbot does not retrieve a finished answer from a database. It generates one, token by token.

At each step, the model considers the tokens it has so far and produces possible next tokens with associated weights. One token is sampled from that list, with heavily weighted tokens more likely to be chosen than lightly weighted ones. That new token is added to the sequence, and the whole process repeats. Generation stops when a length limit is reached or when the model produces a special stop token.

This loop explains two familiar traits. Answers can differ between runs of the same prompt, because sampling does not always pick the single highest-weighted token. And a confident tone is not evidence of accuracy: the process selects for a plausible continuation, so output can sound smooth and still be wrong. OpenAI also distinguishes input tokens (supplied in a request), output tokens (generated by the model), and reasoning tokens, which some models use internally before a visible answer. Reasoning tokens are not shown, but they count toward output usage, so a short answer can use more tokens than its text suggests.

The context window: working memory with a hard edge

The context window is the limit on how many tokens a model can work with in one request. OpenAI puts it plainly: the window limits the tokens in a request, and models also have a separate output limit. The window must hold system instructions, the conversation so far, any files supplied, tool results, and room for the answer, including reasoning tokens.

Size varies by model and product. Anthropic’s support page, updated on October 7, 2026, says that outside its listed newer models, Claude’s context window on paid chat plans is 200,000 tokens, about 500 pages of text or more. Many newer models listed there are shown at 500,000 or 1 million tokens, depending on whether the surface is chat, Claude Code, or Claude Cowork. A headline figure for one product therefore does not automatically apply to another.

A larger window helps fit a long document or conversation into one request. It does not create permanent memory. When a new chat starts, the previous conversation is not in the new window unless the product carries it over, for example as a summary. What feels like remembering across turns inside one chat is the earlier turns being supplied again as context for the next answer.

When a conversation approaches the limit, products handle overflow differently. Anthropic says that for paid plans with code execution enabled, Claude automatically summarizes earlier messages to make room, and that the full history is preserved for reference after summarization. A summary is a compression, though: names, exact numbers, and small conditions are at risk when a long exchange becomes a short recap. OpenAI gives similar advice for oversized inputs: shorten the prompt, remove repeated context, divide large inputs, or summarize material first.

Why long chats forget, and why the middle matters

Forgetting inside a long chat usually has one of three causes. The early material fell out of the window or was replaced by a summary. It is still present, but a later instruction conflicts with it. Or it is present and not contradicted, but the model does not use it well.

That third cause has a measured name: lost in the middle. In research by Liu and colleagues, published in TACL in 2024, models were given a question plus 10, 20, or 30 retrieved passages, only one of which contained the answer. The researchers moved that one passage to different positions and changed nothing else.

Accuracy formed a U shape. With 20 documents, GPT-3.5-Turbo answered correctly about 75.8% of the time when the answer passage came first, 63.2% when it came last, and 53.8% when it sat in the middle. Its score with no documents at all was 56.1%, so in the middle position, supplying the answer passage produced a slightly worse result than supplying nothing. Larger context windows did not fix the pattern in that study for inputs that fit in both versions, and increasing the set from 20 to 50 documents improved that model by only about 1.5%.

Newer models do better on simple retrieval tests, and the effect varies by task. The practical lesson still shapes vendor guidance. Anthropic’s long-context tips, as summarized in coverage of the study, recommend placing long documents above the query, and note that a query at the end can improve quality by up to 30% in Anthropic’s tests on complex, multi-document inputs.

How to work with the window instead of against it

A few habits follow directly from the mechanism:

• Put key instructions at the start of a long prompt, and restate the task and the key constraint again at the end, near the question.
• Keep the prompt as short as the task allows. Remove repeated background, old drafts, and documents the answer does not need.
• For a long document, ask about specific sections and quote the passage you mean, rather than assuming a fact buried in the middle will be used.
• In a very long chat, restate names, numbers, and decisions that matter instead of relying on an early message or an automatic summary to preserve them exactly.
• Start a new chat for a new task. Carrying a long, unrelated history wastes window space and adds competing instructions.
• Check important output against an outside source. Generation produces fluent text, not verified text.

Bottom line

A chatbot reads tokens, not words, at roughly 4 characters or three-quarters of a word per token in English. It writes by sampling one token at a time, and it can only use what fits in its context window for that request. When a long chat seems to forget, the cause is usually a full window, a summary that dropped a detail, or a key fact sitting in the middle of a long prompt. Shorter prompts, restated constraints, and key material placed near the start or end work with that design rather than against it. And because everything in the window is sent to the service, a chat is never private from that service itself; a VPN changes who can watch the connection, not what the service receives.