Why AI Forgets in the Middle of a Conversation

Why AI Forgets in the Middle of a Conversation

· 8 min read

Auf Deutsch lesen →

Listen to this article
Speed

AI narration · automatically generated

Subscribe as a podcast
RSS
Personal note

Stored only locally in your browser — nothing is sent.

Picture an AI’s workspace as a desk of fixed size. Everything you are currently talking about has to sit on it at the same time — every question, every answer, every document. As long as little is on it, the AI finds each note instantly. The fuller the desk gets, the more has to give way, and the harder it becomes to find what matters in the pile.

Once you have understood this one desk, you are already halfway to understanding why the AI gets worse over time. Let’s start with the smallest building block — the thing that ends up on the desk in the first place.

What a token is

We humans read in letters and words. An AI does not. Before it even “sees” a single word, your text is chopped into small building blocks — tokens. A token is usually a fragment of a word: sometimes a whole short word, sometimes a syllable, sometimes just an ending.

Short everyday words like “and”, “the”, “house” are so familiar to the AI that each gets a single token. The longer and rarer a word is, the more pieces it breaks into. The trick behind it: from several tens of thousands to over a hundred thousand known pieces, any word can be assembled — even one the AI has never seen before. So no word is ever a complete mystery.

As a rough rule of thumb: a longer word is often split into two or three tokens — more in languages like German — and a full page of text quickly comes to 600 or 800 tokens. What matters is only the principle: the AI thinks in tokens, not in words. Everything that follows is measured in this currency.

The tool that performs this splitting is called the tokenizer — the “chopper”. It is not a thinking step but a translation step: from our language into the machine’s building-block language and back again. You never get to see it, but it is at work in every single message.

The token window: the AI’s desk

An AI has no memory that grows with every conversation (leaving aside the memory features of newer programs). Instead it has a token window — experts also call it the context window — a workspace of fixed size into which only a certain number of tokens fits. That very desk with a clearly limited surface.

Onto this desk, everything has to fit at once —

  • the invisible base instruction on how the AI should behave,
  • every one of your previous questions,
  • every previous answer from the AI,
  • and every document you upload.

The entire conversation lies on the desk again, in full, with every new answer. The AI does not read “onward”; it reads the whole pile anew each time, in order to know what to say next.

How big is this desk? That depends on the model. Many of today’s common systems hold roughly 128,000 to 200,000 tokens — about the length of a whole novel. The top models of 2026 manage a million tokens and more, several books at once. That sounds boundless. But it is not — and the reason is the real heart of this story.

Why the AI seemingly gets “dumber”

Two quite different effects come together here, and they are worth keeping apart. Both feel the same to the user — “the machine is slipping” — but they have different causes.

First: the desk overflows — and the AI truly forgets. When the token window is full and you keep writing, room has to be made. Most chat programs cope by quietly clearing the oldest material off the desk, or roughly summarizing it. What has fallen off is then literally no longer there for the AI. That is why it “forgets” precisely what you said at the very beginning — the name, the instruction, the one important rule. Not out of carelessness. It simply is no longer on the desk. That is the most honest answer to “why does the AI forget?”: the window is its memory, and a full desk has no room left for old things.

Second: even a not-quite-full desk gets cluttered. The subtler effect sets in even before the overflow. The more lies on the desk, the harder it becomes for the AI to fish out the one thing that really matters from the clutter. Researchers have measured this precisely and given it a memorable name: “context rot”. A 2025 study tested 18 of the leading models, and every single one delivered worse results the fuller its window was.

Especially treacherous is a pattern that a Stanford study described back in 2023 as “lost in the middle”. The AI reliably uses information at the very top and the very bottom of the pile, but readily overlooks what sits in the middle of a long text. In the tests, the hit rate fell from around 75 percent to about 55 percent, purely because the sought-after detail lay in the middle rather than at the edge. It is like a person handed fifty more pages just before an exam: they remember the first page and the last, while the middle blurs.

What matters is what does not happen here: the AI does not grow tired, it does not “teach itself” anything, and it does not degrade permanently. Start a fresh conversation tomorrow and it is back in top form. The drop in performance lives solely in the cluttered desk of the current conversation. That is good news — because it means you can do something about it.

What you can do about it yourself

The most effective rule follows straight from the image of the desk: keep it empty and tidy. Concretely:

  • A new chat for a new topic. The most common mistake is running one mammoth conversation across days and ten topics. Every finished topic is ballast that only distracts the AI on the next one. A fresh chat is a freshly cleared desk.
  • Put the important thing up front or at the end. Do not bury the one decisive instruction in the middle of a long paragraph. Place it at the top — or repeat it at the close of your message.
  • Summarize long conversations and start over. At the end of a long chat, ask the AI for a concise summary of the essentials, copy it into a new conversation — and carry on there at the tidy desk. That is the single most effective move of all.
  • Do not upload everything at once. Handing over five long PDFs “just in case” does not make the AI smarter, only the desk fuller. Give it what belongs to the actual question.
  • When it starts to falter, that is a signal. If the AI contradicts itself or forgets what you agreed, the desk is usually simply full. Then no amount of prodding helps — a restart with a summary does.

The pro fix: many small desks instead of one big one

In the field, a different answer is bringing the biggest leap right now — and it shows where things are headed. If a single desk inevitably becomes cluttered with too much material: why not break the problem into parts and give each part its own fresh desk?

That is exactly the idea behind multi-agent or sub-agent systems. Instead of letting a single AI heave everything into one overflowing window, a lead agent works like a project manager: it breaks the big task into sub-tasks and hands each one to a sub-agent. Each of these sub-agents is its own AI instance with its own, completely empty token window. It deals only with its small part, undisturbed by everything else, and reports back only the result at the end — not the whole pile of thinking it produced to get there.

The advantage is twofold. No single window clogs up, because the work is spread across many clean desks. And the lead agent keeps the big picture, because only the finished results land on its desk, not every intermediate step. It is the difference between one person meant to write a whole book alone and in one go, and an editorial team in which each person takes a chapter and the editor-in-chief brings it together at the end.

A figure from practice shows this is no mere theory: on an internal research benchmark, the AI lab Anthropic reported that such a team of several agents beat the best single AI by a good 90 percent. That is precisely why this principle now sits under the hood of many serious AI tools — including the ones this text was written with.

What stays with you

The feeling that the AI is getting “a little dumber with every sentence” is not deceptive — only the explanation for it is. What you are seeing is not a decline in intelligence but a desk filling up. Once you have understood that, you handle an AI more calmly and more skillfully: the machine has not grown tired. The desk is just full — and a desk can be cleared.

If this piece gave you something to think about, feel free to share it — and the next time your AI loses the thread mid-conversation, think of the full desk before you despair of it.

Sources (selection):