A bigger context window is not a better memory
A desk is not a filing cabinet, and the middle of the desk is a drawer the model rarely opens.
Yes, a two-million-token context window is impressive. No, it does not mean the model remembers you.
A context window is a desk. Everything the model knows about you during one conversation has to sit on that desk at once — your earlier questions, its earlier answers, the files you pasted in, the half-finished draft you asked it to fix on Tuesday. When the desk fills up, older things get pushed off, summarized into a paragraph, or silently squeezed into the corners where the model pays the least attention. A bigger desk delays that moment. It does not prevent it.
If you have ever come back to a chat two days later and found the context is gone — or spent an afternoon endlessly re-explaining yourself to a model you told the same thing last week — this is why. The window emptied. Nothing was ever written down anywhere else.
The desk has a size, and you are not told what it is
Every model has a hard ceiling on how much text it can hold at once. Vendors advertise the ceiling in tokens — 128K, 200K, 1M, 2M. Three things go wrong with that number in practice:
1. The ceiling is not all yours. The system prompt, tool definitions, safety instructions, and the model's own past replies all sit on the same desk. What's left over for the conversation you thought you were having varies by product — and none of the boxes tell you the real number.
2. The last turn in is the most expensive. Attention cost grows with the square of the context length. Long contexts are slow and expensive to run, which is why providers built machinery to avoid paying full price for old tokens — which brings us to compaction.
3. Big does not mean attentive. This is the part the token count never shows you.
Lost in the middle
Researchers at Stanford and Berkeley tested this directly in a 2023 paper, Lost in the Middle (Liu et al.). They hid a fact somewhere in a long document and asked models to find it. The result was U-shaped: models retrieved facts near the beginning and end of the context well, and facts buried in the middle markedly worse — sometimes worse than if the fact hadn't been provided at all.
So the desk metaphor is incomplete in one ugly way: the middle of your desk is a drawer the model rarely opens. Put your project's key constraint in turn 40 of a 60-turn conversation and it is functionally decoration.
The independent needle-in-a-haystack tests (Greg Kamradt's) found the same shape: retrieval accuracy degrades as the haystack grows and as the needle moves toward the middle. Passing the test at 100K tokens is not the same as using 100K tokens well. Chroma's 2025 Context Rot write-up pushed the finding further: performance degrades even on inputs inside the advertised window, and it degrades as conversations get longer and messier — which is to say, as they get real.
Compaction: the invisible eviction
When a coding agent or chat app hits its window limit, it rarely tells you. It just compresses. Older turns get summarized — "the user is building a scheduling tool; they prefer TypeScript" — and the summary replaces the original conversation. The raw back-and-forth, the failed approaches, the exact wording the client used, the reason you rejected the first three designs: gone, replaced by a paragraph that reads like meeting notes written by someone who left early.
This is why the eviction never feels like eviction. The chat continues. The model still speaks confidently. But it is now working from the summary, and in my experience compacted chats lose the worst things: summaries tend to keep what came up most often and drop what mattered once — and the thing that mattered once is usually the constraint. The one-off instruction you gave in week one ("never quote prices without checking the sheet") is exactly the kind of thing that gets elided, and exactly the kind of thing that costs you when the model cheerfully violates it in week four.
Why this is a memory problem, not a window problem
Here is the reframe the whole post rests on:
A context window is where an LLM thinks. Memory is where an LLM keeps.
The window is working memory. It was never designed to persist, and making it bigger is making working memory bigger — a bigger desk, with the same drawer in the middle and the same eviction notice when it overflows. What makes an assistant feel like it remembers you is a second thing entirely: a store outside the window that survives the session, gets written to deliberately, and gets pulled back in when relevant.
That is what "LLM long-term memory" means, and it is why the honest answer to "just use a bigger window" is no. A bigger window changes when the forgetting starts. It does not change whether anything is kept.
What this means if you are choosing a tool
When you evaluate any AI product's memory claims, the window size is marketing. The questions that matter are:
- What gets written down, and when? Is there a deliberate write path, or does the tool rely on summarizing the window when it fills?
- Where does it live? Can you export it, inspect it, correct it? A memory you cannot read is a memory you cannot trust.
- What happens on eviction? When the context compacts, does anything durable survive, or does the summary become the whole record?
If the answer to the first question is "nothing is written down deliberately," you do not have a memory feature. You have a large desk, and two days later the context is gone — exactly like the last tool.
The one-line version
A context window is where an LLM thinks. Memory is where an LLM keeps. The window size is marketing; the questions worth asking are what gets written down, where it lives, and what survives eviction.
Next in this series: what it actually takes for an AI to remember last Tuesday — why the write path, not the read path, is where memory actually gets made.
Disagree with any of this, or recognise the problem? jsingerman@7layer.studio — a human reads it, and corrections get published.