Insight · September 18, 2026

Everything has to
fit in the frame.

A context window is the fixed span of text a model can hold at once. Your prompt, its reply, and everything in between share the same frame. When it is full, something falls out.

01 · The definition

A model has no memory. It has a window.

A large language model does not remember your last conversation. Each time it runs, it reads a block of text and predicts what comes next. The context window is the size of that block. Anthropic calls it the model’s working memory: all the text the model can reference when it generates a response, including the response itself.

That is the whole idea. The window is not storage. Nothing carries over on its own. Everything the model knows in the moment is the text sitting inside the window right now, and the window has a hard edge.

02 · The unit

The window is measured in tokens, not words.

A token is a chunk of text. It can be a whole word, a piece of one, a space, or a mark. OpenAI’s tokenizer notes that in English a token ranges from a single character to a full word, and averages about four characters. Tokens are simply what the model reads, so when a model states a limit, the limit is in tokens.

The current frontier is large. Claude’s latest models hold up to one million tokens in a single request, enough for a very long book with room to spare. A few years ago the largest windows held only a few thousand tokens. The numbers keep climbing, but the shape of the thing does not change. There is always an edge, and the edge is counted in tokens.

The one line to keep

“The window is not memory. It is the desk the model works on, and the desk is only so wide.”

03 · One budget

Input and output draw from the same pool.

The window is one budget, split between what goes in and what comes out. On Claude’s million token models a single request can hold the full history as input and still generate up to 128,000 tokens of reply, but both live inside the same limit. Fill the window with input and there is no room left to answer.

This is why a long prompt can earn a short answer, or none at all. The model is not being lazy. It has run out of frame. And if the input alone is already larger than the window, the request does not run at all. It fails with a plain message: the prompt is too long.

04 · Why there is a wall

The wall is built into the math.

A bigger window sounds like a setting someone could raise. It is not. Attention, the mechanism that lets a model weigh every token against every other token, grows with the square of the length. Double the tokens and you roughly quadruple the work. Ten times the tokens is near a hundred times the cost.

That curve is why windows have edges, and why long requests cost more and run slower. Engineers have pushed the edge out year after year, with better hardware and sharper attention tricks, but the shape of the cost has not moved. The window is a budget the architecture forces, not a limit someone forgot to lift.

05 · What counts against the window

People picture it holding their question. It holds far more.

The system prompt

The standing instructions the model reads before your first word. It sets the role, the rules, and the voice, and it rides along on every turn.

The conversation so far

Every earlier message, yours and the model’s, carried forward in full. A long thread is mostly history by the end.

Every tool definition

The description of each tool the model can call. A rich toolset can spend thousands of tokens before the model does anything.

Documents and images

A page or a picture is tokens too. A single request can carry hundreds of images or pages, and each one has a cost.

The reply itself

The model has to leave room to answer. The output shares the same frame as everything you sent to set it up.

Add it up and the window fills faster than anyone expects. The reply is competing for space with everything you sent to set it up.

06 · Bigger is not better

A larger window is not a longer memory.

It is tempting to treat a million tokens as a place to dump everything and let the model sort it out. That is the wrong instinct. Accuracy and recall fall as the window fills, a drift Anthropic names context rot. A model reads a full window less carefully than a focused one, so what you leave out matters as much as what you put in.

The skill, then, is not filling the window. It is curating it. Send the model what the task needs and little else. A smaller, cleaner window beats a larger, cluttered one, and it costs less to run. This is the whole practice behind context engineering, and the failure it fights is context rot.

07 · How to work with it

Treat the window as a budget you spend.

Once you see the window as a fixed budget, the decisions get simpler. Know roughly how many tokens your prompt spends. Keep the standing instructions tight. Trim the history and the old tool output you no longer need. Leave clear room for the answer.

For anything that outgrows a single window, the answer is not a bigger model. It is structure: retrieval that pulls in only the passages that matter, memory that persists the few facts worth keeping, and compaction that summarizes what came before. The window stays small on purpose, and the system feeds it well.

Closing

The window holds what you put in it.

Count your tokens once. Decide what earns a place in the frame. The model does its best work when the desk in front of it is clear, and the desk is only ever as clear as you keep it.

Written by · September 18, 2026

Donny Smith · ECD, Founder, Bttr.

Over the past 15 years, he has led creative teams and contributed to products used by millions of people worldwide, working with companies including Apple, Opendoor, JP Morgan, GE Aerospace, Pepsi, and Alterra Mountain Company.

Definitions and window sizes · Anthropic context windows documentation, 2026 · token behavior from OpenAI's tiktoken tokenizer and token counting cookbook · attention cost from the standard transformer literature

Share this perspective

More insights

Adjacent perspectives.

8 min read

How to Evaluate a RAG System

Evaluating a RAG system means scoring its two stages on their own, because a retrieval augmented generation answer is only as good as the passages behind it, and a wrong answer has two possible causes a single number cannot tell apart. Retrieval is measured before the model writes anything. The LlamaIndex evaluation guide scores a retriever with hit rate, the share of questions where the right passage lands in the top results, and mean reciprocal rank, which rewards ranking it high. Generation is measured given what was retrieved. Ragas, the open evaluation toolkit now maintained by VibrantLabs, splits the answer into claims and scores faithfulness as the share the retrieved context supports, then scores answer relevance by generating questions from the answer and comparing them to the real one. Read as two pairs, the four metrics locate the fault: high recall with low faithfulness is a generation problem, low recall with high faithfulness is a retrieval one. Anthropic showed the payoff in September 2024, cutting a retrieval failure rate measured by recall at 20 from 5.7 percent to 1.9 percent by adding context, keyword search, and reranking. And it is not a one time check. Models drift, documents go stale, so the teams that trust their systems run these scores on live traffic the way they run unit tests.

8 min read

What Is a Reasoning Model

A reasoning model is a large language model trained to work through a problem in steps before it answers, rather than reply in one pass. It spends extra computation at the moment you ask, called test-time compute, to write out a chain of thought and check its own work. Anthropic describes its extended thinking as letting the very same model give itself more time and effort rather than switching to a separate model, and reported that Claude 3.7 Sonnet reached 84.8% on the hard GPQA science set, with a 96.5% physics subscore, using parallel sampling at a 64K token thinking budget. The habit is trained, mostly by reward. DeepSeek showed in January 2025 that its R1 Zero model learned to reason purely through reinforcement learning with no supervised fine-tuning first, developing self checking and long chains of thought on its own, and shipped DeepSeek R1 under the MIT license at performance comparable to OpenAI o1. The cost is real, so the newest models add a dial. Claude Opus 4.5 hit its best score on SWE-bench Verified using 76% fewer output tokens at a medium effort. The skill is matching how hard the model thinks to how hard the task is.

8 min read

What Is Fine-Tuning

Fine-tuning is training a pre-trained model further on your own labeled examples so a behavior is set into its weights. It changes how a model acts, its tone, its format, its grip on a narrow task, rather than what facts it knows. Google frames the move as taking a general-purpose model and adapting it to your domain on a curated dataset of your own examples. The gains are measurable. When Anthropic opened fine-tuning for Claude 3 Haiku in Amazon Bedrock, a team fine-tuned it to moderate comments and lifted classification accuracy from 81.5% to 99.6% while cutting tokens per query by 85%. Lighter methods such as LoRA keep the base model frozen and train a small set of new weights beside it. The order that saves money is a plain prompt first, retrieval when the model lacks facts, fine-tuning only when it lacks the right behavior. A fact that changes is a retrieval problem. A behavior that drifts is a training problem.

View all insights →

Bttr. Field Brief

The brief Bttr. writes for senior buyers.

Monthly. One signal worth your time on Brand Operating Systems, AI search visibility, and the infrastructure buildout. No filler.

Spend the window well

Everything has to
fit in the frame.

A model has no memory. It has a window.

The window is measured in tokens, not words.

Input and output draw from the same pool.

The wall is built into the math.