Context engineering: your token budget is your architecture

Ask an engineering team why their AI assistant gives wrong answers and you will usually hear one of two things: “the model hallucinates” or “we need a better model”. In my experience, the honest answer is almost always a third one: the context we gave the model was wrong, incomplete, or badly arranged. The model did exactly what it was asked with what it was given.

This article is about treating the context window as what it is — a scarce, expensive, structured resource — and designing it with the same care you would give a database schema or a network protocol. I call this context engineering. It is the part of AI systems work that decides whether a pilot becomes a product.

Tokens are the unit of everything

A language model does not see words. It sees tokens: subword units produced by a tokenizer, typically a byte-pair-encoding variant trained on a large corpus. A common English word is often one token; a rare word, a German compound noun, a product code or a JSON key can be several. Everything you care about in production is denominated in tokens:

• Cost. Providers bill per input and output token. Self-hosted models cost GPU time per token.
• Latency. Time to first token grows with prompt length; total time grows with output length.
• Attention. The model’s capacity to use what you gave it is finite and unevenly distributed across the window.

The practical consequence is that a token budget is not a nice-to-have. For every request type in your system — “answer a policy question”, “summarise a case file”, “draft a reply” — you should be able to say how many tokens go to instructions, to retrieved material, to conversation history, and to the answer, and why.

Long context is not a substitute for retrieval

Context windows have grown from a few thousand tokens to hundreds of thousands, and it is tempting to conclude that retrieval is obsolete: just put everything in. Three things argue against that.

First, cost and latency scale with what you put in, on every single request. A window that is 90 percent irrelevant material is a bill for 90 percent waste.

Second, models do not use long contexts uniformly. Work by Liu and colleagues showed that performance on retrieval-style tasks degrades when the relevant passage sits in the middle of a long prompt rather than at the beginning or end. Newer models have narrowed the gap, but “put it somewhere in the window” is still not the same as “the model will use it”.

Third, and most important for regulated environments: what you put in the window is what the model can leak, misquote or act on. Feeding an assistant an entire document store because you can is a data-minimisation failure waiting to be written up.

Retrieval-augmented generation — selecting a small, relevant set of passages per request — remains the right default. Long context changes how much you can afford to retrieve and how you order it, not whether you retrieve.

Chunking is a modelling decision

The single most underrated design choice in a retrieval system is how documents are split into the units that get embedded and retrieved. Teams often accept a library default (fixed 500-token windows with overlap) and then spend months tuning embedding models to compensate.

Chunks should follow the structure of meaning in your documents. A regulation splits naturally into articles and paragraphs; a contract into clauses; a runbook into procedures; a support ticket into problem, context and resolution. Splitting at those boundaries produces chunks that are self-contained and quotable, which matters twice: once for retrieval quality, and once for the human who will read the citation.

Two techniques pay for themselves almost every time:

1. Contextual prefixes. Prepend to each chunk a short description of where it comes from — document title, section path, date, owner — before embedding. A chunk that says “Section 4.2 of the Q3 outsourcing policy, effective January 2026” retrieves far better than the bare paragraph. Anthropic published a straightforward version of this idea as contextual retrieval; the effect is large and the cost is a one-off preprocessing step.
2. Metadata as a first-class filter. Store jurisdiction, product, validity dates, classification and language alongside each chunk, and filter before similarity search. Most wrong answers I have debugged in enterprise systems came from a semantically similar chunk that was simply from the wrong year, the wrong entity or the wrong language.

Ordering and framing inside the window

Once you have the right passages, how you place them matters. A reliable pattern:

• System instructions first, short, with the task, the role, the output format and the rules for saying “I don’t know”.
• Retrieved passages next, each clearly delimited and labelled with its source, most relevant first and last (the positions models attend to best), less relevant in the middle.
• Conversation history after that, trimmed to what is needed for the current turn.
• The user’s actual question last, restated if the conversation is long.

Delimit passages with explicit markers and require the model to cite them by label. Citation is not decoration: it lets you automatically check whether the answer is grounded in what was retrieved, which is the basis of your evaluation.

Measure, or you are guessing

Every change to chunking, retrieval, ordering or prompts should be evaluated against a fixed set of questions with known-good answers and known-relevant sources. At a minimum, track:

• Retrieval recall at k — did the relevant passages make it into the window?
• Answer faithfulness — is every claim in the answer supported by a retrieved passage?
• Answer correctness — judged against reference answers, by humans for a sample and by a model for scale.
• Tokens and latency per request — so that quality gains are seen with their cost.

The evaluation set should be built with the people who will use the system, in the language they use, from questions they have actually asked. It is the most valuable artefact in the project and the one most often skipped.

What this means for your architecture

Context engineering turns “we use an LLM” into a system with explicit interfaces: a document pipeline that produces well-formed chunks with metadata, a retrieval layer with filters and re-ranking, a prompt assembler with a budget, and an evaluation harness that gates changes. Each part can be owned, tested and replaced — including the model.

It is also the part of the system that your risk and compliance functions will want to see. A documented token budget, a retrieval log and a faithfulness metric are precisely the evidence that turns “the AI said so” into a defensible process.

If your pilot works on ten documents and falls apart on ten thousand, this is where to look first.