Determinism is built around the model, not into it

“Can you make it deterministic?” The question comes up in nearly every conversation about putting an AI agent into a regulated process. Risk functions ask it, auditors ask it, and so does the engineer on call. It is the right question, asked of the wrong component.

In September 2025 a team at Thinking Machines Lab sent one prompt to one open-weight model a thousand times, with the temperature at zero, the setting that is supposed to remove randomness. They received eighty different answers. The cause was not sampling but arithmetic: the result of a matrix multiplication on an accelerator depends on how the work is batched, and the batch depends on how many other people’s requests arrive at the same moment. Kernels that are indifferent to batch size made all thousand answers identical, at a cost in speed. Hosted APIs make no such promise. Anthropic’s documentation says plainly that results are not fully deterministic even at temperature zero, and its newest models no longer accept the parameter at all.

So the honest answer has two halves. No: the model will not become deterministic, and a reproducible model would only be reproducibly wrong where it is wrong. Yes: the system can be, in the properties that matter. The worked example here is Claude Code, Anthropic’s coding agent, because its architecture draws the line between instruction and enforcement sharply, and documents it. The pattern carries over to any agent you build or buy.

What you need is invariants, not identical tokens

Nobody needs the same tokens twice. A regulated process needs certain properties of the outcome to hold on every run. There are five kinds:

• Never. Nothing is pushed to production unasked; no customer record leaves.
• Always. Every change is formatted, tested and logged.
• In order. A plan before an edit, a review before a merge.
• In shape. The output parses, the fields exist, the types are right.
• Before release. Nothing ships unless a check has passed.

These matter more for agents than for chatbots because of arithmetic. An agent’s task is a chain of reading, searching, editing, running and fixing, often dozens of steps long. If each step goes right with probability p, n independent steps go right with p to the power of n. Ninety-five percent per step is 36 percent over twenty steps.

The τ-bench study of 2024 ran each customer-service task eight times, with the agent’s temperature at zero. In the retail domain the best agent of the time solved about 61 percent of tasks in a single attempt, and fewer than 25 percent eight times out of eight. Models have improved a great deal since. The multiplication has not.

The same arithmetic runs in the other direction, and that is the most useful number in this article. Put a deterministic check behind a step (a compiler, a test, a schema validator) and let the agent try again when the check fails. One retry turns a 90 percent step into a 99 percent step; two make it 99.9. Over twenty steps that is 12 percent, then 82, then 98 (figure 1). No recent model upgrade bought as much, and this one costs a loop and a check. It works only for failures a program can detect, and only if the check’s error message travels back to the model: a retry that does not know why the first attempt failed tends to fail the same way.

An instruction in the context window is a probability, not a control

Everything that reaches a model as text works the same way: it shifts the probabilities of what comes next. System prompt, project instructions, retrieved documents and the conversation are tokens in one window, competing for the same attention. A well-written instruction shifts a lot. None shifts to one hundred percent.

Claude Code’s documentation is candid here. CLAUDE.md, the project instruction file, is loaded into every session, and the documentation calls it “context, not enforced configuration”. It arrives as a message after the system prompt, not as part of it. Compliance is not guaranteed; where two instructions contradict each other, the model may pick either. And adherence falls as the file grows: the recommendation is to stay under two hundred lines.

In mid-2025 the IFScale benchmark gave twenty models up to five hundred simple instructions at once; the best followed 68 percent. A 2026 replication by an observability vendor puts the ceiling of the newest models nearer two thousand. The limit moves; it does not disappear, and an instruction file shares the window with the task, the code and every tool result. Every rule you add makes each rule worth a little less.

The devices of this layer are therefore devices of context economy, the discipline of budgeting a context window:

• CLAUDE.md holds what is true in every session: commands, conventions, layout, past mistakes. It is re-read from disk when the conversation is compacted; an instruction given only in chat is not.
• Rules in .claude/rules/ can carry a path pattern and then load only when the agent touches matching files: accessibility rules while a component is open, not during a database migration.
• Skills are procedures with progressive disclosure: a short description sits in the window until the skill is needed, then the body loads. Bundled scripts run without their source entering the context.
• Subagents give a side task a fresh window and return only the result.

All of them raise the probability. None makes it one. This is the layer for judgement and knowledge: conventions, vocabulary, architecture, the reasons behind decisions, everything a program cannot check. The mistake is to keep requirements here that must hold every time.

A guarantee is code that runs without asking the model

Follow a single action through the harness (figure 2). The model proposes a tool call, say git push. That proposal is its only vote. Everything after it is decided by something else.

Hooks are your code, run by the harness at fixed points of its lifecycle; the documentation’s own term for what they provide is deterministic control. A PreToolUse hook receives the proposed call as JSON and can block it, hand it to a person, rewrite its input or let it pass. A block holds in every permission mode, including the one that skips all prompts.

Permission rules come next, in a fixed order: deny, then ask, then allow, first match wins. A deny from any settings file beats an allow from any other, and an organisation can ship managed settings that no user or project file overrides. A hook can tighten this and cannot loosen it: a hook that says allow does not get past a deny rule.

The sandbox is where the guarantee stops depending on reading a command correctly. The documentation calls permission patterns that try to constrain shell arguments fragile, and points to the sandbox for boundaries that must hold. There the operating system (Seatbelt on macOS, bubblewrap on Linux) enforces file and network limits on every shell command and its child processes, even when a prompt injection has taken over the model’s decisions.

PostToolUse hooks run after each action: format, lint, test, log. Stop hooks run when the model declares the task finished, and can refuse: not until the tests pass.

{
  "permissions": {
    "deny": ["Read(./.env)", "Edit(/public/.htaccess)"],
    "ask": ["Bash(git push *)"]
  },
  "hooks": {
    "PostToolUse": [
      {
        "matcher": "Edit|Write",
        "hooks": [
          { "type": "command", "command": "npm run check >&2 || exit 2" }
        ]
      }
    ]
  }
}

A few lines, four guarantees: a secret the file tools cannot read, a generated file they cannot edit, a person in front of every push, and a type check after every edit. The exit 2 is what hands its findings to the model rather than only to you.

Two limits. A hook can also be of type prompt or agent: a model evaluates it, which is useful for judgement calls and probabilistic again. The same holds for auto mode, since August 2026 the default on Anthropic’s individual and team plans, where a classifier model stands in for the person at the approval prompt. Deny and ask rules are evaluated before it, so an ask-rule keeps a person in front of the actions that matter. And hooks run with your full user permissions. Review and version them as code.

Order and shape belong in code

In 2024 Anthropic drew a useful distinction: workflows orchestrate models and tools through predefined code paths, agents direct their own process. The engineering question is never agent or workflow. It is which part of the plan can be known in advance, because that part belongs in code.

Claude Code’s dynamic workflows are the literal version. A JavaScript file holds the loop, the branching and the intermediate results. Each agent() call inside it is a probabilistic worker, but which workers run, in which order, how many in parallel, and that a verification step follows every finding is decided by the script. The documentation puts the difference in one phrase: who holds the plan. The script may not read the clock or draw random numbers (both calls throw), so a resumed run replays the same calls. And a worker given a schema must return JSON that validates; after five failed attempts the call fails instead of passing prose along.

Shape is the best-understood guarantee. Structured outputs on the API compile a JSON schema into a grammar that constrains token generation, so a completed response always parses. That settles form. Whether the value in the field is true is a separate question, and it needs a check of its own.

Research takes the idea furthest. In CaMeL, from Google DeepMind and ETH Zurich, an interpreter executes a control flow derived from the trusted request, so untrusted data can never change which tools are called. It solved 77 percent of a standard agent benchmark with provable security, against 84 percent undefended. Seven points is the price of a proof.

Human guidance belongs at design time, human decisions at a few points

The reflex is to keep a person in the loop for every action. The data says this is the weakest control in the stack. Anthropic reported in August 2026 that Claude Code users approve 97 percent of per-action permission prompts. In a controlled study, 1,053 paid testers each met one clearly dangerous command in their stream of prompts. They stopped it 13.6 percent of the time: about 17 percent early in a session, about 5 percent after fifty prompts. These are a vendor’s numbers, published to justify a classifier as the default reviewer. They also match decades of human-factors research on habituation, and the AI Act names the phenomenon in its oversight article: automation bias.

The same report contains the number that points to the solution. The same users reject 39 percent of plans. Judgement is alive at the level of intent, not keystrokes. So guidance moves to where it works:

• At design time. People write the instruction files, the permission rules, the hooks and the workflow scripts, in version control, reviewed like code. Written once, applied on every run.
• At the plan. Plan mode is enforced, not requested: edits stay blocked until a person approves the plan.
• At the irreversible. Ask-rules for the few actions that leave the machine: push, deploy, send, pay, delete. A boundary stated in conversation is not enough: the documentation notes that it can be lost when the context is compacted.
• At the result. A diff, a pull request, a second pair of eyes.

Undo has limits too: checkpoints rewind the agent’s file edits, not shell side effects or anything remote, so for those the gate must come before the action. And a maintenance rule follows: a correction typed twice goes into the instruction file; a violation that mattered once becomes a gate.

Put every requirement where something can actually hold it

Two questions sort every requirement (figure 3). Would a single miss be an incident? If not, it is guidance, and most requirements are. If it would: can a program decide whether it holds? If not, a person decides, at a point you designed. If it can, encode it, by kind.

What must be true	In Claude Code	In any agent stack	What it does not cover
It never happens	Deny rules, sandbox, a subagent without the tool	Least-privilege credentials, egress rules	Patterns on shell text are fragile; add OS-level isolation
It always happens	Command hooks at lifecycle events	Middleware around every tool call	Hooks judged by a model are probabilistic again
It happens in order	Workflow scripts, plan mode	Orchestration code, pipeline stages	Each step’s content is still model output
It has this shape	Schemas, structured outputs	Constrained decoding, validators	Valid is not the same as true
Nothing ships unproven	Stop hooks, a verifier, CI	Tests, build gates, review	Only what the check checks

The repository behind this website works this way. Its working agreement for the coding agent says: no third-party requests. That sentence is advice. What makes it true is a verifier that fails the build, and with it the deployment, if any script or stylesheet points to another host. The sentence tells the agent what I want; the script makes sure I get it. Wherever a requirement has only the sentence, I know what to write next.

What this means for a regulated organisation

Evidence lives in the deterministic layer. Permission files in version control, managed settings, hook scripts, pipeline logs: these are artefacts an auditor can read. For high-risk systems the AI Act asks for oversight measures built into the system (Article 14), automatic recording of events (Article 12) and consistent performance (Article 15); since the 2026 omnibus, from December 2027 for stand-alone systems. Even where the Act does not apply to your agent, this is the vocabulary your second line will use. “We told the model not to” is not a control description.

Guarantees outside the model survive a change of model. Model updates change behaviour; deny rules, hooks and gates do not care which model proposed the action. The more of your assurance sits in the shell, the cheaper the switch, which is what DORA means by an exit strategy and why lock-in through model behaviour is worth pricing before you sign.

Budget for the shell. The reliable part of an agent system is ordinary software: policies, scripts, tests and pipelines, with owners and reviews of their own. Most of the engineering in a dependable agent is not AI engineering.

Ask of every requirement you have of an agent: who enforces this when the model does not? If the answer is the prompt, it is a wish.