The Harness

Strip the vocabulary away and a harness is four things: a loop that calls the model repeatedly instead of once; tools that let its output cause real effects; plumbing that decides what the model sees each turn; and controls that decide what it's allowed to do and when it's done. Chat interfaces are minimal harnesses (one turn, no tools). Coding agents are heavy harnesses (hundreds of turns, dozens of tools, compaction, sandboxes, test loops).

The reason "harness" became the word of the moment: teams kept discovering that swapping the harness moved results as much as swapping the model. The same frontier model that meanders in a bare chat window ships working code inside a well-built coding harness — because the harness feeds it test output, lets it retry, keeps its context clean, and stops it when it's done. Capability lives in the pair, not the model.

Seven numbered lines, seven design decisions — and the rest of this manual is those lines expanded. Notice what the loop implies:

• The model never acts. It emits a request to act. Your code executes. Everything you control about safety and reliability lives in that gap.
• Results are just more context. The model "perceives" the world only through what you append to history. Garbage tool output in, garbage decisions out.
• Stop conditions are yours to write. Turn caps, token budgets, wall-clock limits, cost ceilings, "no progress in N turns." A loop without them is an outage generator.

The rule of thumb that separates hobby harnesses from production ones: steer with code when you can, prose when you must. "Never call delete_db in prod" written in the prompt is a suggestion the model will occasionally ignore under pressure; the same rule written as a gate in code is physics. Prompts steer judgment (tone, priorities, when to ask vs. act). Code enforces invariants (permissions, budgets, formats).

• Instructions compete with everything else in the window. A 6,000-token prompt the model half-follows loses to a 1,500-token prompt it fully follows. Cut until every sentence has changed behavior in a trace you've read.
• Positive beats negative. "Cite the file path for every claim" outperforms "don't make things up." Models follow patterns better than prohibitions.
• Examples are the strongest steering tokens you own. One worked example of a perfect tool call or output format outperforms three paragraphs describing it.
• Version it like code. The prompt is a build artifact: diff it, review it, and eval every change (Module 07). Silent prompt edits are how regressions ship.

The model chooses tools the way you'd choose from a menu written by a stranger: entirely from the names, descriptions, and parameter hints. It can't read your source. So the schema is the entire interface — and vague schemas produce an agent that guesses, thrashes between similar tools, and hallucinates parameters.

• Name by intent, scope by task. search_orders beats query_db. If two tools overlap, the description must say when to pick each — or the model will alternate between them at random.
• Errors are teaching moments. A stack trace teaches nothing; {"error":"date_range too wide","hint":"max 90 days — split the query"} lets the model self-correct on the next turn. Design error returns for a model reader.
• Budget the output. Every byte a tool returns is context you pay for (Module 04). Return the 20 rows the task needs with a total_matches count, never the 10,000-row dump.
• Fewer, sharper tools. Past a couple dozen tools, selection accuracy degrades. Consolidate (file_read/file_write/file_list → one file tool with an action enum) or gate tool visibility by task phase.
• Mark side effects. Distinguish read-only tools from world-changing ones in the schema itself — your gate layer (Module 05) keys off exactly this.

Two facts drive everything in this module. First: in a loop, context compounds. Every tool result, every retry, every file read stacks into history — a 40-turn session can eat a six-figure token window on results alone. Second: quality degrades before capacity runs out. Models attend less reliably to material buried in a huge window ("context rot"), so an agent at 90% capacity isn't almost-fine — it's already reasoning worse.

• Just-in-time retrieval beats preloading. Don't stuff every maybe-relevant document in up front — give the agent search/read tools and let it pull what the current step needs. Lighter window, fresher attention.
• Compaction is lossy — test the loss. A summary that drops the one constraint that mattered ("customer said don't email them") is worse than truncation. Treat your summarizer as a component with its own evals (Module 07).
• Memory files are the cheapest long-horizon trick. A plain markdown file the agent reads at start and appends decisions to (NOTES.md, CLAUDE.md) gives persistence across sessions with zero infrastructure — and you can read it, which matters for debugging.
• Cost is a context problem. Loops multiply tokens: 30 turns × a fat window is real money per task. Compaction and output-budgeted tools (Module 03) are your cost levers as much as your quality levers.

Score every tool call on two axes: reversibility (can you undo it?) and scope (who or what does it touch?). Reading a file is reversible and private — automate it. Sending an email is irreversible and external — a human sees it first. Deleting production data is irreversible and systemic — the harness shouldn't even own that capability. The matrix below encodes the pattern:

• Allowlist, don't blocklist. Enumerate what's permitted; everything else fails closed. Blocklists lose to creativity — yours and the model's.
• Draft, don't send. For anything external (email, posts, messages), the agent produces the artifact and a human fires it. You keep 95% of the leverage and 0% of the "it emailed the whole client list" incidents.
• Sandbox first, always. Give the agent a disposable environment where AUTO is safe, and promote actions outward only with evidence. Cheap sandboxes are the highest-ROI safety spend in the field.
• Budgets are gates too. Token ceilings, dollar caps, wall-clock limits, max-turns — quantitative gates that stop runaway loops no rule anticipated.
• Audit everything. Every call, every gate decision, every human override into an append-only log. When something goes wrong, the trace is your only forensics.
• Approval fatigue is a failure mode. Gate too much and humans rubber-stamp; the gate becomes theater. Ruthlessly automate the safe stuff so attention is spent where it counts.

The organizing idea is the generator–verifier gap: tasks where checking is much cheaper than doing are the tasks agents crush. Code is the flagship case — tests and compilers are free, instant, incorruptible critics — which is exactly why coding agents matured years ahead of everything else. The engineering corollary: you can often make a task agent-ready by making it checkable — demand structured output you can validate, define acceptance tests up front, decompose fuzzy work into steps with crisp pass/fail edges.

Harness evals are not model evals. Model benchmarks (MMLU, coding leaderboards) tell you about the engine. Your eval tests your car: your prompt, your tools, your compaction, your gates, on your tasks. The recipe is unglamorous and universal:

• A golden set — 10 to 50 real tasks with known-good outcomes. Start with the failures that made you swear; those are your regression tests forever.
• A grader per task type — deterministic where possible (exact match, tests pass, schema valid), an LLM judge with a written rubric where not. Judges get spot-checked against human grades before you trust them.
• Multiple runs per task. Agents are stochastic: a task that passes 6 of 10 runs is a coin flip in production. Track pass rates, not pass/fail.
• Run on every change — prompt edit, tool rename, model swap, compaction tweak. The harness is a system; any change can regress any behavior.
• Read traces. Scores tell you that it broke; the transcript tells you where. The core practitioner skill of this whole field is reading agent traces the way a mechanic listens to an engine.

• Tool-calling reliability beats leaderboard IQ. An agent lives or dies on emitting valid, well-chosen tool calls turn after turn. Test that specifically — it varies more across models than reasoning scores suggest.
• Loops multiply cost. Per-token price × 30 turns × fat context is the real bill. This is why routing and pipelines mix tiers: cheap models for easy stages, frontier for the hard step.
• Local models earn their place when privacy, cost-at-volume, or offline operation matter — and modern open-weight models handle tool calls well enough for pipeline stages and drafts. Serve them behind an OpenAI-compatible endpoint and your harness won't know the difference.
• Long-context quality ≠ long-context size. Two models with the same window degrade differently when it's full. Eval at realistic fill levels, not on toy prompts.

• Prompt caching is the #1 cost lever for loops. An agent re-sends the same system prompt and tool definitions every single turn — exactly the stable prefix providers cache at a fraction of full price. Keep that prefix static (no timestamps, no per-turn edits near the top) and a 30-turn session stops paying full freight thirty times for the same tokens.
• Route by tier. Frontier models for planning, diagnosis, and review; a cheaper tier for the mechanical middle — formatting, extraction, easy pipeline stages. A gateway (Route D) makes the split a config line instead of a rewrite.
• Cap in tokens and dollars. Token budgets miss price changes and model swaps; dollar budgets miss runaway context growth. Set both, per run and per day — budgets are gates that work while humans sleep (Module 05).