Harness Engineering

One task, followed through every part

The model can write the fix. It cannot run anything.

A language model takes text in and gives text out. It can suggest the fix for test_invoice.py, but on its own it cannot open the file, run pytest, or decide whether it is done. Everything that turns its text into actions is the harness: "the runtime that couples an LLM to the world—its loop, its tools, its context, its safety controls, its orchestration, and its extension surfaces" p. 3.

By mid-2026 every major AI lab shipped one of these, and the term "harness engineering" was barely five months old (first used by Mitchell Hashimoto, then defined by LangChain's Vivek Trivedy in February 2026) p. 3. What was missing was a reference: a description, from real source code, of what a harness is made of. That is this paper. It reads eleven production harnesses at source level and does not benchmark or rank them; it describes how they are built p. 1.

At the core: ask the model, run what it asks for, repeat

Every system in the study runs a variation of the ReAct loop: the model thinks, asks for an action, the harness runs it, and the result goes back into the conversation p. 13. The smallest real example is Mini-SWE-Agent, whose loop fits on one screen. Step through it with our task.

def run(self, task):
    self.add_messages(system_prompt)
    self.add_messages(user_prompt(task))
    while True:
        result = self.step()
        if result.role == "exit":
            return result
def query(self):
    self.check_limits()
    msg = self.model.query(self.messages)
    self.add_messages(msg)
def execute_actions(self):
    for action in msg["extra"]["actions"]:
        obs = self.env.execute(action)   # subprocess.run()
        self.add_messages(observation(obs))

Condensed from the paper's Listing 1 p. 14; the project README describes "just some 100 lines of python for the agent class", bash as the only tool, and a completely linear history repo.

A 100-line loop scores like the big ones. So why are the big ones huge?

Mini-SWE-Agent reports 74%+ on SWE-bench Verified, in the same range as systems three orders of magnitude larger p. 5 (self-reported, different models and dates, so not a clean comparison). The paper's first observation: loop sophistication does not predict benchmark performance Obs. 1, p. 12. So is the rest of a production harness wasted code?

Every harness takes a position on the same seven parts

From the 100-line baseline to the million-line Codex, every system "must take a position on the same seven subsystems … even when that position is deliberate absence" p. 4. For each, the paper records the smallest and the largest version found.

Minimal and maximal forms from Table 1 p. 5. Chapters 6–12 take them one at a time.

The hard part of a loop is knowing when it is stuck

The paper finds three loop shapes p. 13: the plain action–observation loop (nine systems), the reflection loop (Aider lints and tests after each edit and re-asks the model, up to 3 times), and a coordinator–worker overlay where a lead agent dispatches workers (Claude Code, Codex; configuration-gated in Hermes).

Suppose our agent keeps running pytest test_invoice.py without changing anything. Each system reacts differently, and you can try that in the Play section of the Gist tab.

Split the prompt so most of it never changes

Every turn re-sends the system prompt. Providers can cache a prompt prefix that is byte-identical to last time, so harnesses arrange the prompt around that. Claude Code splits its dozen-plus sections at a SYSTEM_PROMPT_DYNAMIC_BOUNDARY: identity, rules, tool guidance and tone go before it and stay cached; session guidance, memory, environment and MCP instructions go after it and are recomputed each turn p. 21. OpenHands arrived at the same static/dynamic split independently.

Try it: tokens the model must process

How many tools, and how to edit a file safely

Deferred tool loading

Every tool's schema costs prompt space. Claude Code marks most of its 43 tools as deferred: they stay out of the prompt until the model searches for them with a ToolSearch tool, which cuts the initial prompt by about 40% Table 18, p. 77. Codex and Hermes later added the same idea with BM25 search. The paper recommends it once you pass about 15 tools Rec. 4.

File editing: the most consequential tool

The eleven systems use eight different editing strategies Table 7, p. 28. For our task, the agent must change one line in invoice.py:

When the conversation outgrows the window, summarize the old part

Long debugging sessions eventually overflow the context window. Four strategies appear Fig. 4, p. 31: keep everything (Mini-SWE-Agent), recursive summarization (Aider), pluggable condensers (OpenHands), and threshold compaction, used by seven of the eleven, including all four provider-native systems.

The new frontier: who writes long-term memory?

Compaction has converged. What differs now is cross-session memory, and the paper's Observation 5 sorts it by who writes it and who reviews it p. 34:

And for the repository itself, every system auto-discovers Markdown context files (CLAUDE.md, AGENTS.md, GEMINI.md) from the project up to the root, often loading nested ones just in time when a tool touches that folder (§9.7).

Every command passes through checks before it runs

Safety is where the systems differ most. Claude Code uses three layers: static hook rules, then an LLM permission classifier, then an interactive dialog Fig. 5, p. 35. Codex grew to four p. 36. Follow one command from our task through Codex's stack.

Try it: what would stop each command?

Hand slices of the task to sub-agents with their own context

Nine of the eleven systems can spawn sub-agents, in six patterns from sequential delegation to recursive composition p. 39. Claude Code's coordinator mode is the most structured.

Skills overtook MCP

Two standards compete. MCP is a wire protocol: the agent talks to a separately running server that exposes tools. Skills are a file convention: a folder with a SKILL.md (instructions plus YAML front matter) that the agent reads when relevant. By July 2026 skills led 9 to 8, because Pi adopted skills while rejecting MCP ("build CLI tools with READMEs") Obs. 8, p. 51.

Eight of the nine skill adopters load only each skill's name and description up front and fetch the body on demand, the same trick as deferred tools. Some skills activate only when the agent touches matching files (Claude Code's paths, OpenHands' PathTrigger). And skills now have a supply chain: registries in four systems, trust tiers and quarantine in Hermes, and the first skills written by agents themselves.

No agent frameworks. No vector RAG over code.

The authors grepped about four million lines of Python, TypeScript and Rust for imports of LangChain, LangGraph, LlamaIndex, AutoGen, CrewAI, Pydantic AI, Genkit, Google ADK, Semantic Kernel and more. No production agent code path imports any of them. Gemini CLI uses neither of Google's own frameworks p. 53. Every loop is hand-written in the language's async primitives.

A parallel search for vector stores and embedding libraries found zero systems that retrieve code with embeddings. They use ripgrep, glob, tree-sitter and auto-discovered Markdown files. Embeddings appear only for chat memory, on by default in exactly one system (OpenClaw) p. 54–56.

The platform turn

The paper's thesis is that in the first half of 2026 the harness went from tool to platform Obs. 12, p. 63. Harnesses became importable SDKs (Claude Agent SDK, openai-codex), framework vendors shipped harnesses (LangChain's Deep Agents), plugin marketplaces appeared, Codex shipped an importer for Claude Code's sessions and settings, and Databricks' Omnigent now orchestrates 23 harness adapters behind one API. That also resolves the framework puzzle: harnesses did not adopt frameworks, they replaced them.

What backs these claims

This is a source-code study, not a benchmark. Its evidence is what the code contains, counted across systems:

Independent evidence that structure matters

The paper leans on a separate study, Agentic Harness Engineering (AHE), which evolves a harness automatically from a bash-only seed to 71.9% on SWE-bench Verified. Its ablation shows which parts carry the gain p. 66:

Self-reported SWE-bench Verified figures from spring 2026, which the authors removed from their main table as not comparable: OpenHands 77.6%, Mini-SWE-Agent 74%+, Claude Code 72.7%, Codex 69.1% (p. 11).

How far to trust it

• It reads code, it does not run it. No claims about speed or quality; benchmark numbers are the systems' own (p. 66).
• Claude Code comes from a circulated March 2026 source snapshot, not an official release, the weakest link for reproducibility (p. 67).
• Inventory claims decay in weeks. Three April observations had to be revised by July; tool counts and versions will be out of date. Structural claims (the seven parts, the absences) have held up (p. 63).
• Scoring involves judgment calls, and the Anthropic-guidance match (Obs. 10) shows correlation, not causation.
• The paper was written with substantial help from Claude (via Claude Code), with findings verified against the code, per its disclosure (p. 74). my take worth knowing when it praises Claude Code's design.

The paper is only a few months old, so it's too early to say what it changed. What it gives you today is a shared map: one vocabulary for parts that every agent has, and evidence from real source code instead of opinion.

What to remember. A coding agent is a small loop surrounded by a large amount of engineering whose job is not to make the model smarter but to make it safe, affordable, recoverable and extensible. The field converged fast: the same seven parts, hand-rolled loops, deterministic retrieval, threshold compaction, coordinator–worker sub-agents and skills. And it copies itself: Codex adopted Claude Code's hook vocabulary verbatim; Hermes' code comments credit OpenCode, Codex and OpenClaw p. 62.

Since the paper

Gemini CLI's consumer tier was retired in June 2026 in a move toward the closed-source Antigravity CLI, so the paper's v0.50.0 pin captures the last major open-source state of that lineage (p. 8). Controlled harness × model studies have started (see the update card under What they found, in the Results tab).

What the authors recommend

The paper closes with these 18 recommendations and a 90-line scaffold that implements 10 of them (Listing 3, p. 71–72). The scaffold deliberately omits, because the right answer depends on context: sandboxing, multi-agent orchestration, MCP and skills. "Start here, measure, add the minimum that your observed failure modes demand" (p. 72).

Verdict