Agent Harness Architecture: Runtime Controls Around an AI Agent
Watch on YouTube: Agent Harness Architecture: The Primitives That Make AI Agents Reliable
A useful way to think about an agent is:
Agent = Model + Harness
The model produces actions or text from its current context. The harness is the application around it: it provides tools, state, execution limits, and permission boundaries. A new model call has no durable memory unless the application supplies it.
For many product teams, the harness is where most reliability work happens. The Harness, Not the Model makes the same case from a different angle.
What a bare loop can get wrong
The source video describes several failure modes of an agent running without enough structure. It may try to finish a whole project in one turn, stop after partial progress, spend a new session rediscovering the setup, or leave a half-finished environment for the next run. None of these outcomes is inevitable, but each becomes more likely when plans, state, and checks live only in the prompt.
A graph is one way to add structure
A graph-based harness can represent work as a directed acyclic graph rather than an implicit serial loop. In such a design, dependencies can block later work until earlier work completes; independent steps can run in parallel; error handling can follow an explicit retry and escalation policy; and replanning can be recorded instead of silently changing the plan.
One useful breakdown has six parts:
- An agent loop that assembles context, invokes the model, and applies step limits.
- A tool system that exposes capabilities.
- A memory manager for working, episodic, semantic, and procedural state.
- Context engineering that selects relevant context within a token budget.
- A sandbox that enforces permission boundaries.
- Human approval gates for actions that need review.
The exact components depend on the product. The point is to make runtime policy explicit rather than hoping the model will infer it every time.
Four practical primitives
Tool execution and fallbacks
Tool failures often fall into transient cases, such as timeouts or rate limits, and persistent cases, such as invalid arguments or malformed output. A harness can retry transient failures with a backoff policy and use a fallback for persistent ones. It should return a structured error to the model instead of allowing one failed call to end the whole run without context.
Context and continuity
Some context can be loaded by the application at the start of a run, such as session memory. Other context can be requested by the model through search or summary tools. The goal is not an “optimal” context window in the abstract. It is enough relevant information for the current action without spending the whole token budget on history.
Sandboxes and tests
Models should not be the only judge of their own output. A harness can run code through independent tests inside a sandbox and return the result to the agent. Those results give evidence about the checks that ran. They do not prove that every possible failure is absent.
Bounded escalation
The video proposes a three-stage policy. Infrastructure failures can be retried by the harness. Validation failures can be returned to the model with a limited repair budget. When that budget is exhausted, the system can stop the tool loop, replan the graph, or ask for human approval. The value of this policy is that recovery has a visible limit instead of becoming an endless retry loop.
Reliability is implemented outside the prompt
Prompting can guide an agent, but it does not reliably enforce runtime controls. Permission boundaries, state persistence, independent tests, and retry limits need an implementation that applies them. Stronger models can improve the quality of decisions inside those controls; they do not remove the need for the controls themselves.