What It Takes to Build a Production Agent Harness

A hands-on nanodegree for production agent engineering.

Building a production agent harness in LangChain

An agent needs more than a model and a collection of tools. It also needs an execution layer that manages how the model, tools, and application state interact.

We have published Part 1 of a new hands-on series where we build this layer using LangChain and LangGraph.

Read Part 1 of the series here →

This chapter covers:

• The responsibilities of an agent harness

• How models, messages, prompts, and tools fit together

• How tool calls are generated, executed, and returned

• How the harness determines whether another model call is required

• How state moves through the execution lifecycle

• Where LangChain abstractions fit into the implementation

• How these components combine into a working agent

Everything is explained from scratch, with the corresponding implementation.

Start reading Part 1 here →

---

Why care?

The behavior of an agent is determined by more than the model.

Consider a tool that fails halfway through a run. The surrounding system must decide whether to retry it, expose the failure to the model, request human input, or stop the execution. Similar decisions apply to message history, persistent state, permissions, and interrupted runs.

These responsibilities belong to the agent harness.

This is why production agent platforms expose capabilities such as sessions, checkpoints, tracing, evaluation, human approval, and sandboxed tool execution. Each capability addresses a problem that appears when an agent moves from a single request to a stateful application.

For an AI engineer, this introduces a set of system design questions:

• Which state belongs to the current run?

• Which state should persist across conversations?

• How are tool errors represented and propagated?

• Which transitions can be model-driven?

• Which transitions should remain deterministic?

• How can an interrupted execution resume without repeating completed work?

LangChain provides interfaces for models, messages, and tools. LangGraph provides explicit state transitions and execution control. The series uses both to implement these concepts and examine the behavior of the resulting system.

Part 1 establishes the execution model. The upcoming chapters will extend it with graph-based control, persistence, memory, failure handling, and other production requirements.

Read the first implementation of the production agent harness here →

👉 Over to you: Which part of an agent harness has been the most difficult to implement?

---

Your agent harness should repair itself

An agent can complete a difficult task, save the procedure that worked, and reuse it later.

That is a useful form of runtime learning because the improvement lives outside the model weights and remains available across sessions.

Production failures create a different problem. The useful information is spread across the execution trace, tool results, local code, configuration, and the final outcome.

A patch may fix the incident, but the failure only becomes reusable knowledge when the system also records the expected behavior and tests it again.

This gives agent self-improvement two separate feedback paths:

• successful runs can produce reusable procedures

• failed runs can produce reviewed fixes and regression cases

Hermes implements the first path through persistent skills and offline optimization. Its Opik integration provides the trace data needed for the second.

Let’s look at where the two paths differ and how they fit together.

---

Agent improvement without weight updates

A model does not need to update its weights for an agent system to improve.

The agent can instead change the artifacts around the model. These include its instructions, skills, tool descriptions, retrieval policy, configuration, and evaluation cases. If a later run uses those changes, the system’s behavior has changed even though the underlying model is identical.

That distinction matters because agent failures rarely map to one bad final answer. An agent makes a sequence of decisions. It selects tools, constructs arguments, interprets results, updates its plan, and decides when to stop.

A useful learning record therefore needs more than the final output. It needs enough evidence to identify which decision failed and enough structure to alter future runs.

Procedural memory in Hermes

Hermes Agent treats skills as procedural memory. After completing a complex task, it can preserve the useful method in a SKILL(.)md file. A later task can load that skill instead of deriving the same procedure again.

The saved artifact is readable and editable. It can specify when the procedure applies, which tools to use, what sequence to follow, and which constraints must remain true. This makes the learned behavior easier to inspect than a weight update.

Hermes also manages this collection over time. Its skill curator can revise weak descriptions, merge overlap, and remove stale instructions. That maintenance is necessary because an unbounded skill directory eventually creates its own retrieval problem. The agent must still select the right procedure from everything it has saved.

Hermes has a second path for improving these artifacts through Hermes Forge. It uses GEPA, or Genetic-Pareto Prompt Evolution, to search over variants of prompts, skills, and tool descriptions.

GEPA runs outside the live agent loop. It evaluates candidate variants against a defined task set, uses execution feedback to propose new candidates, and keeps variants that improve the objective without violating constraints. This is optimization over text and configuration, not GPU fine-tuning.

These two mechanisms solve related but different problems:

• runtime skill creation preserves a procedure that worked during a task

• offline evolution compares variants against an evaluation set

Limits of the Hermes skill loop

A procedure saved after a successful run inherits the agent’s judgment about that run. The task may have completed with a brittle method, an unnecessarily expensive route, or an assumption that happened to hold once.

Automatic skill updates carry a related risk. A generated revision can overwrite a carefully written instruction with a weaker one unless the system versions changes and evaluates them before promotion.

Offline optimization has another boundary. Its result depends on the cases and objective supplied to it. A failure that only appears under production inputs will not influence the optimizer until someone captures that case and adds a useful evaluation signal.

A trace records the incident, but it does not convert the incident into changed behavior.

Production failure feedback with Opik

A production incident becomes reusable after the system records four pieces of information:

1. the first step that diverged from expected behavior

2. whether the fault came from the model, a tool, configuration, or application code

3. the change that corrects the fault without hiding another problem

4. the test that will fail if the same behavior returns

Opik (GitHub Repo) connects these records in one debugging workflow. Its Hermes plugin records each agent turn as a root trace, with separate spans for model and tool calls. Ollie can inspect the trace alongside a connected project, propose a code change, rerun the original input, and add the case to a Test Suite.

The full sequence is:

Failed trace -> diagnosis -> proposed diff -> human approval -> rerun -> regression test

Each stage produces evidence for the next one. The workflow does not require the agent to edit production code without review, and it does not treat a plausible patch as proof that the incident is resolved.

Execution traces

The opik-hermes plugin captures a root trace for each Hermes turn. Model spans include the messages, output, provider, token usage, and cost. Tool spans include the tool arguments and returned results.

This structure matters because the visible error often occurs after the actual fault. A final answer may be wrong because an earlier search returned nothing, a tool received malformed arguments, or the agent accepted an invalid intermediate result. The span tree preserves that causal order.

In the image above, we used a Hermes agent for a request about Google’s Gemma model family. Opik recorded the model and tool activity under the same turn, so the complete route was available without reconstructing it from terminal output.

Failure detection

Some failures are explicit exceptions. Others complete normally but produce a poor tool choice, an incomplete answer, excessive latency, or an unexpected cost.

Opik can surface candidate failures through error status, feedback scores, online evaluation, latency, and cost. Its alert system can apply thresholds over these signals and send the resulting event to Slack, PagerDuty, or a general webhook.

Detection only identifies runs that need investigation. It does not determine the root cause or decide which artifact should change.

Trace and source-level diagnosis

A trace explains what happened at runtime. Root-cause analysis often requires the source and configuration that produced it.

Running opik connect from the project directory gives Ollie session-scoped access to that project. Ollie can inspect relevant files, compare them with the failing span tree, and propose a Git-style diff. File writes still require explicit approval.

In my test, I asked Hermes to install pip-audit and scan the current Python environment:

Install the pip-audit tool and run a security
vulnerability scan on the current Python environment.

The run failed during environment validation. The trace showed the failure, while the connected configuration explained its source. The execute_code tool used an unsupported cloud_runner environment value.

Ollie proposed a small configuration change and explained which validation error it addressed. The diff remained pending until I approved it.

The Hermes agent's execute_code tool is
configured with environment type cloud_runner,
which is not a supported value.

Ollie performs the trace and code analysis. The developer decides whether the diagnosis and diff are valid.

Reruns and Agent Sandbox

Approval permits a change to be applied. Verification requires another execution.

Ollie can rerun the agent through opik connect using the original trace input. The updated execution returns to Opik as a new trace, which provides a direct comparison with the failing run.

After approving the environment configuration change, I reran the same pip-audit request. The first execution stopped during validation. The second completed with a supported environment configuration.

Opik’s Agent Sandbox covers broader interactive testing through the Agent Playground. The opik endpoint command runs the local agent and makes it available in the Opik UI. Prompts, model settings, and tool definitions can then be tested while each execution generates a complete trace.

The two commands have different responsibilities:

• opik connect gives Ollie access to the connected project for source inspection, approved file changes, and reruns

• opik endpoint runs the agent locally and connects it to the Agent Playground for interactive testing

They can run together when a debugging session needs both code-aware diagnosis and controlled test executions.

For the Hermes example, the initial pip-audit request failed during environment validation. After the configuration change was approved, the same request completed successfully. The second trace verified the runtime behavior of the fix instead of relying on a diff that only appeared correct.

Regression tests

The failed input becomes reusable only when the system records what should happen next time.

Opik Test Suites express those expectations as natural-language assertions. An LLM judge checks each assertion against the agent output and reports pass or fail. Execution policies can also require multiple successful runs, which helps when the agent is nondeterministic.

The following suite preserves the three Hermes tasks from the original test set. run_hermes is the adapter between an Opik test item and the Hermes CLI.

import subprocess

import opik

def run_hermes(item):
    process = subprocess.run(
        ["hermes", "chat", "-q", item["request"]],
        capture_output=True,
        text=True,
        check=False,
    )
    return {
        "output": process.stdout,
        "error": process.stderr,
        "exit_code": process.returncode,
    }


client = opik.Opik()
suite = client.get_or_create_test_suite(
    name="Hermes Agent Regression",
    project_name="Hermes Agent",
    global_assertions=[
        "The requested task completes without an execution or environment validation error.",
        "The response directly provides all information requested by the user.",
    ],
    global_execution_policy={"runs_per_item": 2, "pass_threshold": 2},
)

suite.insert(
    [
        {
            "data": {
                "request": "Check if the 'requests' package is installed and print its version."
            },
            "assertions": [
                "The response states whether requests is installed and prints its version when available."
            ],
        },
        {
            "data": {
                "request": "List the files in the current directory and identify the project."
            },
            "assertions": [
                "The response lists the current directory contents and identifies the project."
            ],
        },
        {
            "data": {
                "request": "Run Python and print the interpreter version together with the first ten installed packages."
            },
            "assertions": [
                "The response prints the Python interpreter version and the first ten installed packages."
            ],
        },
    ]
)

result = opik.run_tests(test_suite=suite, task=run_hermes)
print(f"Pass rate: {result.pass_rate:.0%}")

The suite does not teach Hermes by itself, but rather the combined Hermes and Opik workflow supplies a durable evaluation signal for prompt revisions, tool updates, configuration changes, and evolved skills.

Each candidate change can be checked against the same failure before release.

Combined Hermes and Opik workflow

Hermes and Opik change different artifacts at different points in the agent lifecycle.

Hermes preserves methods that proved useful during execution. Its offline optimizer can compare new skill, prompt, and tool-description variants against a task set.

Opik supplies production evidence. It records the execution, supports trace-aware diagnosis, keeps code changes behind approval, reruns the original input, and stores the expected behavior as a regression case.

That produces a more complete loop:

1. Hermes completes tasks and saves reusable procedures.

2. Production runs generate traces and feedback in Opik.

3. A failed trace is diagnosed against the project source.

4. A developer reviews the proposed change.

5. The original input is rerun against the updated agent.

6. The failure and expected behavior become a test case.

7. Later changes are evaluated against the accumulated suite.

The system is not autonomously learning from every production incident. It is turning production incidents into structured, reviewable inputs for improvement. That narrower claim is also the useful one.

Hermes and Opik setup

Hermes supports Linux, macOS, WSL2, and native Windows. The following flow uses the managed Linux, macOS, or WSL2 installation.

Install Hermes

curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash

source ~/.bashrc  # or ~/.zshrc

hermes setup

The setup wizard configures the model provider, credentials, and enabled tools.

Install the Opik integration

Install opik-hermes in the same Python environment as Hermes:

~/.hermes/hermes-agent/venv/bin/python3 -m pip install opik-hermes

The package registers the plugin entry point and installs the Opik SDK dependency.

Enable trace collection

Add the plugin to ~/.hermes/config.yaml:

plugins:
  enabled: [opik]

For a local Opik deployment, add the endpoint and project name to ~/.hermes/.env:

# .env file

OPIK_URL_OVERRIDE=http://localhost:5173/api
OPIK_PROJECT_NAME=Hermes Agent

For Comet-hosted Opik, configure the workspace and API key instead:

# .env file

OPIK_API_KEY=...
OPIK_WORKSPACE=your-workspace
OPIK_PROJECT_NAME=Hermes Agent

Verify trace collection

Restart Hermes after enabling the plugin. Each turn should now appear in the configured Opik project.

hermes chat -q "List the files here and report how many there are."

The trace should contain one root record for the turn, model spans, and a span for each tool call.

Connect the project to Ollie

Run the bridge from the agent project directory:

opik connect --project "Hermes Agent"

The connection is opt-in and scoped to the launched session. Ollie can inspect files within the connected project and must request approval before writing changes.

Evaluation and release constraints

This workflow improves the feedback pah, but it does not remove the hard parts of agent evaluation.

• Failure selection still matters. A timeout, a bad answer, and an invalid tool call require different assertions. Converting every anomalous trace into a test will produce a noisy suite.

• LLM judges are not ground truth. Natural-language assertions are convenient for behavioral checks, but deterministic validators remain preferable for schemas, exit codes, database state, and exact tool arguments.

• A passing rerun proves one case. The patch can still affect other tasks. Broader suites and repeated execution are needed before treating it as a safe release.

• Learned artifacts need versioning. Skills, prompts, tool descriptions, configuration, and tests should remain tied to the change that introduced them. Otherwise, the team cannot explain why behavior changed or roll back a weak update.

These constraints are part of the design, not exceptions to it. Agent improvement becomes reliable when the feedback is observable, the change is reviewable, and the result is tested against the behavior that originally failed.

Run Opik locally

Opik is Apache-2.0 licensed, and the complete platform can run inside your own infrastructure. The self-hosted deployment includes the backend, web application, tracing, datasets, evaluations, prompt management, and agent optimization components.

On Linux or macOS, the local stack starts with three commands:

git clone https://github.com/comet-ml/opik
cd opik
./opik.sh

The script starts the full Opik suite. Once the services are healthy, the interface is available at http://localhost:5173. The Hermes plugin can then send model and tool spans to this local deployment using the OPIK_URL_OVERRIDE configuration from the setup above.

Check out Opik on GitHub →

(don’t forget to star 🌟)

👉 Over to you: Which production failure would be most valuable to preserve as a regression case for your agent?

Good day!