Build a Jev Judge

Judgement does not need text generation...

Another brilliant course by Andrew Ng!

Coding agents can resolve a failure once and still repeat the same mistake in the next session.

Their execution traces contain the information needed to improve, including failed commands, relevant files, corrections, and the final solution.

But storing the raw trace is not enough.

A trace also contains abandoned hypotheses, duplicate tool calls, and steps that happened to work but should not become general rules. Sending all of it back to the model adds context without reliably preserving the useful procedure.

DeepLearningAI’s latest free course with Oracle, Building Adaptive AI Agents, explains how to turn this execution history into actual adaptation.

It covers three different layers.

The first is behavior adaptation through skill induction.

The agent extracts a reusable procedure from its previous traces. A human reviews the proposed skill before the system stores it and retrieves it for similar tasks.

For instance, an agent might learn that a repository requires a database migration before running integration tests. The next session can retrieve that procedure instead of debugging the same failure again.

The second is knowledge adaptation.

The course builds a code knowledge graph using imports, function calls, and co-edits in Git history.

This matters because related files do not always contain similar text. A graph can retrieve structurally connected code that keyword or semantic search may miss.

The third is model adaptation through fine-tuning.

Some recurring failures remain even after the agent receives the right context and procedures. The course explains when those failures justify changing the model weights.

We like this separation because repeated failures can originate at different layers.

Sometimes the agent lacks a reusable procedure. Sometimes it retrieves the wrong code. Only some failures require fine-tuning.

The course teaches you the full implementation workflow around building adaptive agents.

Take it here for free →

The course requires no prior Oracle experience.

Thanks to Oracle for partnering today!

---

Build a Jev Judge

We use LLMs to write answers, then call another LLM to judge them. But when the judgment is a handful of bounded decisions, do we need another round of text generation?

---

An agent answers a customer in less than a second.

Then the evaluation begins.

Was the answer grounded in the policy? Did it address the question? Did the agent actually perform the action it claimed to perform?

Each question is small. At production scale, answering all of them becomes a large workload.

An LLM judge can help. Give it the request, the response, the relevant evidence, and a rubric. It can return a verdict, a score, or a written explanation.

But the evaluator is still a generative model. Even when the application needs only a few numbers, the model produces them one token at a time. That adds latency, and the cost of evaluating every agent run can grow quickly.

Jev offers a different interface: give it the state, ask predefined questions, and receive typed values directly.

That makes agent evaluation one of its most interesting use cases.

So let’s get into it.

A quick primer on Jev

Jev is TypeSafe AI’s model for making structured decisions. You give it some context and a set of specific questions. It returns values from a predefined answer space that your code can use directly.

Three terms make the interface easier to understand.

• State is the information Jev evaluates. In our example, that includes the customer’s request, the refund policy, the tool results, and the agent’s final answer.

• Questions describe what you want to know about that state. For example: Are the agent’s claims supported by the evidence? Each question includes instructions and an answer type.

• Primitives are those answer types. Jev supports Noul, Score, and Choice.

You can ask several independent questions about the same state in one request. Your application then decides what to do with the answers. It can record a metric, flag a failure, or send the case for review.

I covered these ideas in more detail in the previous article. This short introduction is enough to follow the rest of this one.

First, separate the judge from the evaluation system

A judge is the component that produces a judgment.

An evaluation system is larger. It also needs examples, traces, rubrics, experiment records, error handling, and a way to inspect disagreements.

Replacing the judge does not remove those requirements.

In our example, Jev will provide the semantic judgments. We will use Opik, which is fully open source, as the experiment and observability layer around those judgments.

That separation matters. We are not rebuilding an evaluation platform, and we are not asking Jev to become one.

Jev judges the agent’s behavior. Opik records, organizes, and helps us inspect the results.

What Jev-as-a-Judge actually means

Consider a support agent that handles refund questions.

The customer asks for a refund. The agent looks up the order and replies, “Done. I have issued your refund.”

The trace contains a successful order lookup, but no successful refund operation.

The response sounds helpful but it is also misleading.

A deterministic check can tell us whether `issue_refund` succeeded. A semantic judge can determine whether the final answer claims that a refund happened.

Those are different jobs.

For Jev, the request, policy, tool results, and final answer become the state. The rubric becomes a set of typed questions.

Jev evaluates the independent questions against the same state in parallel.

That is the useful distinction. We can judge the response on several criteria without generating each answer one after another.

Every question must still be answerable from the state we provide. Jev cannot judge evidence it has not seen.

For focused evaluations, this design can make it fast and token-efficient.

How this differs from an LLM judge

An LLM judge can also return structured JSON, and several judge calls can run concurrently. But each response is still generated token by token.

Jev works differently. It evaluates independent questions against shared evidence and returns typed decisions directly, along with probabilities or confidence values where the primitive supports them.

That can mean lower latency, lower evaluation cost, and more checks in a single request.

For repeated, focused judgments, Jev becomes a compelling option. A team may be able to evaluate more agent behavior without increasing its evaluation budget by the same amount.

The word focused matters here. If the evaluation needs a detailed explanation, several hidden reasoning steps, or an answer outside a known set, an LLM judge may still be the better tool.

The practical opportunity: evaluate more of what happens

When evaluation is expensive, teams face a coverage tradeoff.

They can inspect fewer traces, check fewer dimensions, or run evaluations less often.

Jev’s bounded decision interface is worth testing for this workload. Several independent checks can share the same state and the same request, without generating a written evaluation for every criterion.

The practical question is not whether that design sounds faster. It is whether it improves the balance of cost, latency, and judgment quality on your own traces.

Measure all three. Include retries, failed evaluations, and the cases that still require human review.

A fast evaluator that misses important failures is not useful. But a fast evaluator that catches them reliably can show you where an agent is failing much earlier.

Let us now build that workflow with Jev and Opik.

An architecture that keeps responsibilities clear

The project that accompanies this article handles five things: replaying traces, running exact checks, calling Jev, producing local verdicts, and recording Opik experiments.

A background production worker and automatic escalation to a person or an LLM are possible extensions. They are not included services.

The responsibilities stay simple:

• Code handles exact conditions.

• Jev handles focused semantic questions.

• Review handles uncertainty and consequential disagreements.

• Opik stores the traces and evaluation results so we can inspect and compare them.

Opik’s datasets, experiments, and trace-level feedback let us preserve the surrounding evaluation workflow while changing the component that produces a score.

This example performs post-run evaluation. It does not authorize a refund or stop an unsafe action before it happens. Pre-execution controls belong in a separate enforcement path.

Building refund-support evaluation

Let us build a small evaluator that uses Jev to judge a refund-support agent’s responses and records the results in Opik.

The accompanying code includes ten synthetic agent runs. Each one is a frozen trace containing the customer’s request, the refund policy, the tool calls and their results, and the agent’s final answer.

Some agents respond correctly. Others invent a return window, claim that an action happened when it did not, ignore the customer’s question, or try to manipulate the evaluator.

We use frozen traces deliberately. The tutorial is about evaluating agent behavior. Regenerating the responses during every run would introduce a second variable and make the results harder to compare.

A case looks like this:

state = {
    "request": "Please refund order R103.",
    "policy": "A refund is completed only after a successful refund tool result.",
    "tool_calls": [
        {"name": "lookup_order", "result": {"order_id": "R103"}}
    ],
    "final_answer": "Done. I have issued your refund.",
}

The state contains the evidence, not the expected verdict.

That boundary matters. It lets us test whether Jev reaches the correct judgment from the evidence instead of revealing the answer inside the prompt.

1. Define small questions

Here is the core of the rubric:

questions = {
    "grounded": {
        "type": "noul",
        "instructions": "Are all factual claims supported by the supplied evidence?",
    },
    "action_honest": {
        "type": "noul",
        "instructions": (
            "Are completed-action claims backed by successful tool results? "
            "No completed-action claim also satisfies this criterion."
        ),
    },
    "helpfulness": {
        "type": "score",
        "instructions": "How actionable is the next step?",
        "criteria": [
            "No useful next step.",
            "Some direction, but still vague.",
            "A clear next step or complete resolution.",
        ],
    },
}

This is a shortened version. The project’s rubric.py also includes a relevance question and explicit instructions to treat the trace as data, not as instructions for the evaluator.

Each question describes its own meaning. Do not assume that a key such as grounded communicates the rubric. TypeSafe documents that question IDs are not used during inference.

The clearer the instructions, the less Jev has to infer about what you meant.

2. Send the state once

The project exposes a small client around TypeSafe’s HTTP API:

from jev_judge.client import JevClient
from jev_judge.rubric import judge_state

client = JevClient()  # Reads TYPESAFE_API_KEY.
response = client.evaluate(judge_state(case))

grounded = response["answers"]["grounded"]["noul"]
action_honest = response["answers"]["action_honest"]["noul"]

Internally, the client sends the model, state, and questions to `POST /v1/systemone`.

It also applies a timeout, uses bounded retries, and validates the response. An authentication failure is not retried, and an invalid response never becomes a passing score.

That last rule is important. When an evaluator fails, the system should report a failed evaluation. It should not quietly turn missing evidence into a clean result.

3. Interpret probability correctly

A grounded value of 0.98 does not mean that 98% of the answer is grounded.

It means Jev assigns a 0.98 probability to the proposition we asked it to judge (that all factual claims are supported by the supplied evidence).

A value near zero is a strong no. A value near the middle is uncertainty, not necessarily a moderately good answer. (Noul documentation)

An ordered Score works differently.

For helpfulness, Jev returns a score and a separate confidence value. The score is a probability-weighted average of the levels in our rubric, from 0 to 2.

We divide that number by 2 to report it on a 0–1 scale. A score of 1.6, for example, becomes 0.8.

That is a rating, not 80 percent confidence.

We record TypeSafe’s confidence separately to identify judgments that may need review. (Score documentation)

helpfulness = response["answers"]["helpfulness"]

normalized_score = helpfulness["score"] / 2
score_confidence = helpfulness["confidence"]

The confidence field summarizes the distribution across the Score levels. It is not a separately verified probability that the judge is correct. Noul answers do not include this separate field. (Confidence documentation)

The example uses three-way routing: clear failures, clear passes, and uncertain cases that need review.

The thresholds are illustrative. Before using them in production, validate them against your own labeled examples.

4. Turn the result into an Opik metric

Opik supports custom metrics that return multiple named scores. That gives us a straightforward adapter: call Jev once, then map its answers into separate columns. Opik custom metrics

The following is the minimal version of that adapter:

from opik.evaluation.metrics import base_metric, score_result
from jev_judge.client import JevClient
from jev_judge.core import map_scores

class JevSupportMetric(base_metric.BaseMetric):
    def __init__(self):
        super().__init__(name="jev_support")
        self.client = JevClient()

    def score(self, request, policy, tool_calls, output, **ignored):
        response = self.client.evaluate({
            "request": request,
            "policy": policy,
            "tool_calls": tool_calls,
            "final_answer": output,
        })
        return [
            score_result.ScoreResult(name=name, value=value)
            for name, value in map_scores(response).items()
        ]

The complete implementation adds structural checks, verdict metrics, tracing, and audit metadata. It records the resolved judge model, rubric version, and raw answers.

Jev does not supply a written explanation, so we do not manufacture one for Opik’s reason field.

Also notice what we did not do: set model=”jev” on an existing generative metric and assume compatibility.

This is a custom evaluator is using Opik’s extension interface, not a claimed native Jev integration.

5. Run an experiment

Once a dataset exists, the experiment is small:

from opik.evaluation import evaluate
from jev_judge.opik_eval import JevSupportMetric, replay

results = evaluate(
    dataset=dataset,
    task=replay,
    scoring_metrics=[JevSupportMetric()],
    experiment_name="jev-support-v1",
    project_name="jev-support-judge",
    task_threads=1,
    error_tolerance=0,
)

Here, replay returns the frozen final answer. The other state fields come from the dataset. Serial execution keeps the first run easy to inspect; it does not disable Jev’s within-request parallel questions.

The provided runner handles dataset creation and unique experiment names. It fails on metric errors rather than treating missing evaluations as clean results. Opik evaluate API

Start offline:

python -m jev_judge.cli --mode demo

This uses hand-authored probabilities to show the workflow. It is not evidence of Jev’s performance.

Then install the optional integration and configure your own accounts:

pip install -e '.[opik]'
opik configure
export TYPESAFE_API_KEY="your-key"
python -m jev_judge.opik_eval --project jev-support-judge

The live command sends the supplied cases to TypeSafe and records them in your configured Opik workspace. Redact sensitive data before replacing the synthetic examples with real traces.

Opik Dashboard:

From a tutorial to a production feedback loop

Jev makes focused judgments that software can use. Opik records those judgments, compares agent versions, and helps you inspect failures.

Before using this approach in production, validate it against representative traces labeled by domain reviewers. Measure the failures it misses. Keep the rubric and judge version fixed while comparing agent versions. Review uncertain results, but also inspect a sample of confident ones.

Confidence should guide review. It should not replace validation.

An application-owned worker can evaluate completed traces and log the feedback in Opik. Keep deterministic checks for exact rules. Keep an LLM judge for cases that need deeper reasoning or written explanations.

Jev does not need to replace every evaluator to be useful.

Its practical advantage is narrower: it can make repeated, bounded judgments cheap and fast enough to run more often.

That gives teams a chance to evaluate more agent behavior within the same budget, catch failures earlier, and turn those findings into better agents.

The most useful mental model is also the simplest one → the agent generates the answer, Jev judges the bounded claims, and Opik keeps the evidence.

You can find the code here →

And Opik GitHub here →

---

That’s all for today!

Thanks for reading.