How to Evaluate AI Agents in Practice: Trajectories, Tool Calls and Judges

To evaluate an AI agent, grade the path as well as the answer. Run deterministic checks on which tools it called, in what order, with what arguments and within what budget. Save the LLM judge for outcomes that code cannot check, and show that judge the trajectory. Pydantic Evals, LangChain's agentevals with LangSmith, Mastra and Braintrust all support this layering now, so pick by your stack rather than by feature list. If you still rely on OpenAI's hosted Evals, existing evals go read-only on October 31, 2026, two weeks after this post.

Last updated: October 17, 2026

This guide builds one small refund agent and evaluates it four ways. The order runs from the cheapest check to the most expensive one. Every snippet was either run offline or type-checked against the package versions named in its section. No paid model calls were made, and each section says which of the two applies.

Why final-answer evals miss agent bugs#

A reply such as "Your $42.00 refund for order 12345 is on its way" can look perfect while the run behind it was wrong. The agent may have refunded without looking the order up first. It may have called the refund tool twice, or used twelve tool calls on a two-step task. A string match or a judge that reads only the final message will pass all three of those runs.

OpenAI's agent evals guide describes a trace as the end-to-end record of model calls, tool calls, guardrails and handoffs for one run. It calls trace grading the fastest way to find workflow-level problems, with questions like "did the agent pick the right tool?" LangSmith's complex agent tutorial recommends trajectory evals for a related reason: they give partial credit when an agent gets some steps right but misses the final answer, so you can see where it went wrong instead of only that it did. The same problem applies to agents that say they finished work they did not finish, which we covered in the response looked right is not completion.

%% title: The agent eval stack, cheapest checks first
flowchart TD
    A[Agent run] --> B[Trace: model calls, tool calls, args]
    B --> C{Deterministic checks}
    C -- fail --> X[Block the change]
    C -- pass --> D{Trajectory match}
    D -- below bar --> X
    D -- pass --> E{LLM judge on outcome}
    E -- fail --> X
    E -- pass --> F[Ship, then sample live traffic]
    F --> G[Online evaluators]
    G --> H[Add failures to regression dataset]
    H --> A

The four layers of an agent eval#

1. Tool correctness. Were the right tools called, and were forbidden tools avoided?
2. Trajectory. Did the calls happen in an acceptable order? You can score that with exact, in-order or any-order matching, or with superset and subset rules.
3. Arguments and budgets. Did each tool get the right arguments? Did the run stay under a maximum number of tool calls and model requests?
4. Outcome judge. Is the final answer good? An LLM judge scores it against a rubric, either with no reference answer or against one.

Layers 1 to 3 are plain code. They are deterministic, cost nothing to run, and give the same result every time, so they can run on every case in every experiment. Layer 4 costs money, adds variance, and needs calibration. Here is where each layer lives in the four frameworks covered below:

Each framework has a different strength. Pydantic Evals ships five built-in deterministic agentic evaluators, and their reason strings show the arithmetic behind each score. agentevals lets you override argument matching per tool, down to a comparator function. Mastra puts pass/fail gates and scored thresholds into Vitest assertions. Braintrust's trace-scope scorers can read every span of a run, which suits teams that already store traces there.

Choosing a trajectory match mode#

The match mode decides what counts as a regression, so pick it per case on purpose.

agentevals has four modes, described in its README. strict requires the same messages and tool calls in the same order. unordered requires the same tool calls in any order. subset and superset match partial trajectories. With superset, the agent must make every reference call and may make extra calls. With subset, the agent may make only calls that appear in the reference.

Pydantic Evals' TrajectoryMatch returns a score instead of a pass/fail. exact gives 1.0 only when the sequences are equal. in_order computes an F1 score from the longest common subsequence (precision = LCS / len(actual), recall = LCS / len(expected)), so extra calls can sit between expected ones but lower precision. any_order computes F1 over the multiset intersection. The docs' worked example compares expected [a, b, c] with actual [a, x, b]: the LCS has length 2, so precision and recall are both 2/3 and F1 is about 0.667.

%% title: Trajectory match modes on the same run
flowchart LR
    R["Reference: lookup_order, issue_refund"] --> M{Match mode}
    A["Actual: lookup_order, check_policy, issue_refund"] --> M
    M --> S1["strict: fail"]
    M --> S2["unordered: fail"]
    M --> S3["subset: fail"]
    M --> S4["superset: pass"]
    M --> S5["Pydantic in_order: F1 0.8"]

The four agentevals results in that diagram come from running the Step 2 example below with each mode. The Pydantic score comes from an equivalent three-call run in Step 1. A rule of thumb:

• Extra exploratory calls are fine. Use superset, ToolCorrectness(allow_extra=True), or Mastra's checks.toolOrder, which accepts other calls in between.
• An extra call is itself a bug, for example any destructive tool. Use subset, Pydantic's default allow_extra=False, or checks.didNotCall.
• Policy requires an order, such as lookup before refund. Use strict, TrajectoryMatch(order="exact"), or Mastra's ordering: "strict".

Step 1: Deterministic trajectory checks in Python (Pydantic Evals)#

Pydantic Evals is a code-first Python framework. It does not depend on pydantic-ai, but the agentic evaluators read OpenTelemetry spans, so you need the logfire extra and an instrumented agent. In its data model, a Dataset holds Cases, and dataset.evaluate_sync(task) runs one task against every case and returns a report.

python -m venv .venv && source .venv/bin/activate
pip install 'pydantic-evals[logfire]' pydantic-ai-slim

import logfire
from pydantic_ai import Agent
from pydantic_ai.models.test import TestModel
from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import (
    ArgumentCorrectness,
    MaxToolCalls,
    ToolCorrectness,
    TrajectoryMatch,
)

# Spans stay local; the evaluators read them from the span tree.
logfire.configure(send_to_logfire=False)
logfire.instrument_pydantic_ai()

# TestModel calls every registered tool and needs no API key.
# Swap in a real model (for example 'openai:gpt-5') for realistic arguments.
agent = Agent(TestModel(), system_prompt="You handle refund requests.")


@agent.tool_plain
def lookup_order(order_id: str) -> str:
    """Look up an order before any refund decision."""
    return f"order {order_id}: delivered, $42.00, refundable"


@agent.tool_plain
def issue_refund(order_id: str, reason: str) -> str:
    """Issue a refund for an order that was already looked up."""
    return f"refund issued for {order_id}"


async def refund_task(message: str) -> str:
    result = await agent.run(message)
    return result.output


dataset = Dataset(
    name="refund-agent",
    cases=[
        Case(
            name="refund_after_lookup",
            inputs="Order 12345 arrived broken, please refund it.",
            evaluators=(
                ToolCorrectness(expected_tools=["lookup_order", "issue_refund"]),
                TrajectoryMatch(
                    expected_trajectory=["lookup_order", "issue_refund"],
                    order="in_order",
                ),
                ArgumentCorrectness(
                    tool_name="issue_refund",
                    expected_arguments={},  # tighten with a real model, e.g. {"order_id": "12345"}
                ),
            ),
        ),
    ],
    evaluators=[MaxToolCalls(max_calls=4)],
)

if __name__ == "__main__":
    report = dataset.evaluate_sync(refund_task)
    report.print(include_input=True, include_output=False, include_reasons=True)

We ran this offline with pydantic-evals 2.57.0, pydantic-ai-slim 2.57.0 and logfire 5.2.0. In that version Dataset requires name=, so leaving it out raises a TypeError. All four checks passed. The reason strings show how each score was computed:

TrajectoryMatch => 1.0 | LCS=2, precision=2/2=1.000, recall=2/2=1.000, F1=1.000
MaxToolCalls    => True | 2 tool call(s), budget=4

The model here is TestModel, which calls every registered tool and fills string arguments with "a". That makes it good for testing the eval wiring and useless for testing argument quality. When we set expected_arguments={"order_id": "12345"}, the check failed with a precise reason:

ArgumentCorrectness => False | Tool 'issue_refund' argument mismatch: key 'order_id': expected '12345', got 'a'

We also registered a third tool, close_account, to see how an unexpected call is caught. The cheap checks flagged it:

ToolCorrectness => False | unexpected tools: 'close_account' (x1)
TrajectoryMatch => 0.8   | LCS=2, precision=2/3=0.667, recall=2/2=1.000, F1=0.800
MaxToolCalls    => False | 3 tool call(s), budget=2

On a real model, put ArgumentCorrectness on every tool where a wrong argument costs money, and add MaxModelRequests so a looping agent fails before it runs up your bill.

Step 2: Trajectory match in TypeScript (agentevals)#

LangChain's agentevals works on message lists, either OpenAI-style dicts or LangChain messages, so you can use it without LangGraph. Matching is deterministic and needs no API key.

npm install agentevals @langchain/core

import { createTrajectoryMatchEvaluator, type FlexibleChatCompletionMessage } from "agentevals";

// What the agent actually did (OpenAI-style messages work as-is).
const outputs: FlexibleChatCompletionMessage[] = [
  { role: "user", content: "Order 12345 arrived broken, please refund it." },
  {
    role: "assistant",
    content: "",
    tool_calls: [
      { function: { name: "lookup_order", arguments: JSON.stringify({ order_id: "12345" }) } },
      { function: { name: "check_policy", arguments: JSON.stringify({ reason: "damaged" }) } },
      {
        function: {
          name: "issue_refund",
          arguments: JSON.stringify({ order_id: "12345", amount_cents: 4200 }),
        },
      },
    ],
  },
  { role: "tool", content: "order 12345: delivered, $42.00" },
  { role: "tool", content: "damaged items are refundable" },
  { role: "tool", content: "refund issued" },
  { role: "assistant", content: "Your $42.00 refund for order 12345 is on its way." },
];

// The steps that must happen.
const referenceOutputs: FlexibleChatCompletionMessage[] = [
  { role: "user", content: "Order 12345 arrived broken, please refund it." },
  {
    role: "assistant",
    content: "",
    tool_calls: [
      { function: { name: "lookup_order", arguments: JSON.stringify({ order_id: "12345" }) } },
      { function: { name: "issue_refund", arguments: JSON.stringify({ order_id: "12345" }) } },
    ],
  },
  { role: "assistant", content: "Refund issued for order 12345." },
];

const evaluator = createTrajectoryMatchEvaluator({
  trajectoryMatchMode: "superset", // extra calls (check_policy) are fine
  toolArgsMatchMode: "superset", // actual args may carry extra fields (amount_cents)
  toolArgsMatchOverrides: {
    // Per-tool override: compare order IDs case-insensitively.
    lookup_order: (actual, expected) =>
      String(actual.order_id).toLowerCase() === String(expected.order_id).toLowerCase(),
  },
});

const result = await evaluator({ outputs, referenceOutputs });
console.log(result);

This passed tsc --noEmit (strict, moduleResolution: "bundler") against agentevals@0.0.7 and @langchain/core@1.2.18, and it ran offline:

{ key: 'trajectory_superset_match', score: true, comment: undefined, metadata: undefined }

When we re-ran the same trajectories with toolArgsMatchMode: "ignore" and each mode in turn, the results were strict false, unordered false, subset false and superset true. That run is the source of the diagram above.

Arguments are matched exactly by default. You can set toolArgsMatchMode to "ignore", "subset" or "superset". For a single tool, toolArgsMatchOverrides accepts a mode, a list of fields that must match exactly, or a comparator function, and overrides take precedence over the mode. The same package includes createTrajectoryLLMAsJudge with a TRAJECTORY_ACCURACY_PROMPT, graph trajectory evaluators for LangGraph, and pytest and Vitest/Jest integration through LangSmith. The Python package (pip install agentevals) was at 0.0.9 when we checked.

In LangSmith, these evaluators run as experiments. For production traffic, LangSmith's multi-turn online evaluators wait for a thread to go idle, then score the whole thread. A thread evaluator can use a trajectory variable, a flat ordered list of messages that includes tool calls and their results. On LangSmith Cloud, that variable is only available in the GCP US region (smith.langchain.com).

Step 3: Put the gate in CI (Mastra + Vitest)#

Mastra published its Vitest integration on September 24, 2026. It splits assertions into gates, which are hard pass/fail checks scored per item, and scorer thresholds, which compare an average score. expectEvals(...).toPass() requires every item to pass every gate, and toPass(0.8) requires at least 80% of items to pass. Per the Vitest integration docs, you need vitest 3 or 4, and the @mastra/evals peer range is >=3.0.0 <5.0.0, so pin vitest@4 now that vitest 5 is out.

npm install @mastra/core @mastra/evals zod vitest@4

The agent and its tools:

// src/refund-agent.ts
import { Agent } from "@mastra/core/agent";
import { createTool } from "@mastra/core/tools";
import { z } from "zod";

export const lookupOrder = createTool({
  id: "lookup-order",
  description: "Look up an order before any refund decision.",
  inputSchema: z.object({ orderId: z.string() }),
  execute: async ({ orderId }) => ({ orderId, status: "delivered", totalCents: 4200 }),
});

export const issueRefund = createTool({
  id: "issue-refund",
  description: "Refund an order that was already looked up.",
  inputSchema: z.object({ orderId: z.string(), amountCents: z.number() }),
  execute: async ({ orderId, amountCents }) => ({ orderId, refundedCents: amountCents }),
});

export const refundAgent = new Agent({
  id: "refund-agent",
  name: "Refund Agent",
  instructions: "Look up the order first, then refund it if it is eligible.",
  model: "openai/gpt-5",
  tools: { lookupOrder, issueRefund },
});

The eval, written as an ordinary Vitest test:

// src/refund-agent.eval.test.ts
import { expect, test } from "vitest";
import { expectEvals } from "@mastra/evals/vitest";
import { checks } from "@mastra/evals/checks";
import { createTrajectoryAccuracyScorerCode } from "@mastra/evals/scorers/prebuilt";
import { refundAgent } from "./refund-agent";

const trajectory = createTrajectoryAccuracyScorerCode({
  expectedTrajectory: {
    steps: [
      { stepType: "tool_call", name: "lookupOrder", toolArgs: { orderId: "12345" } },
      { stepType: "tool_call", name: "issueRefund" },
    ],
  },
  comparisonOptions: { ordering: "strict" },
});

test("refund agent looks up before refunding", { timeout: 120_000 }, async () => {
  const result = await expectEvals({
    target: refundAgent,
    data: [
      { input: "Order 12345 arrived broken, please refund it." },
      { input: "Refund order 12345, the box was crushed." },
    ],
    gates: [
      checks.toolOrder(["lookupOrder", "issueRefund"]), // hard pass/fail per item
      checks.noToolErrors(),
      checks.maxToolCalls(4),
    ],
    scorers: { trajectory: [trajectory] }, // categorized: receives the extracted trajectory
  }).toPass(); // every item must pass every gate; toPass(0.8) allows a 20% miss rate

  // Average trajectory score across items, read by dot-path.
  expect(result).toHaveScoreAbove("trajectory.code-trajectory-accuracy-scorer", 0.9);
});

And the config, which registers the matchers and the score-table reporter:

// vitest.config.ts
import { defineConfig } from "vitest/config";
import { MastraEvalsReporter } from "@mastra/evals/vitest";

export default defineConfig({
  test: {
    include: ["src/**/*.eval.test.ts"],
    reporters: ["default", new MastraEvalsReporter()],
    setupFiles: ["@mastra/evals/vitest/setup"],
  },
});

All three files passed tsc --noEmit against @mastra/core@1.78.0, @mastra/evals@1.10.7, vitest@4.1.12 and zod@4.6.7. To run the test without an API key, we pointed it at a copy of the agent whose model was a scripted MockLanguageModelV4 from ai/test (ai@7.0.143). The test passed and the reporter printed:

✓ refund agent looks up before refunding (2 items)
   check-tool-order (gate)                       1.0  ✓
   check-no-tool-errors (gate)                   1.0  ✓
   check-max-tool-calls (gate)                   1.0  ✓
   trajectory.code-trajectory-accuracy-scorer    1.0

We then swapped the order of the scripted calls so the refund came first. The test failed with EvalPassRateError: Eval did not pass (required pass rate: 100%) and gate check-tool-order: pass rate 0% < required 100%, and the trajectory score dropped to 0.0. That run also showed that the names used in trajectories and in toolOrder are the keys of the agent's tools object (lookupOrder), not the tool id (lookup-order). A scripted model only proves the plumbing works. Use a real model to measure how the agent actually behaves.

Two more details are worth knowing. In the @mastra/core@1.78.0 types, thresholds attach only to a flat scorer list ({ scorer, threshold }), while a trajectory scorer needs the categorized scorers: { trajectory: [...] } form to receive the extracted trajectory. That is why the example checks the trajectory score with toHaveScoreAbove instead of a threshold. Also, LLM-backed evals are far slower than Vitest's 5-second default timeout, so give every eval test an explicit timeout.

Step 4: Score the whole trace (Braintrust)#

A Braintrust scorer receives input, output, expected, metadata and trace, and returns a number from 0 to 1. A scorer can run at span scope (one span) or at trace scope (once per completed trace, with access to every span). Online scoring rules also offer a group scope. Autoevals score individual spans only, so trajectory logic belongs in a trace-scope custom-code scorer that calls trace.getSpans.

npm install braintrust autoevals

// refund.eval.ts
import { Eval, traced, type EvalScorer } from "braintrust";
import { Factuality } from "autoevals";

// Each tool call becomes a span of type "tool" inside the eval's trace.
const lookupOrder = (orderId: string) =>
  traced(async (span) => {
    span.log({ input: { orderId } });
    return { orderId, status: "delivered", totalCents: 4200 };
  }, { name: "lookup_order", type: "tool" });

const issueRefund = (orderId: string, amountCents: number) =>
  traced(async (span) => {
    span.log({ input: { orderId, amountCents } });
    return { orderId, refundedCents: amountCents };
  }, { name: "issue_refund", type: "tool" });

// Stand-in for your agent loop: in production a model picks these calls.
async function refundAgent(input: string): Promise<string> {
  const orderId = input.match(/\d{5}/)?.[0] ?? "";
  const order = await lookupOrder(orderId);
  await issueRefund(order.orderId, order.totalCents);
  return `Refunded $${(order.totalCents / 100).toFixed(2)} for order ${orderId}.`;
}

// Trace-scope scorer: runs once per case and reads every tool span.
const lookupBeforeRefund: EvalScorer<string, string, string> = async ({ trace }) => {
  if (!trace) return { name: "lookup_before_refund", score: 0 };
  const tools = await trace.getSpans({ spanType: ["tool"] });
  const names = tools.map((s) => s.span_attributes?.name);
  const lookup = names.indexOf("lookup_order");
  const refund = names.indexOf("issue_refund");
  return {
    name: "lookup_before_refund",
    score: refund === -1 || (lookup !== -1 && lookup < refund) ? 1 : 0,
    metadata: { tools: names },
  };
};

Eval("refund-agent", {
  data: () => [
    {
      input: "Order 12345 arrived broken, please refund it.",
      expected: "Refunded $42.00 for order 12345.",
    },
  ],
  task: refundAgent,
  scores: [Factuality, lookupBeforeRefund],
});

Run it with npx braintrust eval refund.eval.ts. Add --terminate-on-failure to stop on the first failure. This file passed tsc --noEmit against braintrust@3.37.2 and autoevals@0.4.0. Factuality is an LLM judge and needs a model key, so we did not run the full eval. Braintrust's docs list the extra span filters (name, hasError, metadata, duration) as requiring TypeScript SDK v3.33.0+ or Python SDK v0.39.0+. In the 3.37.2 type definitions those fields sit under a filters object (trace.getSpans({ spanType: ["tool"], filters: { hasError: true } })). Check your SDK's types before copying a snippet from the docs.

%% title: Where a trajectory scorer gets its data
sequenceDiagram
    participant T as Test runner
    participant A as Agent
    participant R as Tracer
    participant S as Scorer
    T->>A: run case input
    A->>R: span: model call
    A->>R: span: tool lookup_order
    A->>R: span: tool issue_refund
    A-->>T: final output
    T->>S: output + expected + trace
    S->>R: read tool spans
    S-->>T: score + reason

The diagram shows how all four frameworks work. Pydantic Evals reads an OpenTelemetry span tree, Mastra extracts a trajectory from its observability store or from tool invocations, agentevals reads the message list, and Braintrust queries its own spans. Wherever your traces live, your trajectory scorer reads from there.

Step 5: Add an LLM judge last, and show it the path#

Only add a judge for things code cannot check: tone, whether the reply is faithful to what the tools returned, and whether it promises something the policy does not allow. Pydantic's LLMJudge sees only the case inputs, output and expected output. It does not see the span tree. The Pydantic docs recommend a small custom evaluator that pulls the trajectory out of ctx.span_tree and passes it to judge_input_output. Here are both, alongside the cheap checks, with each case run three times:

from dataclasses import dataclass

from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import (
    EvaluationReason,
    Evaluator,
    EvaluatorContext,
    LLMJudge,
    MaxToolCalls,
    TrajectoryMatch,
)
from pydantic_evals.evaluators.llm_as_a_judge import judge_input_output
from pydantic_evals.otel import SpanTreeRecordingError

from refund_evals import refund_task  # agent + tools from Step 1

JUDGE_MODEL = "openai:gpt-5"  # any pydantic-ai model string; needs that provider's API key


@dataclass
class TrajectoryJudge(Evaluator):
    """LLM judge that sees the tool trajectory, not just the reply."""

    rubric: str

    async def evaluate(self, ctx: EvaluatorContext) -> EvaluationReason:
        try:
            span_tree = ctx.span_tree
        except SpanTreeRecordingError:
            return EvaluationReason(value=False, reason="No span tree available.")
        tools = [
            str(node.attributes["gen_ai.tool.name"])
            for node in span_tree
            if "gen_ai.tool.name" in node.attributes
            and node.status != "error"
            and (node.name == "running tool" or node.name.startswith("execute_tool "))
        ]
        grading = await judge_input_output(
            {"request": ctx.inputs, "tool_trajectory": ", ".join(tools) or "(none)"},
            ctx.output,
            self.rubric,
            model=JUDGE_MODEL,
        )
        return EvaluationReason(value=grading.pass_, reason=grading.reason)


dataset = Dataset(
    name="refund-agent-judged",
    cases=[
        Case(
            name="broken_item_refund",
            inputs="Order 12345 arrived broken, please refund it.",
            expected_output="Confirms the refund for order 12345 and says when it lands.",
        ),
    ],
    evaluators=[
        # Cheap, deterministic checks run first on every repeat.
        TrajectoryMatch(expected_trajectory=["lookup_order", "issue_refund"]),
        MaxToolCalls(max_calls=4),
        # Outcome judge: sees input, output and expected output only.
        LLMJudge(
            rubric="The reply confirms the refund, names the order, and makes no promises the policy does not support.",
            model=JUDGE_MODEL,
            include_input=True,
            include_expected_output=True,
        ),
        # Path-aware judge: also sees which tools ran.
        TrajectoryJudge(
            rubric="The refund was issued only after the order was looked up, and the reply matches what the tools returned.",
        ),
    ],
)

if __name__ == "__main__":
    report = dataset.evaluate_sync(refund_task, repeat=3, max_concurrency=2)
    report.print(include_reasons=True)

This file compiles and imports cleanly with pydantic-evals 2.57.0. We did not run it against a real judge. To test the plumbing, we swapped JUDGE_MODEL for TestModel() and ran it. repeat=3 produced three runs labelled broken_item_refund [1/3] to [3/3], and both judges were invoked on each run. TestModel's verdicts are placeholders, so the run proves only that the wiring works. The TrajectoryJudge filter is a simplified version of the docs' pattern. The docs' version also excludes deferred tool calls and output-function spans, so copy that one if your agent uses either.

Some judge practices from the docs. LangSmith's evaluation concepts say few-shot examples in the grader prompt often improve LLM-as-judge performance. They also note that reference-based evaluators work only offline, while reference-free ones work both offline and online. When absolute scores are noisy, compare runs pairwise: LangSmith supports pairwise evaluators and pairwise annotation queues. Judges have their own failure modes, including persuasion and sensitivity to budget. Our post the judge is now a system you design covers those and jury designs. Agent evals need baseline receipts argues for comparing pairwise against a production baseline.

Step 6: Online evals and the regression loop#

Offline suites only catch the failures you thought to write cases for. Online evaluation scores a sample of real traffic, and every confirmed failure becomes a new offline case.

• Mastra: live scorers run asynchronously next to the agent. sampling.rate (0 to 1) sets the fraction of runs scored. The overview docs say the decision is derived from the trace ID, so scorers at the same rate score the same traces, and eligibility filters run before sampling. Results land in the mastra_scorers table.
• LangSmith: thread-level evaluators wait for an idle period after the last trace in a thread, then score the assembled conversation.
• Braintrust: online scoring rules run at span, trace or group scope.
• Everyone: only reference-free evaluators work online, because production traffic has no expected answer.

OpenAI's May 12, 2026 cookbook, Build an Agent Improvement Loop with Traces, Evals, and Codex, describes a version of this loop. It starts from real Agents SDK traces, adds human and model feedback, generates evals scored as deterministic, llm_judge or hybrid, validates the current harness with Promptfoo (deterministic assertions plus an llm-rubric at a 0.8 threshold), and then hands a ranked list of changes to Codex.

%% title: Offline gate vs online sampling
flowchart LR
    subgraph Offline
        D1[Dataset of cases] --> D2[Run agent N times]
        D2 --> D3[Gates and thresholds]
        D3 --> D4[CI pass or fail]
    end
    subgraph Online
        O1[Production traces] --> O2[Sample by rate]
        O2 --> O3[Reference-free judges]
        O3 --> O4[Review queue]
    end
    O4 --> D1

If you use OpenAI Evals or graders: the deadline#

OpenAI's deprecations page sets this timeline:

The page says "Graders documented for eval workflows are part of this transition," and the graders guide now opens with "OpenAI is deprecating graders as part of the evals and fine-tuning workflows they support." OpenAI's own migration pointer is its Moving from OpenAI Evals to Promptfoo cookbook. If you have graders in JSON, each type maps cleanly to code:

The multi row is the one most relevant to agents. Multi graders are a reinforcement fine-tuning feature, but the guide's own example grades a tool call as 0.5 * function_name + 0.5 * arguments, using two string_check graders. That is ToolCorrectness plus ArgumentCorrectness. The guide warns that exact argument matching under-rewards answers that are only slightly different, such as 1 instead of 1.0 or an abbreviated state name. The subset match modes and per-tool comparators shown above deal with exactly that. Trace grading is still described in OpenAI's agent evals guide. For the wider migration, see OpenAI Agent Builder and Evals are shutting down.

Troubleshooting (from testing)#

• Pydantic agentic evaluators report no tool calls. The span tree is empty. Install pydantic-evals[logfire], call logfire.configure(send_to_logfire=False) and logfire.instrument_pydantic_ai() before the run.
• TypeError: Dataset.__init__() missing 1 required keyword-only argument: 'name'. pydantic-evals 2.57.0 requires name= on Dataset.
• ArgumentCorrectness says arguments are unavailable. The agent was instrumented with include_content=False. The docs say the evaluator then returns False with an explanation instead of crashing.
• Mastra install fails with an npm arborist error. We hit one installing the Mastra packages alongside the others and fell back to npm install --legacy-peer-deps. Also check that vitest is 3 or 4.
• Matcher methods missing on expect(result). The types come from @mastra/evals/vitest. Importing expectEvals from it, as above, brings them in. A plain runEvals call without that import raised TS2339 in our checks.
• Mastra toolOrder never matches. Use the keys of the agent's tools object, not the createTool id.
• Braintrust trace scorer score missing locally. With noSendLogs and no BRAINTRUST_API_KEY, trace.getSpans threw "Please specify an api key", and the scorer's score simply did not appear in the results. It was not recorded as a 0. Set the key when you use trace-scope scorers, and treat a missing score as a failure.
• Flaky pass rates. Agents are nondeterministic. Use repeat (Pydantic) or num_repetitions (LangSmith's aevaluate), and assert a rate such as toPass(0.8) instead of trusting one run.

FAQ#

What is a trajectory eval?#

A trajectory eval grades the ordered list of tool calls an agent made, and usually their arguments, instead of only the final answer. Pydantic Evals scores it with an F1 over the longest common subsequence or the multiset overlap. agentevals and Mastra check strict, unordered, subset or superset matches against a reference.

Should I use an LLM-as-judge for tool calls?#

Not as the first check. Which tool ran, in what order, with what arguments, is a deterministic question that code answers for free and the same way every time. Use a judge for outcome quality, and when you do, pass it the trajectory so it can see how the agent reached the answer.

How many times should I run each case?#

More than once. Agents are nondeterministic, so use repeat=3 in Pydantic Evals or num_repetitions in LangSmith, and assert a pass rate rather than a single result. Mastra's toPass(0.8) requires 80% of items to pass every gate.

What happens to OpenAI Evals on October 31, 2026?#

Existing evals become read-only, and the dashboard and API are scheduled to shut down on November 30, 2026. OpenAI says graders documented for eval workflows are part of the transition, so move grader logic into code (see the mapping table above) or follow OpenAI's Promptfoo migration cookbook.

Do I need a hosted platform?#

No. Pydantic Evals, agentevals and Mastra's Vitest integration all run locally and in CI. Hosted platforms such as LangSmith and Braintrust add stored experiments, side-by-side comparison, annotation queues and online scoring. Our agent eval tools comparison covers those tradeoffs.

Sources#

• Pydantic Evals overview
• Pydantic Evals agentic evaluators
• langchain-ai/agentevals on GitHub
• LangSmith evaluation concepts
• LangSmith multi-turn online evaluations
• LangSmith: evaluate a complex agent
• Mastra evals overview
• Mastra trajectory accuracy scorers
• Mastra: Introducing Vitest Integration for Testing
• Mastra Vitest integration docs
• Mastra multi-turn evals
• Braintrust: write scorers
• Braintrust: custom code scorers
• OpenAI API deprecations
• OpenAI graders guide
• OpenAI: evaluate agent workflows
• OpenAI cookbook: Build an Agent Improvement Loop with Traces, Evals, and Codex

Continue Reading#

• AI Agent Eval Tools Compared: Braintrust vs Promptfoo vs DeepEval - which platform to store and compare these experiments in.
• The Judge Is Now a System You Design - how LLM judges fail, and how juries and calibration help.
• OpenAI Agent Builder and Evals Are Shutting Down: Move the Agent Stack Into Code - the full migration picture before the November 30 shutdown.
• What's New in Mastra (Late 2026): Harness, AgentController GA, Eval Gates and MCP 2.x
• Best AI Agent Frameworks (2026) - CrewAI and other agent frameworks, compared and ranked