Evaluate an agent

Evals let you build a small regression suite for an agent — a set of example inputs with the checks that should hold on the agent’s output — and run that suite whenever you change the agent’s prompt, model, or tools, so you catch a regression before it reaches a live process.

Before you start

• The agent you want to evaluate already exists (see Create an agent).
• You’ve published at least one version of the agent if you want to compare versions (see Version an agent) — evals can also run against the agent’s current live version without any published version.

Build a dataset

1. Open the Evals rail. On the agent’s detail page, open the Evals panel. It lists any datasets already targeting this agent, each with its latest run’s status and pass rate.

2. Create a dataset. Click New dataset and give it a name. A dataset starts as an empty draft — you add cases next.

3. Add cases. In the dataset editor, click Add case. Each case has:

   • a name;
   • inputs — a JSON object matching what the agent expects;
   • an optional expected output — also a JSON object, if the agent’s output should match or be checked against something specific;
   • optional case-level evaluators (see below) that apply only to this case, in addition to the dataset’s evaluators.

   Reorder cases by dragging the handle or using the up/down buttons.

4. Set dataset-level evaluators. Below the case list, use the evaluator builder to add checks that run against every case. Pick a type and fill in its fields:

   Evaluator	Checks
   Contains	Output contains a given value
   Has min fields	Output has at least N fields
   Has field names	Output includes specific field names
   Max duration	The run completed within N seconds
   Is instance	Output is a given type
   Equals expected	Output matches the case’s expected output exactly
   Form definition is valid	Output is a valid form definition
   Summary within limits	A generated summary stays within its length limit
   Agent spans present	The agent actually executed (not a stub/skip)
   LLM judge	A rubric-based judgment from an LLM, scoring or pass/fail

   For an LLM judge evaluator, write a rubric describing what “good” looks like and optionally pick a judge model — leaving it blank lets your organization’s or the platform’s default judge model resolve automatically (see Judge model resolution below).

   If a check you need isn’t in the list, add a Raw JSON row and hand-author the evaluator spec directly.

5. Publish. Click Publish and optionally add a changelog note. This snapshots the current cases and evaluators into an immutable version — runs always target a specific published version, never the live draft, so a run’s result is reproducible even if you keep editing the dataset afterward. The dataset stays editable; publish again whenever you want to cut a new version.

No live/promote step for datasets

Unlike agents and other artifacts, a published eval dataset version has no “promote to live” step — every run explicitly picks which published version to evaluate against.

Import cases in bulk

Instead of adding cases one at a time, use Import in the dataset editor to upload a CSV or JSON file:

• JSON — a top-level array of case objects: name, inputs, expected_output (optional), source_trace_id (optional).
• CSV — columns name, inputs, expected_output, source_trace_id. The inputs and expected_output cells hold JSON text (e.g. a cell containing {"invoice_amount": 500}).

In both formats, inputs is required and must be a JSON object (not an array or a bare value); expected_output, if present, must also be an object. The import preview flags any row that fails these checks and blocks the Import button until every row is clean — up to 500 cases per file. Import is all-or-nothing: either every valid row is created, or none are (a case name collision or other server-side rejection is reported back per row rather than partially importing).

Turn a real run into a case

If you’ve noticed an agent handle (or mishandle) a real request in a particular way, you can capture that run as a regression case directly from its trace:

1. On the Agent Runs page, or from a run’s own detail page, use the Add to eval set action on the run you want to capture.
2. Pick an existing dataset for this agent, or create a new one inline.
3. The dialog pre-fills inputs and expected output from the trace where it can — review and adjust before saving. The new case is stamped with the run’s trace ID for traceability back to where it came from.

Run an eval

1. From the agent’s Evals panel, click Run evals on the dataset you want to run.
2. Choose a published dataset version — required; you must publish the dataset at least once before you can run it.
3. Optionally choose an agent version to evaluate against a specific published snapshot of the agent, instead of its current live version.
4. Set concurrency (1–10) — how many cases run in parallel.
5. Click Run evals. The run starts asynchronously; its status (queued → running → completed/failed/cancelled) and pass rate appear in the dataset’s Runs list, updating automatically while the run is active.

Expand a run row to see per-case results: pass/fail, which assertions passed or failed, any numeric scores, and — importantly — any evaluator that couldn’t produce a verdict at all (for example, an LLM judge whose model failed to resolve). That last category is always shown, separately from pass/fail, so a grading gap never disappears silently into a false “passed.”

Compare two runs

To see how a change affected results, expand a run and use the Compare with… picker to select another completed run of the same dataset — typically one run per agent version you’re comparing. The comparison shows each run’s overall pass rate side by side, then a per-case table aligned by case name, so you can see exactly which cases changed behavior between the two runs.

Evaluate an agent as it runs in a process

By default, an eval run scores the agent’s bare configuration — its own default prompt, model, temperature, and output settings. But an agent called from inside a process is invoked by a specific node, and that node can ask for something different from the agent’s defaults: its own structured-output schema, a tighter temperature, a different token limit. If your dataset only ever evaluates the bare config, a case can pass in Evals and still misbehave in production, because production never actually sent what the eval sent.

To get results that reflect what the process really does, bind the dataset to the process node that invokes this agent.

Bind to a process node

1. Open the dataset and find its Invocation panel.
2. Pick a Process from the dropdown.
3. Pick the node in that process that calls this agent. Only nodes whose configured agent matches the dataset’s target agent are selectable — a node calling a different agent shows disabled with an Agent mismatch badge, so you can’t bind to something the backend would reject anyway.
4. Click Bind to this node.

Once bound, the panel shows a read-only Resolved invocation profile: strict mode, temperature, max tokens, and the structured-output schema — exactly what production sends for that node, not a copy you maintain by hand.

An unbound dataset still runs fine; the Invocation panel just says so with an Evaluating the bare agent config notice, so it’s never a silent gap.

Use field evaluators for structured output

When a node’s output is structured (JSON), a plain Contains check — which stringifies the whole payload — usually won’t match it:

• Output: {"decision": "APPROVE", "reason": "Invoice matches PO"}
• Contains looking for DECISION: APPROVE → fails, even though the agent got the decision right.

Use a field evaluator instead, which reads one field out of the parsed JSON:

• Field equals — field: decision, value: APPROVE → passes.
• Field contains — same idea, a substring match on one field.
• Has field names — asserts the output has a given set of field keys, regardless of their values.

Once a dataset is bound to a node, the evaluator builder lists these field-oriented evaluators first and suggests field names read from the node’s output schema.

Live vs. pinned

Live for gating, pinned for A/B

Live (default) always re-resolves the bound node’s current profile — edit the node’s schema or temperature in the process designer, and the very next eval run picks it up automatically. That’s what you want for regression gating: the eval should track the node as it evolves.

Pinned freezes the resolved profile the moment you switch to it, so every run afterward replays that exact snapshot even if the node changes later — useful when comparing agent versions head-to-head and you don’t want an unrelated node edit to sneak into the comparison. Use Re-pin to current when you’re ready to intentionally capture the node’s latest profile.

Reading run results

A run’s row (and its detail header) carries a profile badge — live @ <node>, pinned, or bare config for an unbound dataset — so you always know what a run actually evaluated. When you compare two runs whose badges differ (bound against unbound, live against pinned, or two different nodes), the comparison view shows a warning that the runs aren’t apples-to-apples, since something besides the agent version changed between them.

CLI parity

The same binding is available from the CLI:

sp evals datasets create --name "..." --agent-id <agent-id> \
  --bind-process <process-id> --bind-node <node-id> --profile-mode live

sp evals datasets update <dataset-id> --profile-mode pinned
sp evals datasets repin <dataset-id>

sp evals runs start <dataset-id> --profile-mode pinned

datasets create/update take --bind-process/--bind-node (both or neither) and --profile-mode (live or pinned); datasets repin re-resolves a pinned dataset’s frozen snapshot; runs start --profile-mode overrides the dataset’s stored profile mode for that one run without changing the dataset itself.

Judge model resolution

When an LLM judge evaluator doesn’t specify a model explicitly, the platform resolves one in order: your organization’s configured default judge model, then a platform-wide fallback. Setting an explicit model on the evaluator itself always takes priority over either default.

Org default judge model — not yet configurable in Settings

As of this writing, your organization’s default judge model can only be set through the API, not from a Settings page in the app. If no org default is configured, eval runs still work — they fall back to the platform default — but you can’t yet pick your organization’s preferred judge model from the UI.

See eval traffic separately from production

Every agent run started by an eval shows up on the Agent Runs page tagged with an eval badge. By default, Agent Runs shows only Production traffic, so eval runs don’t clutter your day-to-day view — switch the filter to Eval or All to see them. Following a View traces → link from a run’s results takes you straight to that run’s eval traces.

Related

Create an agentBuild the agent you'll evaluate.

Version an agentPublish agent versions to compare in eval runs.

AI agentsWhat an agent is and how it runs.