AI Agent Evaluation: How to Build an Evaluation Set and Set Pass Criteria

AI agent evaluation means testing an agent against a fixed set of real cases, each with the outcome a good employee would reach, and scoring how often the agent gets there. Good evaluation scores more than the final answer. It checks whether the agent used the right tools, followed policy, stayed within its permissions and handed over to a person when it should. You run the same evaluation set before every release, and every real failure becomes a new case.

This post gives a template for evaluation cases, a scoring rubric, and a way to set pass criteria. For how evaluation fits into the full build, see our guide to AI agent development.

Why evaluation comes first

Without an evaluation set, nobody can say whether an agent is good enough to launch, whether a new prompt made it better or worse, or whether a cheaper model is acceptable. Arguments about quality become opinions.

OpenAI's evaluation best practices describe evals as "structured tests for measuring a model's performance," and stress that "Evaluation is a continuous process." Its practical guide to building agents recommends setting up evals to establish a performance baseline before optimizing, for example before swapping in smaller models to cut cost.

Our own generative AI process starts the same way: create a test set, then iterate retrieval, prompts and models against it. Our enterprise knowledge assistant, for example, runs an evaluation suite that measures answer accuracy before every release, and feedback from users feeds improvements to retrieval.

Step 1: Define success criteria

Decide what "good enough" means before building. Anthropic's guide to defining success criteria says good criteria are specific, measurable, achievable and relevant. Instead of "good performance," name the behavior. Instead of "accurate," name the measure and the threshold.

For an agent, criteria usually cover four areas:

• Outcome. Did the task end correctly: the right refund, the right booking, the right answer?
• Process. Did the agent use the right tools with the right inputs, and avoid unnecessary steps?
• Policy. Did it stay within the rules: limits, tone, data handling, permissions?
• Handover. Did it escalate when it should, and only when it should, with a useful summary?

Step 2: Build the evaluation set

Collect real cases from the process the agent will handle: tickets, emails, documents, requests. For each, record what a good employee did. Then add cases deliberately:

• Common cases, in roughly the proportion they occur.
• Edge cases: unusual formats, missing information, multiple requests in one message.
• Cases where the right answer is to hand over.
• Hostile inputs: messages that try to make the agent ignore its instructions.

OpenAI's guidance suggests logging as you develop "so you can mine your logs for good eval cases." Real interactions make better tests than invented ones.

	Field	Example (support agent)
Case ID and source	Unique ID; where the case came from	T-0142, past helpdesk ticket
Input	What the agent receives	Customer email: late order, wrong size, wants to return one item
Context	Records the tools would return	Order details, carrier status, returns policy
Expected outcome	What a good employee would do	Delivery update sent; return label created for the jacket
Expected tool use	Tools that should and should not be called	Order lookup, carrier lookup, create return; no refund call
Policy checks	Rules that must hold	Within return window; no refund above threshold without approval
Handover expected?	Yes or no, and why	No
Tags	Category, difficulty, risk	returns, multi-request, low risk

Source: BufferSol evaluation case template

BufferSol Technologies · buffersoltech.com

Step 3: Choose how to grade

Use the simplest grader that works for each check:

Automated judges save time on qualities code cannot check, but OpenAI's guidance is clear: "Use human feedback to calibrate automated scoring." Have people grade a sample, compare with the judge and adjust the rubric until they agree.

Step 4: Set pass criteria and a release gate

Turn the success criteria into thresholds for launch. A useful release gate has three parts:

1. Overall task success on the evaluation set at or above the agreed target.
2. Zero tolerance on the cases that matter most: no unauthorized high-risk actions, no data exposed to the wrong person, no policy violations on tagged high-risk cases.
3. No regression: no category gets worse than the last release.

Set the targets with the process owner, based on how the current manual process performs and on the cost of a mistake. A drafting assistant that a person reviews can launch with a lower bar than an agent that acts on its own.

Step 5: Run it continuously

Run the evaluation set:

• Before every release, including prompt, tool, model and rule changes.
• When a model provider releases a new version you plan to adopt.
• On a schedule in production, using a sample of live cases graded after the fact.

Every failure found in testing, review or production becomes a new case. In our AI customer support agent, cases escalated to people with a summary are a natural source: each one shows where the agent's limits are.

Evaluating multi-step behavior

Agents take several steps, so a single score per case can hide where things went wrong. Store the full trace for every evaluation run, then look at failures step by step:

• Did the agent understand the request correctly at the start?
• Did it call the right tool first, or wander through unnecessary lookups?
• Did it read the tool results correctly, or ignore information it had?
• Did it stop at the right point, or keep going after the task was done?

Two extra measures help here. The number of steps per case shows whether the agent is efficient, and it drives cost. The share of cases that hit the maximum-turn limit shows where the agent gets stuck. Both should be tracked per category, alongside task success, so you can see which kinds of request make the agent work hardest.

Comparing designs fairly

The evaluation set is also how you choose between options: two prompts, two models, a single agent or several. Keep the set fixed while you compare, run each option on every case, and look at results by category, not just the overall score. An option that is slightly better overall but worse on high-risk cases is usually the wrong choice.

Record cost per case and time per case next to quality. A cheaper model that matches the expensive one on routine categories can handle those steps, while the stronger model handles the hard ones. That is the kind of decision only an evaluation set can support with evidence.

Common mistakes

• Writing only easy cases, so the agent looks better than it is.
• Scoring only the final reply and missing a wrong or risky tool call along the way.
• Changing the evaluation set and the agent at the same time, so results cannot be compared.
• Trusting an automated judge without checking it against people.
• Treating evaluation as a launch task rather than a permanent part of the system.

Getting started

Pull fifty recent cases from the process you want to automate, fill in the template above for each one, and agree three pass criteria with the process owner before anyone writes a prompt. Our AI agent development and generative AI teams can help you build the set and the grading, or you can talk to an engineer.