# AI Agent Evaluation: How to Build an Evaluation Set and Set Pass Criteria

> AI agent evaluation explained: how to build an evaluation set from real cases, what to score, how to set pass criteria and how to run evals before each release.

- Canonical: https://buffersoltech.com/blog/ai-agent-evaluation
- Author: BufferSol Editorial Team
- Published: 2026-10-22 · Updated: 2026-10-22

## Key takeaways
- An evaluation set is a list of real cases with the outcome a good employee would reach. It is the single most useful thing to build first.
- Score more than the final answer: whether the agent used the right tools, followed policy and handed over when it should have.
- Set pass criteria before building, and make them specific and measurable, such as task success on the evaluation set and zero policy violations on high-risk cases.
- Run the evaluation set before every release, like regression tests, and add every real failure as a new case.
- Automated graders, including LLM-as-judge, save time but must be checked against human ratings.

AI agent evaluation means testing an agent against a fixed set of real cases, each with the outcome a good employee would reach, and scoring how often the agent gets there. Good evaluation scores more than the final answer. It checks whether the agent used the right tools, followed policy, stayed within its permissions and handed over to a person when it should. You run the same evaluation set before every release, and every real failure becomes a new case.

This post gives a template for evaluation cases, a scoring rubric, and a way to set pass criteria. For how evaluation fits into the full build, see our [guide to AI agent development](/blog/ai-agent-development-guide).

## Why evaluation comes first

Without an evaluation set, nobody can say whether an agent is good enough to launch, whether a new prompt made it better or worse, or whether a cheaper model is acceptable. Arguments about quality become opinions.

OpenAI's [evaluation best practices](https://developers.openai.com/api/docs/guides/evaluation-best-practices) describe evals as "structured tests for measuring a model's performance," and stress that "Evaluation is a continuous process." Its [practical guide to building agents](https://cdn.openai.com/business-guides-and-resources/a-practical-guide-to-building-agents.pdf) recommends setting up evals to establish a performance baseline before optimizing, for example before swapping in smaller models to cut cost.

Our own generative AI process starts the same way: create a test set, then iterate retrieval, prompts and models against it. Our [enterprise knowledge assistant](/portfolio/enterprise-knowledge-assistant), for example, runs an evaluation suite that measures answer accuracy before every release, and feedback from users feeds improvements to retrieval.

## Step 1: Define success criteria

Decide what "good enough" means before building. Anthropic's guide to [defining success criteria](https://platform.claude.com/docs/en/docs/test-and-evaluate/define-success) says good criteria are specific, measurable, achievable and relevant. Instead of "good performance," name the behavior. Instead of "accurate," name the measure and the threshold.

For an agent, criteria usually cover four areas:

- **Outcome.** Did the task end correctly: the right refund, the right booking, the right answer?
- **Process.** Did the agent use the right tools with the right inputs, and avoid unnecessary steps?
- **Policy.** Did it stay within the rules: limits, tone, data handling, permissions?
- **Handover.** Did it escalate when it should, and only when it should, with a useful summary?

## Step 2: Build the evaluation set

Collect real cases from the process the agent will handle: tickets, emails, documents, requests. For each, record what a good employee did. Then add cases deliberately:

- Common cases, in roughly the proportion they occur.
- Edge cases: unusual formats, missing information, multiple requests in one message.
- Cases where the right answer is to hand over.
- Hostile inputs: messages that try to make the agent ignore its instructions.

OpenAI's guidance suggests logging as you develop "so you can mine your logs for good eval cases." Real interactions make better tests than invented ones.

```infographic
{"type":"compare","title":"An evaluation case, field by field","columns":["Field","Example (support agent)"],"rows":[
{"label":"Case ID and source","a":"Unique ID; where the case came from","b":"T-0142, past helpdesk ticket"},
{"label":"Input","a":"What the agent receives","b":"Customer email: late order, wrong size, wants to return one item"},
{"label":"Context","a":"Records the tools would return","b":"Order details, carrier status, returns policy"},
{"label":"Expected outcome","a":"What a good employee would do","b":"Delivery update sent; return label created for the jacket"},
{"label":"Expected tool use","a":"Tools that should and should not be called","b":"Order lookup, carrier lookup, create return; no refund call"},
{"label":"Policy checks","a":"Rules that must hold","b":"Within return window; no refund above threshold without approval"},
{"label":"Handover expected?","a":"Yes or no, and why","b":"No"},
{"label":"Tags","a":"Category, difficulty, risk","b":"returns, multi-request, low risk"}
],"source":"BufferSol evaluation case template"}
```

## Step 3: Choose how to grade

Use the simplest grader that works for each check:

| Check | Grader |
| --- | --- |
| Was the right tool called with valid inputs? | Code: compare the trace with the expected tool calls |
| Is the structured output correct? | Code: compare fields with expected values |
| Did the agent hand over when expected? | Code: check the exit condition |
| Was the reply accurate, on-policy and in the right tone? | LLM-as-judge with a written rubric, calibrated by people |
| Was a borderline judgment call right? | Human review |

Automated judges save time on qualities code cannot check, but OpenAI's guidance is clear: "Use human feedback to calibrate automated scoring." Have people grade a sample, compare with the judge and adjust the rubric until they agree.

## Step 4: Set pass criteria and a release gate

Turn the success criteria into thresholds for launch. A useful release gate has three parts:

1. **Overall task success** on the evaluation set at or above the agreed target.
2. **Zero tolerance** on the cases that matter most: no unauthorized high-risk actions, no data exposed to the wrong person, no policy violations on tagged high-risk cases.
3. **No regression**: no category gets worse than the last release.

Set the targets with the process owner, based on how the current manual process performs and on the cost of a mistake. A drafting assistant that a person reviews can launch with a lower bar than an agent that acts on its own.

## Step 5: Run it continuously

Run the evaluation set:

- Before every release, including prompt, tool, model and rule changes.
- When a model provider releases a new version you plan to adopt.
- On a schedule in production, using a sample of live cases graded after the fact.

Every failure found in testing, review or production becomes a new case. In our [AI customer support agent](/portfolio/ai-customer-support-agent), cases escalated to people with a summary are a natural source: each one shows where the agent's limits are.

## Evaluating multi-step behavior

Agents take several steps, so a single score per case can hide where things went wrong. Store the full trace for every evaluation run, then look at failures step by step:

- Did the agent understand the request correctly at the start?
- Did it call the right tool first, or wander through unnecessary lookups?
- Did it read the tool results correctly, or ignore information it had?
- Did it stop at the right point, or keep going after the task was done?

Two extra measures help here. The number of steps per case shows whether the agent is efficient, and it drives cost. The share of cases that hit the maximum-turn limit shows where the agent gets stuck. Both should be tracked per category, alongside task success, so you can see which kinds of request make the agent work hardest.

## Comparing designs fairly

The evaluation set is also how you choose between options: two prompts, two models, a single agent or several. Keep the set fixed while you compare, run each option on every case, and look at results by category, not just the overall score. An option that is slightly better overall but worse on high-risk cases is usually the wrong choice.

Record cost per case and time per case next to quality. A cheaper model that matches the expensive one on routine categories can handle those steps, while the stronger model handles the hard ones. That is the kind of decision only an evaluation set can support with evidence.

## Common mistakes

- Writing only easy cases, so the agent looks better than it is.
- Scoring only the final reply and missing a wrong or risky tool call along the way.
- Changing the evaluation set and the agent at the same time, so results cannot be compared.
- Trusting an automated judge without checking it against people.
- Treating evaluation as a launch task rather than a permanent part of the system.

## Getting started

Pull fifty recent cases from the process you want to automate, fill in the template above for each one, and agree three pass criteria with the process owner before anyone writes a prompt. Our [AI agent development](/services/ai-agents) and [generative AI](/services/generative-ai) teams can help you build the set and the grading, or you can [talk to an engineer](/contact).

## Frequently asked questions
### How many test cases does an AI agent evaluation set need?
Fifty to a hundred real cases are usually enough to compare designs during a prototype, as long as they include the common cases, the awkward ones and the cases where the agent should hand over. The set should grow over time: every failure found in testing or production becomes a new case, so coverage improves where it matters.

### What is LLM-as-judge?
It is using a language model to grade another model's output against a rubric, for example whether a reply followed policy or matched the brand's tone. It scales well for qualities that are hard to check with code. Its scores need calibrating against human ratings on a sample, because the judge can be wrong or biased in consistent ways.

### How is evaluating an agent different from testing normal software?
Normal software gives the same output for the same input, so tests check exact results. Agents can take different valid paths and phrase answers differently each time. So agent evaluation scores outcomes and behaviors against criteria across many cases, and tracks rates such as task success and correct handovers, rather than expecting identical output.

## Sources
1. [Evaluation best practices](https://developers.openai.com/api/docs/guides/evaluation-best-practices), OpenAI
2. [Define success criteria and build evaluations](https://platform.claude.com/docs/en/docs/test-and-evaluate/define-success), Anthropic
3. [A practical guide to building agents](https://cdn.openai.com/business-guides-and-resources/a-practical-guide-to-building-agents.pdf), OpenAI