Evaluating AI Agents in Production: Accuracy, Cost and Reliability
An agent that's 95% reliable per step succeeds end-to-end only about a third of the time across a 20-step chain — here's how to actually measure whether an agent is ready for production.
By Naeem Akhtar · 11 min read
In short
- A 95% per-step success rate completes a 20-step task only about 36% of the time, so per-step accuracy misleads.
- Evaluate five things together: task completion, tool-call accuracy, trajectory, cost and latency, and groundedness.
- Combine offline evaluation in CI with online scoring of real traffic. Offline catches regressions; online catches drift.
- Gate promotion from dev to production on defined thresholds for every dimension.
Why per-step accuracy is the wrong headline metric
It's intuitive to ask "how accurate is the model?" and treat a high per-step accuracy number as evidence the agent is production-ready. It isn't, because failures compound across a multi-step task. An agent that's correct 95% of the time on each individual step completes an end-to-end task correctly only about 36% of the time across a 20-step chain — reliability multiplies against you with every additional step, tool call, or decision point. Evaluation is also what decides when an agent may act alone, which is the focus of our human-in-the-loop workflow design patterns.
This is why task completion — measured on the actual end state, not on whether each intermediate step looked reasonable — is the metric that matters, and why it needs to be measured on realistic multi-step tasks rather than isolated single-turn prompts.
The five dimensions that cover the real failure surface
| Dimension | What it catches |
|---|---|
| Task completion | Whether the end state is actually correct, verified against ground truth — not whether the transcript looks plausible. |
| Tool-call accuracy | Whether the agent picked the right tool, with the right arguments, at the right step. |
| Trajectory / step count | Whether it reached the right answer via a sane path — an agent that loops five extra times before succeeding is a reliability risk waiting to happen under load. |
| Cost and latency | Whether the agent is economically and practically viable to run at the volume the business actually needs. |
| Groundedness | Whether the agent's claims are actually supported by retrieved context or tool output, rather than a confident hallucination stitched in. |
Shipping against only one or two of these — usually just task completion — is how teams end up with an agent that scores well in testing and then quietly burns through budget or produces ungrounded answers in production.
pass@k: measuring reliability, not just correctness
A single successful run doesn't tell you much about whether an agent is reliable — LLM outputs are non-deterministic, and a task that succeeds once can fail the next time on the exact same input. pass@k measures whether the agent completes the same task class successfully across k independent attempts, typically k=3 to k=10.
Why this matters
An agent with a high single-run success rate but a low pass@k score is telling you it's inconsistent — which is often worse for user trust than an agent that's consistently mediocre, because inconsistency is unpredictable and hard for a team to build workarounds around.Offline and online evaluation are both required, not either/or
Offline evaluation runs a fixed dataset of known tasks with known-good outcomes, in CI, on every meaningful change — this catches regressions reproducibly before anything reaches a real user. Online evaluation scores real production traffic as it happens, catching drift, novel failure modes offline testing never anticipated, frustrated users, and adversarial or edge-case inputs that a curated test set wouldn't include.
- Offline: fast, reproducible, catches known failure modes before release, but blind to anything not represented in the test set.
- Online: catches real drift and novel failures, but by definition happens after the agent is already live in front of users.
- Teams that skip offline testing ship regressions repeatedly. Teams that skip online monitoring find out about drift from a support ticket instead of a dashboard.
Eval-gated promotion: making this a process, not a one-time check
The teams that consistently ship reliable agents treat evaluation as a continuous discipline: automated metrics for coverage, model-based screening for efficiency at scale, and human expert judgment for the correctness only domain knowledge can verify — combined, not chosen from. No agent version should move from development to staging, or staging to production, without meeting defined thresholds on all five dimensions simultaneously. This is exactly what a decisioning-layer rebuild caught on a vendor compliance pipeline we worked on: the original set of 24+ disconnected automations had no shared evaluation or visibility, so failures were discovered by a partner complaining, not by a dashboard. A single pipeline with a standing evaluation set turns that into something the team can see coming.
Why long chains compound errors
A single-step accuracy figure hides how quickly reliability decays across a multi-step task. If every step succeeds independently with the same probability, the chance of completing the whole task is that probability raised to the number of steps.
| Per-step reliability | 5 steps | 10 steps | 20 steps |
|---|---|---|---|
| 90% | 59% | 35% | 12% |
| 95% | 77% | 60% | 36% |
| 98% | 90% | 82% | 67% |
| 99% | 95% | 90% | 82% |
| 99.5% | 98% | 95% | 90% |
Real steps are not fully independent, and retries and checks can improve the odds, but the pattern holds: shorter chains, verification steps and narrow tools matter more than a slightly better model.
A sample agent scorecard
| Dimension | Metric | Example gate before release |
|---|---|---|
| Task completion | End-to-end success rate and pass@k on the test set | No drop versus the current version; target set per workflow |
| Tool-call correctness | Valid calls with correct arguments | Invalid or unsafe calls at or near zero |
| Groundedness | Claims supported by retrieved or tool-provided evidence | Unsupported claims below an agreed rate |
| Safety and policy | Refusals and escalations when required | All red-team cases handled correctly |
| Cost | Cost per completed task | Within budget at expected volume |
| Latency | p50 and p95 time to completion | Within the user-facing target |
| Escalation quality | Human handoffs with sufficient context | Reviewers rarely need to re-ask basic questions |
Tooling landscape
Evaluation tools change quickly, so choose by the workflow you need rather than by name. Categories worth covering:
| Need | Examples of tools in this space | Notes |
|---|---|---|
| Tracing and observability | LangSmith, Langfuse, OpenTelemetry-based tracing | Capture every step and tool call per run |
| Offline test runs and comparisons | Braintrust, promptfoo, custom CI scripts | Run a fixed set on every change and diff results |
| RAG-specific metrics | Ragas, DeepEval | Retrieval and groundedness scoring |
| Human review workflows | Annotation tools or an internal queue | Calibrates automated scoring against people |
| Cost and latency monitoring | Cloud monitoring plus model-usage dashboards | Track per task, not just per token |
Pitfalls when using an LLM as the judge
- Judges favour longer, more confident answers even when they're wrong; use rubrics with specific criteria.
- Scores drift when the judge model changes; pin the judge version and re-baseline on changes.
- A judge can share blind spots with the model being judged; validate it against human-labelled samples.
- Position and wording bias affect pairwise comparisons; randomise order and test with swapped positions.
- Never let the judge be the only gate for high-risk behaviour; keep deterministic checks and human review for those.
Frequently asked questions
Why does an agent with 95% per-step accuracy fail so often end-to-end?
Because reliability compounds multiplicatively across steps: 0.95 raised to the 20th power is roughly 0.36. A long multi-step task with many decision points or tool calls needs much higher per-step reliability than an intuitive read of "95% accurate" suggests, which is why task-completion rate on realistic multi-step tasks is a more honest metric than per-step accuracy alone.
What is pass@k in AI agent evaluation?
pass@k measures whether an agent can successfully complete the same task class across k independent attempts (commonly k=3 to k=10), rather than relying on a single run. It's a reliability metric — an agent that succeeds inconsistently across repeated attempts on the same task is a production risk even if any individual run looks fine.
Do I need both offline and online evaluation for an AI agent?
Yes. Offline evaluation (a fixed set of known tasks run in CI on every change) catches regressions before release but is limited to what's in the test set. Online evaluation (scoring real production traffic) catches drift and novel failure modes offline testing can't anticipate, but only after the agent is already live. Relying on only one leaves a real gap.
Should cost and latency be part of an AI agent's evaluation criteria?
Yes — an agent that's accurate but slow or expensive to run isn't shippable at real business volume. Cost and latency should be tracked as first-class evaluation dimensions alongside accuracy, tool-call correctness, and groundedness, not measured separately or added as an afterthought once accuracy looks acceptable.
What metrics should I track for an AI agent in production?
Task completion (including pass@k), tool-call correctness, groundedness, safety and policy adherence, cost per completed task, latency and the quality of human escalations. Track them offline in CI and online on live traffic.
Can I use an LLM to evaluate my AI agent?
Yes, with care. Use clear rubrics, pin the judge model version, validate it against human-labelled examples and keep deterministic checks and human review for high-risk behaviour.
How large should an agent evaluation set be?
Start with a few dozen realistic tasks that cover your main flows and known failure cases, then grow it with every production incident or novel failure you find.
Need help with Agentic AI Development? Talk to the engineers.
Tell us what you want to build. You'll get a straight answer on scope, approach and cost from the people who do the work.
Keep reading
Related guides
Agentic AI
Pause an AI Agent for Human Approval with LangGraph interrupt()
How to stop an agent before a risky action, wait as long as a person needs, and resume exactly where it left off. Covers interrupt(), checkpointers, the rules that cause duplicate actions, and approval design.
13 min read
Agentic AI
Human-in-the-Loop AI Agent Frameworks Compared: LangGraph, CrewAI, OpenAI and More
Every serious agent framework can pause for a human decision, but they do it differently. This guide compares how six of them pause, store state, resume and handle risk, and what each leaves for you to build.
14 min read
Agentic AI
Human Approval Checkpoints for AI Agents in Regulated Finance
What FINRA, SR 11-7, the EU AI Act and GDPR actually say about human oversight, and how to turn it into engineering: risk-tiered checkpoints, reviewers who really review, and an evidence trail. Not legal advice.
13 min read