AI Evaluation

Evaluating AI Agents in Production: Accuracy, Cost and Reliability

An agent that's 95% reliable per step succeeds end-to-end only about a third of the time across a 20-step chain — here's how to actually measure whether an agent is ready for production.

By Naeem Akhtar · 11 min read

In short

  • A 95% per-step success rate completes a 20-step task only about 36% of the time, so per-step accuracy misleads.
  • Evaluate five things together: task completion, tool-call accuracy, trajectory, cost and latency, and groundedness.
  • Combine offline evaluation in CI with online scoring of real traffic. Offline catches regressions; online catches drift.
  • Gate promotion from dev to production on defined thresholds for every dimension.
01

Why per-step accuracy is the wrong headline metric

It's intuitive to ask "how accurate is the model?" and treat a high per-step accuracy number as evidence the agent is production-ready. It isn't, because failures compound across a multi-step task. An agent that's correct 95% of the time on each individual step completes an end-to-end task correctly only about 36% of the time across a 20-step chain — reliability multiplies against you with every additional step, tool call, or decision point. Evaluation is also what decides when an agent may act alone, which is the focus of our human-in-the-loop workflow design patterns.

This is why task completion — measured on the actual end state, not on whether each intermediate step looked reasonable — is the metric that matters, and why it needs to be measured on realistic multi-step tasks rather than isolated single-turn prompts.

02

The five dimensions that cover the real failure surface

DimensionWhat it catches
Task completionWhether the end state is actually correct, verified against ground truth — not whether the transcript looks plausible.
Tool-call accuracyWhether the agent picked the right tool, with the right arguments, at the right step.
Trajectory / step countWhether it reached the right answer via a sane path — an agent that loops five extra times before succeeding is a reliability risk waiting to happen under load.
Cost and latencyWhether the agent is economically and practically viable to run at the volume the business actually needs.
GroundednessWhether the agent's claims are actually supported by retrieved context or tool output, rather than a confident hallucination stitched in.

Shipping against only one or two of these — usually just task completion — is how teams end up with an agent that scores well in testing and then quietly burns through budget or produces ungrounded answers in production.

03

pass@k: measuring reliability, not just correctness

A single successful run doesn't tell you much about whether an agent is reliable — LLM outputs are non-deterministic, and a task that succeeds once can fail the next time on the exact same input. pass@k measures whether the agent completes the same task class successfully across k independent attempts, typically k=3 to k=10.

Why this matters

An agent with a high single-run success rate but a low pass@k score is telling you it's inconsistent — which is often worse for user trust than an agent that's consistently mediocre, because inconsistency is unpredictable and hard for a team to build workarounds around.
04

Offline and online evaluation are both required, not either/or

Offline evaluation runs a fixed dataset of known tasks with known-good outcomes, in CI, on every meaningful change — this catches regressions reproducibly before anything reaches a real user. Online evaluation scores real production traffic as it happens, catching drift, novel failure modes offline testing never anticipated, frustrated users, and adversarial or edge-case inputs that a curated test set wouldn't include.

  • Offline: fast, reproducible, catches known failure modes before release, but blind to anything not represented in the test set.
  • Online: catches real drift and novel failures, but by definition happens after the agent is already live in front of users.
  • Teams that skip offline testing ship regressions repeatedly. Teams that skip online monitoring find out about drift from a support ticket instead of a dashboard.
05

Eval-gated promotion: making this a process, not a one-time check

The teams that consistently ship reliable agents treat evaluation as a continuous discipline: automated metrics for coverage, model-based screening for efficiency at scale, and human expert judgment for the correctness only domain knowledge can verify — combined, not chosen from. No agent version should move from development to staging, or staging to production, without meeting defined thresholds on all five dimensions simultaneously. This is exactly what a decisioning-layer rebuild caught on a vendor compliance pipeline we worked on: the original set of 24+ disconnected automations had no shared evaluation or visibility, so failures were discovered by a partner complaining, not by a dashboard. A single pipeline with a standing evaluation set turns that into something the team can see coming.

06

Why long chains compound errors

A single-step accuracy figure hides how quickly reliability decays across a multi-step task. If every step succeeds independently with the same probability, the chance of completing the whole task is that probability raised to the number of steps.

Per-step reliability5 steps10 steps20 steps
90%59%35%12%
95%77%60%36%
98%90%82%67%
99%95%90%82%
99.5%98%95%90%

Real steps are not fully independent, and retries and checks can improve the odds, but the pattern holds: shorter chains, verification steps and narrow tools matter more than a slightly better model.

07

A sample agent scorecard

DimensionMetricExample gate before release
Task completionEnd-to-end success rate and pass@k on the test setNo drop versus the current version; target set per workflow
Tool-call correctnessValid calls with correct argumentsInvalid or unsafe calls at or near zero
GroundednessClaims supported by retrieved or tool-provided evidenceUnsupported claims below an agreed rate
Safety and policyRefusals and escalations when requiredAll red-team cases handled correctly
CostCost per completed taskWithin budget at expected volume
Latencyp50 and p95 time to completionWithin the user-facing target
Escalation qualityHuman handoffs with sufficient contextReviewers rarely need to re-ask basic questions
08

Tooling landscape

Evaluation tools change quickly, so choose by the workflow you need rather than by name. Categories worth covering:

NeedExamples of tools in this spaceNotes
Tracing and observabilityLangSmith, Langfuse, OpenTelemetry-based tracingCapture every step and tool call per run
Offline test runs and comparisonsBraintrust, promptfoo, custom CI scriptsRun a fixed set on every change and diff results
RAG-specific metricsRagas, DeepEvalRetrieval and groundedness scoring
Human review workflowsAnnotation tools or an internal queueCalibrates automated scoring against people
Cost and latency monitoringCloud monitoring plus model-usage dashboardsTrack per task, not just per token
09

Pitfalls when using an LLM as the judge

  • Judges favour longer, more confident answers even when they're wrong; use rubrics with specific criteria.
  • Scores drift when the judge model changes; pin the judge version and re-baseline on changes.
  • A judge can share blind spots with the model being judged; validate it against human-labelled samples.
  • Position and wording bias affect pairwise comparisons; randomise order and test with swapped positions.
  • Never let the judge be the only gate for high-risk behaviour; keep deterministic checks and human review for those.
10

Frequently asked questions

Why does an agent with 95% per-step accuracy fail so often end-to-end?

Because reliability compounds multiplicatively across steps: 0.95 raised to the 20th power is roughly 0.36. A long multi-step task with many decision points or tool calls needs much higher per-step reliability than an intuitive read of "95% accurate" suggests, which is why task-completion rate on realistic multi-step tasks is a more honest metric than per-step accuracy alone.

What is pass@k in AI agent evaluation?

pass@k measures whether an agent can successfully complete the same task class across k independent attempts (commonly k=3 to k=10), rather than relying on a single run. It's a reliability metric — an agent that succeeds inconsistently across repeated attempts on the same task is a production risk even if any individual run looks fine.

Do I need both offline and online evaluation for an AI agent?

Yes. Offline evaluation (a fixed set of known tasks run in CI on every change) catches regressions before release but is limited to what's in the test set. Online evaluation (scoring real production traffic) catches drift and novel failure modes offline testing can't anticipate, but only after the agent is already live. Relying on only one leaves a real gap.

Should cost and latency be part of an AI agent's evaluation criteria?

Yes — an agent that's accurate but slow or expensive to run isn't shippable at real business volume. Cost and latency should be tracked as first-class evaluation dimensions alongside accuracy, tool-call correctness, and groundedness, not measured separately or added as an afterthought once accuracy looks acceptable.

What metrics should I track for an AI agent in production?

Task completion (including pass@k), tool-call correctness, groundedness, safety and policy adherence, cost per completed task, latency and the quality of human escalations. Track them offline in CI and online on live traffic.

Can I use an LLM to evaluate my AI agent?

Yes, with care. Use clear rubrics, pin the judge model version, validate it against human-labelled examples and keep deterministic checks and human review for high-risk behaviour.

How large should an agent evaluation set be?

Start with a few dozen realistic tasks that cover your main flows and known failure cases, then grow it with every production incident or novel failure you find.

Work with Eveonix

Need help with Agentic AI Development? Talk to the engineers.

Tell us what you want to build. You'll get a straight answer on scope, approach and cost from the people who do the work.