Evaluating an AI agent before production means testing its behaviour across a repeatable set of tasks rather than eyeballing a handful of final answers. A workable pre-production suite combines four things: deterministic checks, LLM-as-a-judge or statistical scoring, simulated multi-turn scenarios, and tracing that records what the agent actually did. That suite is then re-run whenever a prompt, model, tool, or retrieval configuration changes.
This guide is aimed at practitioners who already have an agent running — in a terminal, a notebook, or a prototype workflow — and want a defensible answer to the question “is this reliable enough to ship?” It focuses on what to measure, how to build the first suite, and where the approach breaks down. The same harness applies whether the agent’s model is hosted by a provider or served locally.
Why the terminal vibe check breaks down
The typical starting pattern is easy to recognise. You wire up a model, hand it a few tools, run three or four prompts in a terminal, and judge the output by eye. If the answer looks reasonable, you ship it. Google’s developer guidance on agent evaluation describes exactly this pattern and explains why it fails: you tweak a system prompt to fix an edge case in one workflow, and the agent silently stops calling a required tool in another (Google Developers Blog, October 2026).
Agents also fail differently from ordinary software. An agent can get stuck in a retry loop, fire off multiple tool calls, and still return a polite, plausible-sounding final answer. The final answer is the one thing a human reviewer tends to look at, and it is the least informative signal available.
The costs of that hidden failure are concrete: token and cloud spend rise quietly, downstream API rate limits and quotas get exhausted, users wait on a loading spinner, and duplicate side effects can be written into external systems — all while the output looks fine on the surface. Trust is hard to win back after the first bad impression.
What “reliability” means for an agent
Agentic applications add multi-step reasoning, dynamic tool use, retrieval of external knowledge, and long-running conversations. According to Maxim AI’s pre-production evaluation framework, that complexity produces several characteristic risks: latent failure modes that appear only under certain personas or tasks, poor retrieval behaviour that leads to hallucination or irrelevant output, inconsistent prompt behaviour across versions and environments, and uncontrolled token usage and latency under realistic loads (Maxim AI).
That translates into a set of dimensions worth measuring separately:
- Task success — did the agent complete the job the user asked for?
- Tool-call correctness — was the right tool called, with the right arguments, the right number of times?
- Trajectory efficiency — how many steps, retries, and loops did it take to get there?
- Cost and token usage — measured per task, not per session.
- Latency — including the slow tail, not just the average.
- Side effects — what did the agent change in external systems, and was that change idempotent?
- Retrieval quality — for RAG agents, whether the retrieved context was relevant and sufficient.
- Consistency — whether the same task behaves the same way across prompt versions and environments.
The four building blocks of a pre-production evaluation suite
Maxim’s framework argues that a disciplined approach must combine simulation, LLM evaluation (LLM-as-a-judge plus statistical metrics), deterministic checks, and real-world observability. Each covers a blind spot the others have.
Deterministic checks
These are ordinary assertions: the agent must call a specific tool, must not exceed a fixed number of steps, must return a response matching a schema, must not create more than one record. They are cheap, fast, and stable, and they catch the regressions that prompt edits tend to introduce. Their limit is coverage — they only test what you thought to assert.
LLM-as-a-judge and statistical metrics
For open-ended output quality, a judge model scores responses against a rubric, and aggregate metrics track trends across a task set. Judges are the practical way to evaluate questions such as “is this summary faithful to the retrieved documents?”, but they are noisy, so rubrics need to be explicit and judge scores should be spot-checked against human labels.
Simulation
Simulation generates multi-turn conversations, personas, and adversarial inputs that a small hand-written prompt list will never reach. It is how you find the latent failure modes that appear only under a particular task or user style.
Tracing and observability
Tracing records the full trajectory — tool calls, arguments, intermediate steps, timings — so that a failed evaluation produces a diagnosable artifact rather than a single bad string. The same instrumentation carries into production, where it becomes the source of new test cases.
A practical sequence for a first evaluation suite
None of this requires a commercial platform. A spreadsheet of tasks, a script, and a JSON log will do for the first iteration; the point is repeatability, since any repeatable evaluation system is better than none.
- Collect 10–20 real tasks. Pull from actual usage, support questions, and known edge cases rather than inventing prompts. Include at least a few tasks that are known to be hard.
- Write pass/fail criteria per task before running anything. State the expected tool calls, the acceptable final answer, the step ceiling, and the side effects that must not occur. Deciding this after seeing output is how criteria drift.
- Implement deterministic assertions first. They give you a stable baseline and catch prompt regressions immediately.
- Add judge-based scoring for the open-ended parts. Use a rubric with explicit criteria, and keep a small human-labelled sample to check whether the judge agrees with you.
- Add simulated scenarios once the basics are stable. Multi-turn conversations and different personas are the next layer of coverage.
- Record full trajectories, not just final answers. A retry loop or a duplicated tool call is invisible in the final string.
- Gate changes on the suite. Run it before and after every prompt, model, or tool change, and treat new failures as blocking. Keep the passing set as a regression baseline.
Expected result: after a full run you should be able to say, for each task, whether it passed, which assertion failed, and how many steps and tokens it consumed. If you cannot answer those three questions, the suite is not yet doing its job.
Illustrative scenario: an agent that publishes to a CMS
Consider an automation agent that drafts an article and creates a post in a content management system. (This is an illustrative design, not a reported deployment.) The visible output — a well-written draft — can look perfect while the agent has called the publishing tool twice, created duplicate posts, and burned several thousand tokens retrying a failed API call.
A minimal suite for that agent would assert that exactly one post is created per run, that the publishing call is idempotent or keyed, that the step count stays under a fixed ceiling, that latency stays within budget, and that the draft passes a faithfulness check against the source material. The duplicate-post failure is invisible to a reviewer reading the draft, and obvious to a deterministic assertion on the number of created records.
Running evaluations against local models
If the agent’s model is served locally, evaluation runs do not incur per-token API charges, which makes it practical to sweep a larger task set on every change. The tradeoff is hardware time, plus the fact that a local model’s failure modes are not the same as a hosted model’s — a suite that passes against a small local model does not predict behaviour on a larger hosted one, and vice versa. The harness itself is portable; the results are not.
Limitations and tradeoffs
Evaluation is a progressive practice, not a one-time gate. Several limits are worth stating plainly:
- Judges are noisy. LLM-as-a-judge scores vary between runs and models; without a human-labelled calibration sample, a score change may be judge variance rather than an agent regression.
- Deterministic checks only cover known failure modes. They are excellent regression detectors and poor discovery tools.
- Simulation is not production. Simulated personas and generated tasks approximate real usage; they do not replace it.
- Test sets drift. Prompts, tools, and user behaviour change, so the task set needs periodic refresh from real traces.
- Passing the suite is not a guarantee. Offline evaluation and production observability answer different questions, which is why the framework treats them as complementary layers rather than alternatives.
The next step
Pick ten real tasks, write down what “correct” means for each — including the tool calls and side effects — and run the agent against them before and after your next prompt change. That single loop converts an unverifiable vibe check into a regression test, and it is the foundation everything else builds on.
If you are building an evaluation suite for an agent, share your task set, environment, and the failures you found in the community discussion — concrete failure traces are more useful to other practitioners than pass rates alone.
FAQ
Why is checking the final answer not enough?
Because an agent can get stuck in a retry loop, make multiple redundant tool calls, or trigger duplicate side effects and still produce a polite, plausible final answer. The trajectory and the external effects carry the failure signal that the final string hides.
What is the minimum viable evaluation suite?
A repeatable task set with written pass/fail criteria, deterministic assertions on tool calls and side effects, a judge-based score for open-ended quality, and trajectory logging. Simulation can be added once those basics are stable.
How do I catch retrieval problems in a RAG agent?
Evaluate retrieval behaviour as its own dimension. Poor retrieval is a documented cause of hallucinated or irrelevant output, so score whether the retrieved context was relevant and sufficient, separately from whether the final answer reads well.
Do offline evaluations replace production monitoring?
No. The pre-production framework treats offline evaluation and real-world observability as complementary layers: offline tests gate changes before release, while production tracing surfaces failure modes that the test set never contained.