AIForum.

How to Automate Prompt Testing in WordPress with an AI API

On this page

THE SHORT ANSWER

Automate prompt testing in WordPress by registering eval suites during wp_ai_evals_init, defining cases with inputs, tasks, expected values, and evaluators, then running tagged subsets in CI. The AI API supplies the model call; the eval layer supplies repeatability and pass/fail reasons.
Quick facts

Automating prompt testing in WordPress means treating prompts and model calls as testable components rather than ad hoc experiments. The documented pattern in Automattic’s wp-ai-evals package is to register a suite during the wp_ai_evals_init action, define cases that pair an input with a task and an expected value, score outputs with evaluators, and select subsets with tags. The AI API supplies the model call; the eval layer supplies repeatability, pass/fail reasons, and CI-friendly structure. This guide is for WordPress developers and technically curious site builders who want a reproducible way to check prompts before they reach production.

What the eval vocabulary gives you

The wp-ai-evals authoring documentation defines a small vocabulary that maps well to automated prompt testing:

  • Suite: a namespaced collection owned by one plugin or feature.
  • Case: one input, task, expected value, evaluator set, and metadata record.
  • Task: the callable that invokes the system under test.
  • Evaluator: scores task output from 0 to 1 and returns pass/fail plus a reason.
  • Tag: an orthogonal selector such as smoke, safety, quality, or slow.

The hierarchy is suite → case. Tags let you run flexible subsets without forcing each case into one group. That matters when a fast pull-request check should run only smoke cases, while a nightly job runs quality, safety, and slow cases.

Prerequisites and integration points

Before automating anything, confirm four things:

  1. A WordPress plugin or feature to test. The suite should belong to the plugin or feature that owns the prompt.
  2. The eval package. The Hello Dolly AI sample uses automattic/wp-ai-evals as a Composer development dependency, as described in the sample README.
  3. An AI API path. The sample uses the WordPress AI Client and a connector-backed agent with an explicit WP_AI_Client_Ability_Function_Resolver allowlist. It selects models through the Connectors API and does not store a key or model ID in the plugin.
  4. A runner. The retrieved documentation does not prescribe a single CLI command. It gives the integration point: register suites during wp_ai_evals_init. The sample also uses PHPUnit tests to verify the surrounding WordPress integration.

Decide whether you are testing the task layer or a REST route. The sample’s logged-in REST chat route is protected by a REST nonce, capability check, input limits, and a per-user rate limit. Route-level eval cases must supply an authenticated context and account for those controls; task-level cases can call the underlying function directly.

Step 1: Register a suite

Small plugins can register suites directly during wp_ai_evals_init. Larger plugins can load a directory:

add_action( 'wp_ai_evals_init', static fn(Registry $registry) => $registry->load_directory( __DIR__ . '/suites' ) );

The loader accepts standalone PHP files that return a Suite, or modular directories. A documented layout looks like this:

evals/suites/
├── content-quality.php
└── agent-safety/
    ├── suite.php
    └── cases/
        ├── refuses-lyrics.php
        └── grounding/
            └── sources.php

Case files may return one EvaluationCase or an iterable of cases. Files load in stable lexical order and retain normal PHP access to plugin callbacks, tasks, and custom evaluators.

Step 2: Define cases and datasets

A case is where prompt testing becomes concrete. The documentation shows datasets expanding into stable cases with ordinary iterables and Suite::add_cases():

$rows = [
    'caching' => ['input' => 'Explain object caching.', 'expected' => 'database'],
    'blocks'  => ['input' => 'Explain a block theme.', 'expected' => 'Site Editor'],
];

$suite->add_cases(
    (static function () use ($rows): iterable {
        foreach ($rows as $id => $row) {
            yield EvaluationCase::make($id)
                ->input($row['input'])
                ->task(static fn(string $input) => my_plugin_answer($input))
                ->expected($row['expected'])
                ->evaluate_with(new ContainsText())
                ->tag('dataset', 'quality');
        }
    })()
);

Use stable case IDs. They become the labels you read in CI output and failure reports. If the same prompt is tested against several models, keep the input and expected value stable and vary the task or model target instead.

Step 3: Choose tasks that exercise the AI API

The task is the callable that invokes the system under test. The sample README lists callable, Ability, and direct Prompt tasks. In a WordPress AI API context, a task might call a plugin function, invoke a registered Ability, or send a prompt through the Connectors API.

Keep the task focused. If you are testing whether a prompt produces a grounded answer, the task should call the same code path that production uses. The Hello Dolly AI sample uses source-backed knowledge functions to keep the model grounded, and its system instruction prohibits reproducing song lyrics. Those are prompt and architecture decisions that eval cases can then check.

Step 4: Pick evaluators for the claim you are testing

Evaluators score output from 0 to 1 and return pass/fail plus a reason. The sample covers exact, substring, regex, JSON Schema, callback, latency, and LLM-judge evaluators, including a weighted multi-item grounding rubric. Use the cheapest evaluator that can answer your question:

Evaluator typeBest forTradeoff
ExactFixed expected strings or IDsBrittle when wording varies
Substring / regexRequired phrases or patternsCan pass without understanding
JSON SchemaStructured output and tool argumentsDoes not judge content quality
CallbackCustom plugin logicNeeds maintenance as code changes
LatencyPerformance budgetsEnvironment-dependent
LLM-judgeOpen-ended quality and groundingCost, nondeterminism, judge choice

For safety and factual grounding, a judge evaluator is often necessary, but it should not be the only line of defense. The sample pins a judge model independently and supports model-aware agent callbacks and comparison summaries. That helps separate prompt changes from model changes.

Step 5: Add metadata, repetitions, and tags

The documentation describes metadata, repetitions, and tag-based subsets. Repetitions matter because model output varies. A single pass does not prove stability. Metadata helps you record which model, provider, prompt version, or dataset row produced a result.

Tags keep the suite usable. A practical split is:

  • smoke: a few fast cases that must pass on every pull request.
  • safety: refusal, privacy, or policy checks that may run on a schedule.
  • quality: broader answer quality and grounding checks.
  • slow: expensive judge or cross-provider comparisons for nightly runs.

These tag names come from the documentation’s examples. You can add your own, but keep the set small enough that contributors know what to run.

Step 6: Wire the suite into CI and local workflow

Automation is mostly plumbing: install Composer development dependencies, bootstrap WordPress, run the eval runner with a tag filter, and fail the job when a required case fails. The retrieved documentation does not include a specific CLI command, so use the runner exposed by your project or the package. The important part is that the same suite runs locally and in CI.

Store evaluator reasons, not just pass/fail counts. When a case fails, the reason is the first diagnostic clue. Also record the model target and provider because a prompt that passes on one model may fail on another. The sample supports exact cross-provider model targets, which makes that comparison explicit rather than accidental.

Diagnosing common failures

A format case fails after a prompt edit

Check the expected value and the evaluator. Exact and regex evaluators are intentionally strict. If the output is correct but worded differently, move the check to a substring or LLM-judge evaluator, or update the expected value with a documented reason.

A grounding case fails

This usually means the model made a claim that the source-backed functions did not support. Add or tighten the source data, adjust the system instruction, and keep the grounding rubric in the suite. The sample’s weighted multi-item grounding rubric is designed for this kind of check.

A route-level case fails with permission or nonce errors

The sample’s REST route requires a logged-in user, a REST nonce, capability checks, input limits, and a per-user rate limit. Test with a valid user context, and separate rate-limit tests from functional tests so one does not mask the other.

Results are flaky

Increase repetitions, pin the judge model, and separate fast deterministic evaluators from slow probabilistic ones. If a case is flaky only on one provider, record that as a model or provider difference rather than a universal prompt failure.

Limitations and tradeoffs

Automated prompt testing does not remove uncertainty. LLM-judge evaluators are probabilistic, cost tokens, and depend on the judge model. Exact evaluators are cheap and fast but brittle. Cross-provider tests improve coverage but multiply cost and latency. Local models can be tested through the same provider-neutral Connectors API path, but the sources do not provide benchmark numbers, and results will depend on the model, quantization, context length, and hardware.

There is also no universal pass threshold. A safety case may require 100% pass, while a quality case may tolerate a lower score. Define the threshold per tag or per case, and document it next to the suite.

An illustrative WordPress suite

The Hello Dolly AI sample is a documented example, not a benchmark result. According to its README, it provides a dynamic chat block, a connector-backed agent, three read-only WordPress Abilities, and two eval suites registered through automattic/wp-ai-evals. Its two suites contain nine cases. The sample also demonstrates parameterized datasets, metadata, repetitions, tag-based subsets, callable/Ability/direct Prompt tasks, exact cross-provider model targets, model-aware agent callbacks, comparison summaries, and an independently pinned judge model.

For a new plugin, a smaller version of that structure is enough: one suite, three cases, and two tags. Add a smoke case for the main answer path, a safety case for a refusal or policy boundary, and a quality case with a judge evaluator. Run it locally, then add it to CI.

Next action

Register one suite during wp_ai_evals_init, define a dataset-backed case with a stable ID, and run it against your AI API path. Once the smoke case passes reliably, add a safety tag and a scheduled quality run. If you get stuck, share your WordPress version, provider, model target, suite layout, and the evaluator reason in the Ai-Forum.org community so others can compare environments and reproduce the issue.

FAQ

Do I need a live AI API to run prompt tests?

Not for exact, substring, regex, JSON Schema, or callback evaluators if the task is mocked. Tasks that invoke a model need a configured provider. The Hello Dolly AI sample uses the Connectors API for provider-neutral model selection.

How many eval cases should I start with?

Start small. The Hello Dolly AI sample has two suites with nine cases, but a new plugin can begin with one suite and three cases: a smoke case, a safety case, and a quality case.

Can I test a local model with this setup?

The documented architecture uses provider-neutral model selection through the Connectors API, so a local model can be used if it is exposed through a compatible provider path. Results will depend on the model, runtime, and hardware; the sources do not provide benchmark numbers.

What is a judge model?

A judge model is the language model used by an LLM-judge evaluator to score open-ended output. The sample pins a judge model independently so prompt changes can be separated from judge changes.

How do I handle flaky model outputs?

Use repetitions, pin the judge model, and separate fast deterministic evaluators from slow probabilistic ones. If flakiness appears only on one provider, record it as a provider or model difference.

References

  1. https://github.com/Automattic/wp-ai-evals/blob/HEAD/docs/authoring-evals.md
  2. https://github.com/Automattic/wp-ai-evals/blob/HEAD/examples/hello-dolly-ai/README.md

Related terms

Suite
A namespaced collection of eval cases owned by one plugin or feature.
Case
One input, task, expected value, evaluator set, and metadata record.
Task
The callable that invokes the system under test, such as a plugin function or AI API call.
Evaluator
A scorer that rates task output from 0 to 1 and returns pass/fail plus a reason.
Tag
An orthogonal selector such as smoke, safety, quality, or slow that creates flexible test subsets.
LLM-judge
An evaluator that uses a language model to score open-ended output quality or grounding.
Connectors API
The WordPress API used in the sample for provider-neutral model selection without storing a key or model ID in the plugin.
Abilities API
WordPress APIs that can expose read-only tools with JSON schemas and permission callbacks for AI agents.

Leave a Reply

Your email address will not be published. Required fields are marked *