A regression test for a local model in CI does two things: it runs a fixed battery of checks against a pinned model, quantization, and runtime, and it compares the resulting scores against a committed baseline. If a score drops past a threshold, the build fails.
That is the whole idea, and it is worth stating plainly because most of the engineering effort goes into the second half: making the checks deterministic enough that a red build means a real capability loss rather than sampling noise. This guide is written for practitioners who already run local models through Ollama, llama.cpp, vLLM, or a similar runtime and want a gate in their pipeline before a quantization swap, runtime upgrade, prompt edit, or RAG change reaches users. Two open-source projects illustrate the pattern from different angles: probelock, a capability lockfile that scores tool-calling probes with code instead of an LLM judge, and CIT-Eval Harness, a behavioral prompt-suite runner that emits a PASS/FAIL gate from violation-style metrics.
What counts as a regression for a local model
Cloud model APIs change on the provider’s schedule. Local models change on yours, which is both the advantage and the trap. The weights file may stay identical while everything around it moves:
- Quantization. Swapping Q8_0 for Q4_K_M reduces memory, but it can also reduce the model’s ability to emit valid tool arguments or pick the right tool.
- Runtime and kernels. A new build of the inference server, a different GPU backend, or changed context handling can shift outputs even with the same weights.
- Prompt and chat templates. A reworded system prompt or a corrected template changes the token stream the model actually sees.
- Tool schemas. Adding, renaming, or re-describing a function changes what the model can select and how it fills arguments.
- Retrieval. Different chunks in context produce different answers to the same question.
Two broad families of checks catch these. Capability checks ask whether the model can still do a narrow, verifiable job — emit schema-valid JSON, call the correct tool, avoid inventing tools that do not exist. Behavioral checks run a broader prompt suite and score each response against a violation rubric. Both families can be scored deterministically; both can also be scored by an LLM judge, which buys coverage at the cost of reproducibility.
Prerequisites before you touch CI
Wiring a gate into a pipeline is the easy part. The following pieces need to exist first, or the gate will be noise:
- A pinned identity for the thing under test. Record model name, quantization, runtime and version, context length, and sampling settings. A bare model family name is not a pinned identity; the model plus quantization plus runtime is closer.
- A fixed probe set. The same inputs must run every time. Deriving probes from your shipped tool schemas, as probelock does, keeps them aligned with what your agent actually exposes.
- A deterministic scorer. JSON-schema validation, exact match, and tool-name checks all produce the same score for the same output. Prose quality does not.
- A committed baseline. The baseline is an artifact you version alongside the code, not a number someone remembers.
- A threshold policy. Decide in advance how large a drop is a failure, and how much run-to-run variation you will tolerate.
- A runner that can host the model. Local inference generally means a self-hosted runner with the model already pulled and cached, rather than a fresh cloud runner that downloads gigabytes on every job.
Pattern 1: a capability lockfile with probelock
probelock describes itself as a capability lockfile for local models that catches silent tool-calling regressions in CI, deterministically and without an LLM judge. Its design has three practical consequences.
Probes come from your tool schemas. You point it at the OpenAI-style tool definitions your agent already ships, and it generates a fixed, reproducible battery of capability checks. You write no test cases — which removes the most common reason teams never build a suite in the first place.
Scoring is code, not judgment. Every probe is scored by JSON-schema validation, exact match, or a tool-name check. There is no model grading a model, so a score of 0.67 means the same thing on Tuesday as it did on Monday.
The gate is an exit code. probelock gate exits non-zero when a capability drops past the threshold, which is exactly what a CI job needs.
The capabilities it reports are specific and readable: arg_validity, arity_robustness, format_adherence, needle_in_tools, no_hallucinated_tool, required_args, structured_output, tool_discrimination, tool_permission, tool_restraint, and tool_selection. Each one maps to a failure mode an agent developer recognizes.
The project frames its distinction from general-purpose test frameworks directly: promptfoo is a test framework you author, while probelock is a lockfile you commit. That is a useful way to decide which you need. If you want to write bespoke scenarios, author them. If you want a committed artifact that fails when a swap degrades a known capability, a lockfile is the lighter tool.
Pattern 2: a behavioral gate with CIT-Eval Harness
CIT-Eval Harness approaches the problem from the behavioral side. Developed as evaluation infrastructure for Constitutional Identity Training research, it runs a structured prompt suite against any local LLM via Ollama, computes violation-style metrics, compares the run against a validated baseline, and emits a PASS/FAIL gate.
Its mechanics are worth understanding because they map onto the general problem of gating on fuzzy outputs:
- JSONL prompt suites with domain tags — prompts are labeled benign, edge, unsafe, or mixed, so a regression can be localized to a category instead of appearing as one aggregate number.
- A heuristic violation scorer producing reproducible scores in the range [0, 1] per prompt.
- Multi-threshold metrics — ViolRate and ViolAUC computed at configurable thresholds such as 0.5, 0.9, and 0.95.
- Baseline comparison with drift statistics — mean, p95, and max delta, which help separate a real shift from ordinary variation.
- A CI gate that exits with code 2 when
regression_rateexceeds a configurable threshold.
The harness also caches responses in JSONL, which makes repeated runs near-instant — a meaningful property in CI, where an evaluation job that takes twenty minutes tends to get disabled within a month. It ships with Docker support, per-domain aggregation, and a lightweight Tkinter GUI for inspecting runs locally.
What a regression actually looks like
The probelock README documents a comparison between llama-3.1-8b @ Q8_0 and the same model at Q4_K_M, both served through Ollama. This is the project’s own example, not an independent test, but it illustrates the shape of the output a gate produces:
| Capability | Baseline (Q8_0) | Candidate (Q4_K_M) | Delta | Status |
|---|---|---|---|---|
| arg_validity | 1.00 | 0.67 | -0.33 | REGRESSION |
| arity_robustness | 1.00 | 0.67 | -0.33 | REGRESSION |
| format_adherence | 1.00 | 1.00 | +0.00 | ok |
| needle_in_tools | 1.00 | 0.33 | -0.67 | REGRESSION |
| no_hallucinated_tool | 1.00 | 0.67 | -0.33 | REGRESSION |
| required_args | 1.00 | 1.00 | +0.00 | ok |
| structured_output | 1.00 | 0.33 | -0.67 | REGRESSION |
| tool_discrimination | 1.00 | 0.33 | -0.67 | REGRESSION |
| tool_permission | 1.00 | 0.67 | -0.33 | REGRESSION |
| tool_restraint | 1.00 | 0.67 | -0.33 | REGRESSION |
| tool_selection | 1.00 | 0.67 | -0.33 | REGRESSION |
The reported pattern is the one that matters: the smaller quant scores 0.33–0.67 on several capabilities where the larger quant scored 1.00. Two capabilities are unaffected. Without a baseline, the Q4 model would have looked fine in casual use — it still answers, still emits JSON most of the time — while quietly failing on tool discrimination and structured output. That is precisely the silent regression the gate exists to catch.
Wiring the gate into a pipeline
Both tools are ordinary command-line programs, so any CI system that can run a shell command and read an exit code can host them. The sketch below is illustrative scaffolding, not a documented configuration for either project — replace the placeholder commands with the flags from whichever tool you adopt.
name: model-regression
on: [pull_request]
jobs:
gate:
runs-on: [self-hosted, gpu]
steps:
- uses: actions/checkout@v4
- name: Ensure local runtime is up
run: ollama serve &
- name: Capability gate
run: probelock gate # exits non-zero on a capability drop
- name: Behavioral gate
run: cit-eval-gate # placeholder: exits 2 when regression_rate exceeds threshold
Three operational notes apply regardless of tool. First, keep the model cached on the runner; pulling weights on every job is the fastest way to make a gate unpopular. Second, run the gate on pull requests that touch prompts, tool schemas, runtime pins, or quantization settings, rather than on every commit. Third, treat a baseline update as a deliberate, reviewable change — a commit that says you accept the new behavior — not as a command someone runs to make red go green.
Limits, tradeoffs, and failure modes
Deterministic scorers check only what code can check. Schema validity and tool selection are objective; tone, helpfulness, and open-ended reasoning are not. The tradeoff is explicit: probelock avoids an LLM judge to keep scores reproducible, which means it will not notice a model that becomes terser, more evasive, or less useful while still emitting valid JSON. CIT-Eval’s violation-style metrics cover a different slice — behavioral drift against a rubric — but they are heuristic scorers, not a general quality measure.
Sampling settings leak into results. If temperature, top-p, or the random seed vary between runs, scores will vary too. Pin them, or expect occasional flaky gates.
Thresholds are a judgment call. Set them too tight and the gate cries wolf until someone disables it; set them too loose and it passes a real regression. Drift statistics such as mean, p95, and max delta exist to help make that call with data rather than instinct.
Baselines go stale. Every accepted model change invalidates the baseline. Budget time for regenerating it and reviewing the diff.
Both projects are young and small. probelock was created in June 2026 and CIT-Eval Harness in January 2026; at the time of writing they are early-stage repositories with single-digit star counts. That is not a reason to avoid them — it is a reason to read the code, pin a version, and be prepared to maintain your own harness if a project stalls. Licensing is permissive in both cases: probelock under Apache License 2.0 and CIT-Eval Harness under the MIT License.
Choosing a starting point
If your agent ships OpenAI-style tool definitions and your main worry is tool-calling correctness after a quantization or runtime change, start with probelock: probe generation is free, scoring is deterministic, and the gate is a single exit code. If your concern is broader behavioral monitoring — prompt suites with domain tags, violation rates, drift statistics — start with CIT-Eval Harness. There is no conflict in running both; they check different things and fail for different reasons.
The concrete next step is small: pick one model, one quantization, and one runtime, record a baseline this week, and attach the gate to a single pull-request job. Add a second model or a stricter threshold only after the first gate has proven it fails for real reasons and passes when nothing changed.
If you have already wired a local-model gate into a pipeline, the community would benefit from the details — your runtime, quantization, threshold, and the first regression it caught. Share the configuration and the failure output so others can compare notes.
FAQ
Do I need an LLM judge to regression-test a local model?
No, and probelock deliberately avoids one: every probe is scored by code using JSON-schema validation, exact match, or a tool-name check, which keeps scores reproducible. The tradeoff is coverage — a deterministic scorer will not catch changes in tone or open-ended quality, only in the specific behaviors it checks.
What is the difference between probelock and promptfoo?
The probelock README states the distinction directly: promptfoo is a test framework you author, while probelock is a lockfile you commit. With probelock, probes are derived from your existing OpenAI-style tool definitions and you write no test cases; with a general test framework, you author the scenarios yourself.
Can these gates run on hosted CI runners?
The tools themselves are ordinary command-line programs, but local inference needs the model available on the machine. In practice that means a self-hosted runner with the model already cached, rather than a fresh hosted runner that downloads weights on every job. CIT-Eval Harness also caches responses in JSONL, which shortens repeated runs.
How do I keep the gate from failing randomly?
Pin your sampling settings, including temperature and seed, so identical inputs produce identical outputs. Then use the drift statistics a harness reports — mean, p95, and max delta — to set a threshold that tolerates normal variation but still catches a genuine capability drop.