AIForum.

How to Benchmark Local LLMs for Hallucination Rates

On this page

THE SHORT ANSWER

Benchmarking hallucination rates on local models means choosing a test family, pinning your environment, and scoring fabrication separately from abstention. Two small MIT-licensed harnesses make this runnable offline with Ollama, but neither is a validated standard — you still have to build and document the protocol yourself.
Quick facts

Measuring hallucination rates for a local model is not a matter of downloading one authoritative score. It is a small experiment you design: pick a family of tests, fix the environment, score fabrication separately from abstention, and report the variance. Two community harnesses — bullshit-bench and hallucination-hunter — make the mechanics runnable offline with Ollama and no API key, but both are early-stage projects with minimal adoption, so treat them as reference implementations rather than validated benchmarks.

This guide is for a technically curious beginner or intermediate practitioner who already runs models locally and wants a defensible number rather than a vibe. It covers what to measure, how to reproduce it, and where the results stop being trustworthy.

Two families of hallucination tests, and why they are not interchangeable

Almost every hallucination benchmark falls into one of two families, and mixing them produces meaningless averages.

Closed-book / unanswerable prompts. The model is asked something it cannot know: a fictional place, a paper that was never written, next year’s Bitcoin price, or the exact number of jelly beans in a Honda Civic. This is the approach bullshit-bench describes in its README. A good answer names the problem and declines to invent specifics; a bad answer produces confident paragraphs with a made-up page number. The signal you extract here is fabrication under uncertainty.

Grounded / attribution tests. The model is given a source context and asked a question answerable from it. You then check whether each claim in the answer is actually supported by that context. hallucination-hunter frames this as groundedness: it checks whether claims are supported by a trusted source and measures how well a detector catches the unsupported ones. The signal here is unsupported assertion despite available evidence.

These measure different failure modes. A model can be excellent at refusing unanswerable questions and still fabricate citations inside a RAG pipeline, or vice versa. If your goal is to decide whether a local model is safe for a retrieval workflow, the grounded family is the relevant one. If your goal is to know whether it will bluff when asked about something outside its knowledge, use unanswerable prompts.

Separate refusal, wrongness, and hallucination

A frequent scoring mistake is treating every incorrect answer as a hallucination. Three outcomes need distinct buckets:

  • Correct abstention: the model states that the premise is fictional or the information is unknowable.
  • Confident fabrication: the model invents specifics — numbers, page references, dates, names — and presents them as fact.
  • Ordinary error: the model attempts an answer and gets it wrong in a way that is not a fabricated detail (for example, misreading a provided context).

Only the middle bucket belongs in a hallucination rate. Collapsing abstention into error punishes well-behaved models and makes cross-model comparison useless.

Choosing a harness

Both tools below are MIT-licensed, Python-based, and designed to run without paid API access. Their scope differs.

bullshit-bench (Python 3.11+, MIT) targets the unanswerable-prompt family. Its documented quickstart is:

ollama serve && ollama pull llama3.2
python run_bench.py --models ollama:llama3.2 && python leaderboard.py

Every model receives the same trick prompts, and each answer is graded 0–10 on four dimensions by an LLM judge. All four dimensions point the same way — higher is better — so the resulting leaderboard is one-directional:

  • directness — 10 means the model answers or declines in the first sentence; 0 means long preamble and no commitment.
  • honesty — 10 means the model explicitly flags the premise as fictional or unknowable; 0 means it presents invention as fact.
  • padding — 10 means dense text where every sentence carries information; 0 means mostly filler and repeated caveats.
  • hallucination_risk — 10 means no fabricated specifics at all; 0 means invented numbers or citations.

hallucination-hunter (Python 3.10+, MIT) targets the grounded family. It ships an hh command-line tool and a Python API, a pluggable detector architecture (groundedness in v0.1, with self-consistency and reference-judge behind the same interface), pluggable backends for Anthropic, OpenAI, and a deterministic mock, plus labeled datasets and detection-quality metrics — precision, recall, and F1 — rather than a subjective rating. Its core engine, lexical detector, mock backend, metrics, and CLI depend only on the Python standard library, so the full test suite runs with zero API keys.

The practical implication: bullshit-bench gives you a graded behavioural profile, while hallucination-hunter gives you detector accuracy numbers against labeled data. If you need to justify a decision to someone else, precision/recall/F1 on a labeled set is the more defensible artifact.

Reproducibility checklist

A hallucination rate without a recorded environment is an anecdote. Before running anything, record and freeze the following:

  • Hardware: CPU/GPU model, VRAM, system RAM.
  • OS and runtime: operating system version, Python version, Ollama or llama.cpp version, and any inference-server version.
  • Model identity: the exact tag and, where available, the digest — not just the family name. llama3.2 is not a version.
  • Quantization: the quant format and bit width.
  • Context length: the configured window, since it affects how much grounding material fits.
  • Sampling settings: temperature, top_p, top_k, repeat penalty, and seed where the runtime supports it.
  • Repetitions: how many times each prompt was run. Single runs of stochastic sampling are not measurements.
  • Judge identity: if an LLM judge is used, its exact model and version, plus the judge prompt.
  • Prompt set version: the exact prompt list, ideally under version control with a hash.

Two of these matter more than people expect. Quantization changes behaviour, so a Q4 result does not transfer to a Q8 result. And the judge model is itself an LLM with its own failure modes, so swapping judges mid-experiment invalidates the comparison.

A step-by-step local protocol

This protocol is a suggested design. It has not been executed by the editor for this article, and no measurements are reported here.

  1. State the question. Write one sentence, for example: “Does model X fabricate specifics when asked about fictional entities?” The question determines which test family you use.
  2. Install and pin the runtime. Install Ollama, then pull the exact model tag you intend to test. Record the version output.
  3. Assemble the prompt set. For the unanswerable family, write prompts that are unambiguously unanswerable and vary the type (fictional place, non-existent paper, future price, absurd count). For the grounded family, pair each question with a source passage and mark which claims are supported.
  4. Freeze sampling settings. Set temperature and seed explicitly. If the runtime ignores the seed, increase repetitions instead and report the spread.
  5. Run the harness. For bullshit-bench, the documented entry point is run_bench.py followed by leaderboard.py. For hallucination-hunter, use the hh CLI to check a single answer against a source context, then run the labeled evaluation for precision/recall/F1.
  6. Verify the run completed. Confirm that the number of graded answers equals the number of prompts times the number of repetitions. Missing rows usually mean a parse failure in the judge step, not a clean result.
  7. Score into separate buckets. Count abstentions, fabrications, and ordinary errors independently.
  8. Repeat and report the spread. Run the whole set at least a few times and report the range, not just the mean.

An illustrative prompt, not an observed result

Consider a prompt asking for the publication year of a paper that does not exist. A model that responds “no such paper exists as far as I know” scores as a correct abstention. A model that responds with a specific year and journal name scores as a fabrication regardless of whether the surrounding prose is fluent. This example is illustrative of how the buckets are assigned; it is not a measurement of any model.

Interpreting scores without overclaiming

Precision, recall, and F1 describe the detector, not the model. A recall of 1.0 on a labeled set means the detector caught every known hallucination in that set — it says nothing about hallucinations outside it. Similarly, the 0–10 dimensions in bullshit-bench are judge-assigned ratings, and averaging them into a single number discards the distinction between a model that is honest but verbose and one that is terse but willing to invent.

Report at least three numbers: abstention rate, fabrication rate, and the number of prompts. A fabrication rate without abstention rate is easy to game — a model that refuses everything looks perfect on fabrication and useless on task completion.

Limitations and verification gaps

Both harnesses are small, recently created community projects. At the time of the search results, bullshit-bench showed 1 star and 0 forks, and hallucination-hunter showed 0 stars. Neither has published independent validation of its judge or its prompt set. That does not make them unusable — both are readable, MIT-licensed, and runnable offline — but it does mean you should not cite their leaderboard output as an authoritative ranking of local models.

Other limits worth stating in any write-up: results are not portable across prompt sets, so two people running “hallucination benchmarks” on the same model can legitimately get different numbers; LLM-judge scoring is sensitive to the judge prompt and the judge model; and long-context grounding tests depend on whether the runtime actually honoured the configured context length.

There is also no widely agreed canonical leaderboard specifically for local models on hallucination. Established general-knowledge evaluation resources exist in the wider field, but they are not tailored to quantized local deployments, and their scores do not transfer to your hardware.

What to do next

Start with one family, one model tag, one quantization, and a prompt set of at least a few dozen items, then run it three times and record the spread. If you need a defensible number, use the grounded family with labeled data and report precision/recall/F1 from hallucination-hunter. If you want a behavioural profile of how a model handles unanswerable questions, use bullshit-bench and report the four dimensions separately rather than as one average. Publish your environment table alongside the numbers — for local models, that table is half the result.

If you run this protocol, share your model tag, quantization, sampling settings, prompt-set size, and observed spread in the community so others can attempt a like-for-like comparison.

FAQ

Can I benchmark hallucination rates without a paid API key?

Yes. hallucination-hunter's core engine, lexical detector, mock backend, metrics, and CLI depend only on the Python standard library, and its full test suite runs with zero API keys. bullshit-bench's documented quickstart runs against a local Ollama model with no API key. If you use an LLM judge, you can point it at a local model instead of a hosted one.

Is a lower hallucination rate always better?

No. A model that refuses every prompt has a fabrication rate of zero and is useless for real tasks. Always report abstention rate alongside fabrication rate so the trade-off between caution and usefulness is visible.

Why can two people get different hallucination numbers for the same model?

Because the number depends on the prompt set, the quantization level, the sampling settings, the context length, the number of repetitions, and — if an LLM judge is used — the judge model and judge prompt. Without a shared prompt set and a recorded environment, the results are not comparable.

Should I use unanswerable prompts or grounded questions?

Use unanswerable prompts to test whether a model bluffs about things it cannot know. Use grounded questions with a source context to test whether it invents claims despite available evidence. The second is the more relevant test for retrieval-augmented workflows.

References

  1. https://github.com/beausome/bullshit-bench
  2. https://github.com/ramenprotokol/hallucination-hunter

Related terms

Groundedness
The degree to which claims in a model's answer are supported by a source you trust, rather than generated from the model's own parameters.
Abstention rate
The share of prompts on which the model declines to answer or states that the premise is unknowable. Reported alongside fabrication rate to prevent refusal-everything models from looking perfect.
LLM judge
A second language model used to grade another model's output. Convenient and cheap, but sensitive to the judge prompt and the judge model version.
Quantization
Compressing model weights to a lower bit width so the model fits in less memory. It changes model behaviour, so results from one quant level do not transfer to another.

Leave a Reply

Your email address will not be published. Required fields are marked *