AIForum.

How to Measure Hallucination Rates in Local Summarization Models

On this page

THE SHORT ANSWER

To measure hallucination rates in local summarization models, score each generated summary against its source document with a source-grounded detector such as Vectara HHEM-2.1, document your threshold, and report hallucination rate, factual consistency rate, answer rate, and sample size. This guide explains the metrics, a reproducible protocol, and the limits of the numbers.
Quick facts

To measure hallucination rates in local summarization models, score each generated summary against its source document with a source-grounded detector such as Vectara HHEM-2.1, then report the percentage of summaries that fall below a threshold you document before testing. The score is not a general truthfulness score. It measures whether the summary is factually consistent with the source that was supplied to the model.

This guide is for practitioners running summarization models on their own hardware or in a private environment. It explains what the metrics mean, how to build a reproducible local evaluation, and where the numbers can mislead you. The protocol is a recommended design, not a report of editor-run benchmark results; the supplied sources do not include local model scores or hardware measurements.

What hallucination rate means in summarization

In document summarization, the source document is the reference. A hallucination is content in the summary that is factually incorrect or unrelated to the source. Vectara’s HHEM leaderboard describes hallucinations as instances where a model introduces factually incorrect or unrelated content in its summaries. The leaderboard evaluates how often an LLM introduces hallucinations when summarizing a document.

Vectara’s HHEM-2.1 model outputs a hallucination score between 0 and 1. A score of 0 indicates complete hallucination, while 1 represents perfect factual consistency. That score is the raw signal. The hallucination rate is a summary statistic: the percentage of source-summary pairs that your evaluation classifies as hallucinated.

The exact cutoff used to convert a continuous HHEM score into a binary hallucination label matters. The supplied leaderboard source lists a Hallucination Rate (%) metric, but the excerpt does not state the threshold used to produce it. Treat the threshold as a configuration choice. Record it, justify it, and keep it fixed when comparing models.

Why local models need their own measurement

A model’s behavior can change with quantization, runtime, context length, prompt template, and sampling settings. A summary generated by a local model on your hardware is not automatically equivalent to a summary generated by the same model in a hosted service. If you want a number that describes your deployment, you need to measure your deployment.

Local evaluation also keeps sensitive source documents inside your environment. Vectara describes HHEM-2.1-Open as the open source version of HHEM-2.1. If you use it locally, verify the model card and license for your use case before deploying it in a production workflow.

The metrics to report

Vectara’s HHEM leaderboard lists four metrics that are useful for local summarization evaluation:

  • Hallucination Rate (%): the share of summaries classified as containing hallucinated content under your threshold.
  • Factual Consistency Rate (%): the share of summaries classified as factually consistent with the source under the same threshold.
  • Answer Rate (%): the share of prompts for which the model produces a usable summary. This matters because a model that refuses or returns empty output can appear to have a low hallucination rate while failing the task.
  • Average Summary Length: a simple check for truncation, runaway verbosity, or format drift.

Report the sample size alongside these metrics. A hallucination rate of 5% from 20 summaries is much less stable than the same rate from 500 summaries. If you calculate confidence intervals, state the method. If you do not, say that the numbers are descriptive.

A reproducible local evaluation protocol

The following protocol is a recommended design. It does not claim that any particular local model will produce a specific score.

1. Define the task and the unit of analysis

Decide whether you are evaluating single-document summarization, multi-document summarization, or a specific content type such as support tickets, meeting notes, or WordPress posts. The unit of analysis is usually one source-summary pair. Keep the task definition stable across models. If you change the prompt, the summary length target, or the document type, you are running a different evaluation.

2. Freeze the generation configuration

Record the model name and version, quantization, runtime and version, context length, prompt template, system prompt, sampling settings, and hardware. Use deterministic or near-deterministic settings where possible, and note when you cannot. These details are part of the result. Without them, another practitioner cannot reproduce your measurement.

3. Build a fixed source-summary dataset

Select source documents that represent your real workload. Store the original source text, the exact prompt, the generated summary, and the model configuration. Do not edit summaries after generation. If the model truncates the source because it exceeds the context window, record that as a failure mode rather than silently trimming the document.

4. Score each summary against its source

Run a source-grounded hallucination detector such as HHEM-2.1 over each source-summary pair. The model returns a score between 0 and 1, where 0 is complete hallucination and 1 is perfect factual consistency. If you need a fully local evaluation loop, use the open source HHEM-2.1-Open version and confirm its license and runtime requirements first.

5. Convert scores to rates with a documented threshold

Choose a threshold before you look at the results. For a threshold t, an illustrative calculation is:

  • Hallucination Rate = (number of summaries with score < t) / total summaries × 100
  • Factual Consistency Rate = (number of summaries with score ≥ t) / total summaries × 100

The supplied sources do not specify the exact cutoff used by the HHEM leaderboard. Verify the current model card or leaderboard code if you want to match its methodology. The important reproducibility step is to publish your own threshold and keep it constant across comparisons.

6. Add human spot checks

Sample summaries with low, middle, and high scores. Read each summary against its source and record whether you agree with the detector. This step catches systematic problems that a single score can hide, such as a detector that misses subtle numerical errors or a prompt that encourages the model to add outside knowledge. Report the agreement rate and the number of spot-checked items.

7. Report the result with context

Publish the metrics, sample size, threshold, model configuration, and dataset description. Separate your measured results from provider claims and from illustrative examples. If you compare two local models, change only one variable at a time where possible. If you must change several variables, say so.

An illustrative calculation

Suppose you evaluate 50 source-summary pairs with a fixed threshold. If 5 summaries score below the threshold, the hallucination rate is 10% and the factual consistency rate is 90%. This is an illustrative arithmetic example only. It is not a benchmark result, and it does not describe any specific local model. The value of the example is the reporting format: sample size, threshold, hallucination rate, factual consistency rate, answer rate, and average summary length.

Common pitfalls that distort local hallucination rates

  • Threshold shopping: changing the cutoff after seeing results until the number looks good. Fix the threshold first.
  • Ignoring answer rate: a model that refuses to summarize can have a low hallucination rate but poor utility.
  • Mixing tasks: extractive summaries, abstractive summaries, and multi-document summaries have different failure modes. Do not average them without separating them.
  • Truncated source documents: if the model never saw the full source, the summary cannot be fully grounded in it. Record context-window failures.
  • Prompt leakage: if the prompt includes facts that are not in the source, the detector may still score the summary as inconsistent with the source even though the model followed the prompt.
  • Judge limitations: HHEM is a model, not ground truth. It can make mistakes. Human spot checks are still necessary for high-stakes content.
  • Small samples: a handful of summaries can produce unstable rates. Report the sample size and avoid overinterpreting small differences.
  • Version drift: changing the runtime, quantization, or model revision can change outputs. Pin versions for comparisons.

Interpreting the numbers

A low hallucination rate does not automatically mean a summary is useful. HHEM measures factual consistency with the source, not completeness, readability, or coverage of the most important points. A model can produce a short, factually consistent summary that omits critical information. Check answer rate and average summary length, and review a sample manually.

Use the same dataset, prompt, threshold, and detector version when comparing models. Provider-published leaderboards are useful context, but the HHEM leaderboard is maintained by Vectara and is not a substitute for a local test on your documents. Vectara’s own announcement describes hallucinations as a broad category that includes large errors, copyrighted output, nuanced false facts, and bias. A single source-grounded score will not capture every one of those risks.

Applying this in a local content workflow

If a local summarization model generates excerpts, meta descriptions, or post summaries for a WordPress workflow, treat the original post as the source document. Generate summaries in a staging environment, score them with a fixed evaluation harness, and require human review before publication. Store the source, prompt, model version, score, and reviewer decision. That record lets you diagnose regressions when you update the model or change the prompt.

Limitations and verification gaps

The supplied sources do not provide local hardware benchmarks, recommended HHEM thresholds, or licensing details for HHEM-2.1-Open. They also do not include a dataset recipe for local summarization evaluation. Before adopting a specific detector or threshold, check the current model card, leaderboard code, and license. If you need to compare against Vectara’s published numbers, reproduce its dataset and configuration rather than assuming your local setup is equivalent.

Next steps

Start with a small, fixed evaluation set that reflects your real documents. Pin your local model and runtime versions. Score each summary with a source-grounded detector, document your threshold, and report hallucination rate, factual consistency rate, answer rate, average summary length, and sample size. Add human spot checks before you trust the number for publication or automation. If you run a local summarization evaluation, share your task definition, model configuration, threshold, and observed failure modes in the community so others can reproduce and compare.

FAQ

Is a low hallucination rate enough to trust a local summarization model?

No. A low hallucination rate only means the summaries are factually consistent with the source under your threshold. Also check answer rate, summary length, coverage of important points, and human spot checks.

What threshold should I use for HHEM scores?

The supplied sources do not state the exact cutoff used by the HHEM leaderboard. Choose a threshold before testing, document it, and verify the current model card or leaderboard code for the recommended cutoff.

Can I run HHEM-2.1 locally?

HHEM-2.1-Open is described as the open source version of HHEM-2.1. Check the model card and license for your use case before deploying it locally.

How many summaries should I evaluate?

There is no single number in the supplied sources. Report your sample size and uncertainty; start with a fixed set that covers your document types and expand until results stabilize.

Does HHEM measure completeness?

No. It measures whether summary content is factually consistent with the source, not whether the summary includes every important point.

References

  1. https://www.vectara.com/blog/measuring-hallucinations-in-rag-systems
  2. https://huggingface.co/spaces/vectara/leaderboard/raw/main/src/display/about.py
  3. https://huggingface.co/vectara/hallucination_evaluation_model

Related terms

Hallucination rate
The percentage of generated summaries classified as containing hallucinated content under a documented threshold.
Factual consistency
The degree to which a summary is supported by its source document; HHEM scores 1 as perfect factual consistency.
Answer rate
The percentage of prompts for which the model produces a usable summary rather than refusing or returning empty output.
HHEM
Hughes Hallucination Evaluation Model, Vectara's model for detecting hallucinations in document summaries.
Grounded summarization
Summarization constrained to facts present in the supplied source document.

Leave a Reply

Your email address will not be published. Required fields are marked *