Citations in a retrieval-augmented generation (RAG) chatbot are not a formatting detail added at the end. They are a data contract that has to survive three stages: retrieval, generation, and rendering. Each retrieved chunk must carry an identifier into the model’s context, the model must be instructed to cite that identifier, and the interface must turn the identifier back into a link the user can follow to the original passage.
This guide covers that pipeline for someone building or debugging a RAG assistant. It focuses on the mechanics that make citations verifiable rather than decorative, and on the failure modes that make citations actively misleading.
What citations actually accomplish
According to RedHop’s guide to RAG citations, three reasons come up most often, roughly in order of frequency. The first is trust and verifiability: a RAG answer looks authoritative because it is grounded in documents, but without citations the user cannot tell which sentences came from those documents and which the model paraphrased or invented. The second is debugging: when the answer is wrong, the first question is whether the retriever found the right chunk, and citations make that visible in the interface instead of buried in logs. The third is compliance. In legal, medical, financial, or enterprise deployments, “where did this answer come from” is often treated as non-negotiable rather than a nice-to-have.
The Agentset documentation frames the same idea more narrowly: citations link each statement in the generated response back to the specific chunk it came from, so users can verify information and explore the original documents.
Carry stable chunk IDs, not positions
The most common design mistake is numbering retrieved chunks by their position in the assembled context. That works until the context is assembled from more than one search, or until a reranker reorders results. Agentset’s guidance is explicit on this point: include each chunk’s id in the context you pass to the model, because IDs stay stable no matter how many searches produced the context, which makes them more reliable than position-based numbering.
In practice, the context you hand to the model looks like a sequence of small JSON objects, each pairing an identifier with the chunk text:
import { Agentset } from "agentset";
const agentset = new Agentset({
apiKey: process.env.AGENTSET_API_KEY,
});
const ns = agentset.namespace("YOUR_NAMESPACE_ID");
const results = await ns.search("What is multi-head attention?");
// Keep the id with each chunk so the model can cite it
const context = results
.map((r) => JSON.stringify({ id: r.id, text: r.text }))
.join("nn");The equivalent Python path follows the same shape:
import json
import os
from agentset import Agentset
client = Agentset(
namespace_id="YOUR_NAMESPACE_ID",
token=os.environ["AGENTSET_API_KEY"],
)
results = client.search.execute(query="What is multi-head attention?")
# Keep the id with each chunk so the model can cite it
context = "nn".join(
[json.dumps({"id": r.id, "text": r.text}) for r in results.data]
)The important property is not the JSON wrapper. It is that the identifier the model sees is the same identifier your application can resolve later, after generation, when you need to attach a link, a page number, or a preview.
Instruct the model to cite by chunk ID
Retrieval alone does not produce citations. The system prompt has to ask for them, and it should ask in terms of the identifier you supplied. The Agentset example prompt begins by constraining the answer to the supplied context and then requiring the model to cite chunk IDs:
You are an AI assistant. Answer questions based ONLY on the provided context.
Guidelines:
1. If the context does not contain information to answer the query, state clearly:
"I cannot answer this question based on the available information."
2. Only use information directly supported by the context.
3. Cite the chunk id for each claim.There is an important exception worth knowing before you copy this pattern into a hosted product. Agentset notes that in its hosted chat and playground, citations are automatic, and the platform always applies its own citation rules on top of your system prompt. In that setting you should customize the prompt freely but not add citation-format instructions, because they will conflict with rules the platform is already enforcing.
The general lesson generalizes beyond one vendor: find out whether your stack owns citation formatting or expects you to own it, and only one side should be writing the format rules.
Rendering: the inline footnote pattern
RedHop describes two UI patterns used by consumer AI products and details the one associated with Perplexity. In that pattern, the answer text contains numbered markers interleaved with the prose, and the sources appear in a panel or list at the bottom. Hovering a marker shows a source preview. RedHop’s illustrative example of the output reads:
The governing law of this contract is Delaware [1], with arbitration in San Francisco [2]. EU customers have an override that applies…
Two properties make this pattern work. First, the markers are placed at the level of the claim, not the paragraph, so a reader can check a specific sentence. Second, the marker resolves to a preview of the actual chunk, which means a wrong citation is visible immediately rather than after a click into a PDF.
If you are building the interface yourself, the minimum viable version is a marker that maps to a stored record containing the chunk text, the document title, and a deep link or page reference. The marker numbering can be presentation-only; the underlying mapping should use the stable chunk ID.
Failure modes to design against
RedHop names the two failure modes that matter most: the model hallucinates a citation that does not exist, or it points at the wrong chunk. Both are cheap to detect if you validate citations against the set of IDs you actually retrieved.
- Unknown ID. The model emits a citation for an identifier that was never in the context. Reject it, strip it, or regenerate. Never render a marker you cannot resolve.
- Resolvable but wrong. The ID exists, but the cited chunk does not support the sentence. This is harder to catch automatically. Hover previews and a retrieval log are the practical defenses, because a human reviewer can see the mismatch in seconds.
- Position drift. Numbered references break when results are reordered or when multiple searches are merged. Stable IDs avoid this class of bug entirely.
- Uncited claims. The model answers from general knowledge and attaches no marker. A prompt that requires a citation per claim, plus a check for uncited sentences, surfaces this before the user does.
One operational note: log the retrieved chunk IDs alongside the generated answer. When a user reports a bad citation, that log is what tells you whether the retriever or the generator is at fault.
Tradeoffs and limitations
Citations are not free. Every chunk ID and every instruction about citation format consumes context tokens, and the extra generation step of attaching markers can add latency. More importantly, citation granularity is bounded by your chunking strategy. A citation points at a chunk, not at a sentence, so a 1,000-token chunk gives the reader a large area to search. Smaller, overlapping chunks improve citation precision but increase the number of retrieved items and the chance of near-duplicate citations.
There is also a trust caveat that applies to every implementation: a citation is evidence that a passage was retrieved and cited, not proof that the passage supports the claim. Verification still requires the reader, or an automated check, to compare the sentence against the chunk.
A verification checklist
- Confirm every retrieved chunk carries a stable ID into the context.
- Confirm the system prompt requires a citation per claim and defines the citation format, unless the platform owns formatting.
- Validate every emitted citation against the retrieved ID set and handle unknown IDs explicitly.
- Render markers as resolvable links with a chunk preview, not as plain numbers.
- Test with a question whose answer is absent from the corpus and confirm the assistant declines instead of citing something.
- Log retrieved IDs with each answer so wrong-chunk reports are diagnosable.
Where to start
Build a small evaluation set first: ten to twenty questions where you know which chunk contains the answer. Run them through the pipeline and check two numbers, citation validity (does every marker resolve?) and citation accuracy (does the cited chunk actually support the sentence?). That pair of checks will tell you more about your citation implementation than any amount of prompt tuning, and it gives you a baseline to compare against when you change chunking or switch models.
If you are working through this on your own stack, the community is a good place to compare notes on chunk sizes, citation formats, and the failure cases that only show up in production. Share your setup, the model you are running, and the exact behaviour you are seeing.
FAQ
Should I number sources by their position in the context?
No. Agentset's documentation recommends including each chunk's id instead, because IDs stay stable no matter how many searches produced the context, which makes them more reliable than position-based numbering.
Why does my model cite a source that does not exist?
This is one of the two failure modes RedHop identifies: the LLM hallucinates a citation that does not exist, or it points at the wrong chunk. Validate every emitted citation against the set of chunk IDs you actually retrieved and handle unknown IDs explicitly.
Do I need to write citation-format instructions in my system prompt?
It depends on the platform. Agentset notes that in its hosted chat and playground citations are automatic and its citation rules are always applied on top of your system prompt, so you should not add citation-format instructions there. When you own the pipeline, you do need to instruct the model to cite by chunk ID.
Does a citation prove the answer is correct?
No. A citation shows that a passage was retrieved and referenced. Verifying that the passage actually supports the sentence still requires a reader or an automated comparison between the claim and the chunk text.