Building a retrieval-augmented generation (RAG) workflow for internal documentation means connecting a document pipeline, a retrieval layer, and a grounded answer prompt so that every generated claim can be traced back to the passage that supports it. The practical benefit is verification: a colleague can open the cited passage and check the answer instead of trusting it.
This guide is for a technically curious beginner or intermediate practitioner who needs an assistant over handbooks, runbooks, API references, or policy pages. It follows the four-part pattern used by the LlamaIndex CitationQueryEngine — index the data, retrieve the relevant nodes, add citations to those nodes, then synthesize the final response — and expands each part into the operational detail that internal documentation demands.
What a source citation actually is
Two different things get called citations, and conflating them causes most of the confusion in this area.
The first is retrieval metadata: the file path, page, section heading, chunk identifier, version, and effective date attached to each retrieved passage. This metadata exists whether or not the model ever mentions it.
The second is an inline citation: a marker such as [1] or [2] that appears inside the generated sentence and maps back to a specific retrieved node. The LlamaIndex CitationQueryEngine is built for the second case — it produces in-line citations in the response generated from the retrieved nodes.
Neither kind proves that an answer is correct. A citation is a pointer, not a guarantee, which is why a validation step belongs in the workflow rather than in a later cleanup sprint.
The workflow at a glance
A workable internal-documentation RAG flow has eight stages, in this order:
- Define the corpus and name its owner.
- Ingest source files and extract text without losing structure.
- Normalize the extracted content.
- Chunk with structure awareness and attach metadata to every chunk.
- Embed and index, keeping a keyword or sparse index alongside the vector index.
- Retrieve a broad candidate set, filter by metadata, rerank, and apply thresholds.
- Assemble the final context and generate a grounded answer with inline citations.
- Validate the citations and render a source viewer before the answer reaches the reader.
Documented architectures for high-accuracy document assistants describe essentially this sequence, running from scope definition and ingestion through citation validation and a source viewer.
Prerequisites
- A named document corpus with a human owner who can arbitrate conflicts.
- A working extraction path for every file type in that corpus — Markdown, PDF, DOCX, HTML, wiki exports, or ticket dumps.
- An embedding model, run locally or through an API.
- A vector store that supports metadata filtering, plus a keyword or sparse index.
- An LLM with a context window large enough for the assembled passages plus the instructions.
- A decision about access control: which people are allowed to see which documents.
The configuration described below is illustrative. The available sources document the architecture and the citation engine but do not benchmark a specific stack, so treat parameters as starting points to verify against your own corpus.
Step 1 — Define scope before indexing anything
Scope determines what the assistant is allowed to answer. Decide which systems of record are in bounds, which are explicitly out, and what the assistant should say when a question falls outside them. A narrow, well-owned corpus produces better answers than a broad crawl of every file share, because retrieval quality degrades when near-duplicate drafts and obsolete versions compete for the same query.
Write down the corpus, the owner, and the refresh cadence. This is also the point to settle access control, since retrofitting permissions after indexing is far more work than filtering at retrieval time.
Step 2 — Extract text without destroying structure
Extraction is where most internal deployments quietly lose accuracy. A PDF converted to a flat stream of text loses the heading hierarchy, the table rows, and the list nesting that gave the content meaning. Preserve headings as explicit markers, keep tables as structured rows rather than collapsed lines, and retain the source path for every extracted block.
After extraction, normalize: strip repeated headers and footers, collapse whitespace that came from page breaks, and record the document’s version or effective date if one exists. Normalization is not cosmetic — it prevents the same sentence from appearing as three slightly different chunks.
Step 3 — Chunk with structure and metadata
Chunk along document structure rather than at a fixed character count. A section that answers a question is usually a coherent block of prose, a procedure, or a table, and splitting mid-procedure produces fragments that retrieve well but answer badly. Where chunks must be split, keep a small overlap so that a sentence spanning the boundary survives in at least one chunk.
Attach metadata to every chunk: source path, section heading, chunk ID, document version, effective date, and an access tag. That metadata does three jobs — it becomes the citation payload, it powers metadata filters at retrieval time, and it lets you expire stale content without reindexing everything.
[1] source: handbook/expenses.md#approval-limits (v2026-03, effective 2026-03-01)
[2] source: runbooks/oncall.md#escalation (v2026-01)The block above is an illustrative context format, not output from a specific tool. The point is that each passage carries a stable identifier the citation step can reference.
Step 4 — Retrieve broadly, then narrow
Dense semantic retrieval alone tends to miss exact identifiers — error codes, flag names, ticket numbers, product SKUs — that internal documentation is full of. Pair it with keyword or sparse retrieval, then combine the two candidate sets. Documented guidance for high-accuracy assistants recommends exactly this pairing, followed by query understanding, metadata filtering, a deliberately broad candidate set, reranking, and carefully chosen thresholds.
The order matters. Retrieve widely first, then filter and rerank, because a reranker can only reorder what retrieval returned. After reranking, remove redundant and conflicting passages so the model is not asked to reconcile two versions of the same policy. If two chunks genuinely conflict, that is a signal to surface both with their effective dates rather than to let the model pick one silently.
Step 5 — Assemble context and prompt for grounded answers
Assemble the final context as numbered passages with their identifiers attached, and instruct the model to answer only from those passages. The prompt should also define the refusal behavior: when the context does not support an answer, the assistant says so instead of improvising. A grounded-answer prompt is a documented stage in its own right, not an afterthought bolted onto the retrieval layer.
Step 6 — Generate inline citations
This is the stage the CitationQueryEngine addresses directly. Its workflow is deliberately small: index the data, retrieve relevant nodes for a query, add citations to the retrieved nodes, and synthesize the final response. The engine then returns a response containing in-line citations that correspond to the nodes used.
workflow:
1. index = build_index(documents)
2. nodes = retrieve(index, query)
3. nodes_with_citations = add_citation_markers(nodes)
4. answer = synthesize(query, nodes_with_citations)The pseudocode mirrors the documented steps rather than reproducing a specific API. If you build the citation step yourself, the requirement is the same: the marker in the text must resolve to a passage that was actually in the context window.
Step 7 — Validate citations before returning the answer
Generation can produce a citation marker that points at the wrong passage, or an answer whose claims are not supported by any passage at all. Validate before the answer leaves the system: check that every marker resolves to an identifier present in the context, that the cited passage plausibly supports the sentence it is attached to, and that unsupported claims trigger a refusal or a caveat. Citation validation is a named stage in documented assistant architectures, and it is the step that separates a demo from something a support team can rely on.
Step 8 — Build a source viewer
A citation that cannot be opened is decoration. The source viewer should let a reader click through to the exact chunk, see the surrounding section, and check the document version and effective date. This is also the cheapest feedback mechanism you will get: when readers repeatedly ignore or dispute a citation, that is a retrieval problem you can measure.
Verification: test answerable and unanswerable questions
Build two evaluation sets. The first contains questions the corpus definitely answers, each paired with the passage that should be retrieved. The second contains questions the corpus does not answer, where the correct behavior is a refusal rather than a confident guess. Run both after every change to chunking, embeddings, or the prompt, and keep the results as a regression baseline. Documented practice for these systems includes testing answerable and unanswerable questions, building an evaluation harness, and running controlled experiments rather than tuning by impression.
Failure modes and fixes
| Symptom | Likely cause | Check first |
|---|---|---|
| Answers cite a passage that does not contain the claim | Citation markers generated independently of the context ordering | Whether markers resolve to context identifiers, and whether validation runs before return |
| Exact error codes or flag names never retrieved | Dense-only retrieval | Whether a keyword or sparse index is queried alongside the vector index |
| Two contradictory answers to the same question | Obsolete and current versions both indexed | Version and effective-date metadata, and whether stale chunks are filtered |
| Confident answer to an out-of-scope question | No refusal behavior defined in the prompt | The unanswerable half of the evaluation set |
| Table content returned as garbled text | Extraction collapsed table structure | The extraction step for PDF and DOCX sources |
Limitations and tradeoffs
Several constraints shape what this workflow can promise. Access control has to be enforced at retrieval, not at the interface, or a filtered document can still leak into context. Documents with versions and effective dates need explicit handling, or the assistant will confidently quote a superseded policy. Scanned documents and images require OCR and often degrade citation precision, because the text layer is approximate. Conversation memory improves follow-up questions but can carry an earlier, wrong assumption into later turns, so it needs a boundary.
Internal corpora also carry a prompt-injection surface: a document that contains instructions can influence the model. Treat retrieved text as untrusted data, keep instructions separate from context, and do not let retrieved content trigger tool calls without review. Cost and latency scale with the candidate set, the reranker, and the assembled context, so the broad-then-narrow design is a deliberate trade of compute for recall.
Finally, none of this removes the need for human review where accuracy matters. The workflow makes verification cheap for the reader; it does not make verification unnecessary.
Where to start
Pick a pilot corpus of a few dozen documents with a single owner, build the answerable and unanswerable question sets before you tune anything, and instrument retrieval so you can see which chunks were used for each answer. Then add the citation validation step and the source viewer. If you are running models locally, the same stages apply, but verify memory headroom for the embedding model, the reranker, and the generation model together rather than sizing for one in isolation.
If you are working through this on your own stack, share your corpus type, your retrieval configuration, and the exact failure you hit in the Ai-Forum community — a reproducible symptom is far more useful to the next reader than a general impression.
FAQ
Do source citations guarantee the answer is correct?
No. A citation is a pointer to a retrieved passage, not proof that the passage supports the claim. That is why documented architectures include a citation validation step before the answer is returned.
Should I use dense vector search or keyword search for internal docs?
Use both. Dense semantic search handles paraphrased questions, while keyword or sparse search catches exact identifiers such as error codes, flag names, and ticket numbers that internal documentation is full of.
How do I stop the assistant quoting an outdated policy?
Store version and effective-date metadata on every chunk, filter or flag stale versions at retrieval time, and surface both passages with their dates when two versions genuinely conflict.
Can this workflow run with local models?
The pipeline stages are the same whether the embedding model, reranker, and generator run locally or through an API. The available sources do not benchmark a local configuration, so memory and latency need to be verified on your own hardware.