How a RAG assistant answers a question

A language model can produce a fluent answer without having the right information. Retrieval-augmented generation, usually shortened to RAG, adds a search step before the answer is written. The model receives selected material alongside the question, so it can use that material as context.

Start with a concrete question

Imagine an assistant for an internal company handbook. Someone asks: “How many days of parental leave does our policy provide?” The answer needs the current handbook, not a plausible policy from a different company.

Three steps in the workflow

• Retrieve: search the approved documents for passages related to parental leave. Keep the document title, version and location with each passage.

• Provide context: give the model the question and the selected passages. Ask it to use the supplied material, cite its sources and say when the information is missing.

• Generate: produce a readable answer that refers back to the relevant passages. Let the reader open the source and check it.

What RAG changes — and what it does not

In a typical RAG workflow, the retrieved passages are supplied at answer time. This is different from retraining the model on every document update. Keeping the source collection current can therefore improve access to new information without a new training run.

Retrieval is not a guarantee of correctness. Search can return the wrong passage, overlook an exception or surface an outdated version. The model can still misread the context or write an unsupported claim.

A useful first experiment

• Choose five short, non-sensitive documents on one topic.

• Write ten realistic questions, including two that the documents cannot answer.

• For each question, inspect the retrieved passages before judging the final wording.

• Check whether the answer is supported, whether the cited source opens and whether missing information is acknowledged.

Keep the first version small

A narrow, testable assistant is easier to improve than a system that claims to answer everything. Begin with a clear document collection and a small set of questions. Measure the mistakes, improve the search and keep the reader able to verify the evidence.

Further reading: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., 2020)