Insight · September 8, 2026
Answers grounded
in your facts.
A model on its own answers from memory and sometimes invents. RAG gives it a search step, so it answers from your documents instead of guessing.
01 · The shift
A model is a closed book until you give it a way to read.
Retrieval augmented generation has a clean origin. Patrick Lewis and his co authors named it in 2020, in a paper called Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. The idea was simple, and it has held up. Pair a language model with a search step, so it can pull facts from a set of documents instead of answering only from what it memorized in training.
That pairing solves a real problem. A model on its own is a closed book. It knows what was in its training data, it cannot see your files, and when it reaches the edge of what it knows, it often writes a clean and confident answer that happens to be wrong. RAG gives the model something to read first, and grounds the answer in what it finds.
02 · The problem
Memory alone is stale, private to no one, and unsourced.
A model trained last year has a fixed picture of the world from last year. Ask it about a price that changed in March, a policy you rewrote in June, a customer who signed last week, and it cannot know. The facts you care about most are usually the freshest ones, and they live in your systems, not in the training data.
There is a second gap. Even when a plain model answers correctly, it cannot tell you where the answer came from. There is no document to point at, no clause to cite, no way to check the work. For anything that has to pass an audit or earn trust, an answer with no source is not much better than a guess.
The one line to keep
“Fine tuning teaches the model. Retrieval just shows it the page.”
03 · What RAG actually is
Four steps · retrieve, augment, generate, cite.
Strip away the tooling and every RAG system runs the same shape. A search step in front of the model, a set of documents behind it, and one forward pass that turns a question into a grounded answer.
01
Index
Your documents are split into passages and turned into vectors, then stored so they can be searched by meaning rather than by exact words. This happens once, ahead of time, and refreshes when the documents change.
02
Retrieve
The question is turned into the same kind of vector, and the store returns the passages that sit closest to it in meaning. This is the search step that a plain model never has.
03
Augment
Those passages are placed into the prompt alongside the question, so the model reads the relevant facts before it writes a word. The model no longer answers from memory alone.
04
Generate
The model writes the answer from the passages in front of it, and can point back to the exact source each fact came from. The answer is grounded in your documents, not the training data.
04 · Not a bigger brain
Two ways to give a model your knowledge.
There are two ways to make a model know what you know. You can fine tune it, which retrains the weights on your data so the knowledge is baked in. Or you can retrieve, which leaves the model as it is and hands it the right documents at question time. RAG is the second way.
The difference is where the knowledge lives. Fine tuning writes it into the model, so changing a fact means training again, and the model still cannot cite what it drew on. Retrieval keeps the knowledge outside the model, in documents you can edit, replace, or delete any time, and every answer can name the passage it stood on. Fine tuning is the right tool for a fixed style or format. For facts that move, retrieval is almost always the better trade.
This is also why RAG works the same way whichever model sits at the center. Swap the model for a newer one and the documents stay put. The knowledge was never inside the model to lose.
05 · One question that shows the difference
A question the model cannot answer from memory.
Take a support question. What is our refund window for enterprise annual plans, and does it apply mid contract. A plain model has never seen your refund policy. It answers with a number that sounds right, drawn from the thousands of policies it saw in training, and it may be off by weeks. The answer reads well and cannot be trusted.
RAG changes the shape of the answer. It embeds the question, searches the passages of your own policy handbook, pulls the clause that covers enterprise annual plans, and writes the answer from that clause. If the clause changed yesterday, the answer changes today, because the knowledge lives in the document, not the model. And the answer can point at the exact paragraph it came from.
index split the policy handbook into passages, embed each ask what is the refund window for enterprise annual plans embed turn the question into the same kind of vector search return the passages closest in meaning augment place those passages next to the question answer write from the passages, cite the clause they came from
A retrieval pass · the answer carries its source
06 · When it pays off
Reach for RAG when the facts change and the source matters.
RAG earns its place when the answer depends on data that moves. Policies, prices, product specs, support tickets, internal docs. Anything you would rather edit than retrain. It earns its place again when you need to show provenance, cite a source, or pass an audit, because a retrieved answer can name the document behind it and a memorized one cannot. This is why retrieval became the default first move for most enterprise AI.
Under the hood, RAG is one move inside a larger discipline. The real job is managing what the model sees on any given run, which is the whole subject of context engineering. Retrieval decides which facts make it into the prompt. And when a single pass is not enough for a layered question, an agent can run retrieval in a loop, searching again until the evidence holds, which is what agentic RAG adds on top of the pipeline described here.
07 · Where it goes wrong
The answer is only as good as what you retrieve.
RAG is not free, and it is not magic. It adds a retrieval system you have to build and keep fresh. If the index is stale, the answer is stale. If the documents are split badly, the search returns half a thought and the model fills the gap on its own. Whatever is wrong in the store shows up, worded confidently, in the answer.
The quality of the retrieval is the whole game. A model handed the right passage writes a grounded answer. The same model handed the wrong passage writes a grounded sounding answer that is wrong, and now it comes with a citation that makes it look more trustworthy, not less. Most of the work in a good RAG system is not the model at all. It is the chunking, the embeddings, and the search that decide what the model gets to read.
And if the knowledge never changes and fits inside the prompt, you may not need retrieval at all. RAG is the answer to knowledge that is large, private, or moving. When it is none of those, a plain prompt is simpler and just as right.
Closing
Answers grounded in your facts.
Pick the question your model keeps getting almost right, the one whose answer lives in a document it has never seen. Give it a search step. The first time it answers from your own page and names the source, you will stop asking a closed book to remember what it was never shown.
Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, NeurIPS 2020
Share this perspective
More insights
Adjacent perspectives.
Bttr. Field Brief
The brief Bttr. writes for senior buyers.
Monthly. One signal worth your time on Brand Operating Systems, AI search visibility, and the infrastructure buildout. No filler.