Insight · October 14, 2026
Score the retrieval,
score the answer.
A RAG system fails in two places for two reasons. Evaluating it means measuring each stage on its own, with metrics that tell you which one broke.
01 · In one paragraph
What evaluating a RAG system means.
Evaluating a retrieval augmented generation system means scoring its two stages on their own. Retrieval metrics ask whether the right passages were found and ranked near the top. Generation metrics ask whether the written answer is grounded in those passages and actually responds to the question. One end to end number hides which stage failed.
That split is the whole method. A RAG answer is only as good as the passages behind it, so a wrong answer has two possible causes, and a single score cannot tell them apart. Separate the two and every result points at a fix.
02 · Two stages, two failures
A wrong answer has two possible causes.
A RAG system runs in two moves. First it retrieves: it searches your documents and hands the model a short set of passages. Then it generates: the model reads those passages and writes an answer. Each move can fail, and they fail for opposite reasons.
The retriever can hand over the wrong passages, or bury the right one beneath noise. When that happens the model is writing from the wrong source, and no amount of a better model saves it. The generator can do the reverse: it can be handed the right passage and still write something the passage never said, or answer a question the user did not ask.
This is why one accuracy number is a trap. It tells you the answer was wrong without telling you where. The practice that replaced it is to measure retrieval before the model writes anything, and to measure generation given exactly what was retrieved. Ragas, the widely used open evaluation toolkit now maintained by VibrantLabs, is built around that split, grouping its metrics by the component each one scores.
The one line to keep
“A RAG answer is only as good as the passage behind it. Grade the retrieval and the writing apart, or you are guessing.”
03 · Where the scores come from
Two kinds of judge, counting and reading.
Retrieval has clean arithmetic behind it because you can label which passages were supposed to come back. The LlamaIndex evaluation guide scores a retriever with hit rate, the share of questions where the correct passage appears in the top results, and mean reciprocal rank, which rewards putting that passage high rather than merely somewhere on the list. These are counting metrics. They need a small set of questions paired with the passages that answer them.
Generation is harder to count, so the field uses a second kind of judge: a strong model reading the output against a rubric. Ragas calls these its LLM based metrics, ones that use a model underneath to do the evaluation, as opposed to the non LLM metrics that do not. Faithfulness and answer relevance are both scored this way. The judge is not grading taste. It is checking a specific, mechanical thing: does each claim trace to the retrieved text, does the answer match the question.
You do not need a huge labeled set to start. A few dozen real questions, each paired with the passage that answers it and a short reference answer, is enough to score every metric here and to catch the failures that matter before users do.
04 · One measured example
A retrieval score that moved a real number.
Anthropic published a clean demonstration of retrieval scoring in September 2024. The team measured a pipeline by its retrieval failure rate, defined as one minus recall at 20, the share of relevant passages that did not appear in the top 20 the retriever returned. A lower number means the evidence reached the model more often.
They then changed the retriever and watched the number move. Adding context to each chunk before embedding cut the failure rate by 35 percent, from 5.7 to 3.7. Running keyword search alongside the vectors cut it by 49 percent, to 2.9. Adding a reranking pass, scoring the top 150 chunks with a Cohere reranker and keeping the best 20, cut it by 67 percent, to 1.9. The lesson is not the specific trick. It is that a single retrieval metric turned each change into a measurable decision instead of an argument.
Retrieval failure rate · one minus recall@20 · lower is better Baseline embeddings 5.7% Context added to chunks -35% · 3.7% Context plus keyword search -49% · 2.9% Context, keyword, reranking -67% · 1.9%
Source · Anthropic, Introducing Contextual Retrieval, Sept 2024
05 · The four that matter
Two for retrieval, two for the answer.
Context recall
Did we fetch enough
A retrieval metric. Ragas measures how many of the relevant pieces of information were successfully retrieved, by checking what share of the claims in a reference answer the retrieved context supports. Low recall means the answer was missing its evidence before the model ever wrote a word.
Context precision
Is the top of the list clean
A retrieval metric. Ragas defines it as the retriever’s ability to rank relevant chunks higher than irrelevant ones, scored as the mean precision at each rank. It matters because most pipelines only pass the top few chunks to the model, so junk near the top crowds out the passage that held the answer.
Faithfulness
Is the answer grounded
A generation metric. Ragas splits the answer into individual claims and scores the share the retrieved context supports, so a fully faithful answer invents nothing the passages do not back. This is the direct measure of whether the system hallucinated.
Answer relevance
Does it address the question
A generation metric. Ragas generates a few artificial questions from the answer, then averages the cosine similarity between those and the real question. A grounded answer that quietly drifts to the wrong subject scores high on faithfulness and low here.
Read the four together and they locate the fault. High context recall with low faithfulness means the evidence was there and the model ignored it, a generation problem. Low recall with high faithfulness means the model wrote honestly from passages that never held the answer, a retrieval problem. The pair you fix is the pair that scored low.
06 · Not a one time check
Evaluation is a standing instrument, not a launch gate.
A RAG system that scored well at launch does not stay that way. The model behind it gets updated, your documents grow and go stale, and the questions real users ask drift away from the ones you tested. A pipeline that was faithful in March can quietly start inventing in June, and nothing in the product tells you.
So the teams that trust their systems treat these scores the way they treat evals: a suite that runs on every change and on a sample of live traffic, not a report written once before launch. When a score drops, you see it the same day, and because the metrics are split by stage you already know where to look.
The two levers you reach for most live upstream of the answer. How you split documents sets the ceiling on recall, which is why chunking is a real tuning decision, and a reranking pass is the fastest way to lift precision once recall is in place.
07 · Where the scores mislead
Good numbers can still hide a bad system.
Faithfulness scored on its own is the classic trap. An answer that copies the retrieved passage word for word is perfectly faithful and may still be useless, because it answers the wrong question. That is exactly why answer relevance exists, and why the two are read as a pair rather than one at a time.
The judge is a model, so it carries a model’s limits. An LLM based score is an estimate, steadier than a human reading a thousand answers but not a ground truth. Sample the cases it marks wrong and read them yourself before you trust a threshold, and keep the judge and the system under test on separate models so one is not grading its own voice.
Finally, your own rules sit outside these four. A citation that has to resolve, a format a downstream system parses, a length limit, a refusal the policy requires: those are checks you add on top, not metrics the toolkit ships. The four cover whether the system retrieved and answered well. Whether it answered the way your product needs is yours to define.
Closing
Measure the two halves, and the fix names itself.
Score retrieval before the model writes, score generation against what it was given, and read the four metrics as two pairs. Run them on live traffic, not once at launch. A RAG system you can measure stage by stage is one you can actually improve.
Written by · October 14, 2026
Donny Smith · ECD, Founder, Bttr.
Over the past 15 years, he has led creative teams and contributed to products used by millions of people worldwide, working with companies including Apple, Opendoor, JP Morgan, GE Aerospace, Pepsi, and Alterra Mountain Company.
LinkedInShare this perspective
More insights
Adjacent perspectives.
Bttr. Field Brief
The brief Bttr. writes for senior buyers.
Monthly. One signal worth your time on Brand Operating Systems, AI search visibility, and the infrastructure buildout. No filler.