How to Run Evals on a RAG Pipeline: Test Set, Metrics and CI
The short answer
RAG evaluation is a loop, not a one-time audit. You collect real questions, write down what a good answer looks like and which passages contain it, run the pipeline, and score two things: did retrieval find the right passages, and did the model answer correctly from them. Then you automate the run so it repeats whenever anything changes.
The split matters most. A RAG system has two components that fail in different ways. If you only score the final answer, a bad result tells you something is wrong but not where. Scored separately, the numbers point at the fix: low retrieval recall means you work on chunking, search or reranking; good retrieval with bad answers means you work on the prompt, the context or the model. If you need a refresher on the components themselves, see our RAG architecture guide.
How to run RAG evals, step by step
- Write down what a good answer isBefore any metric, agree on the rules: must answers cite sources, what should the system say when the documents don't cover the question, which errors are unacceptable (a wrong price, a wrong dosage, another customer's data). These rules become your scoring criteria.
- Collect 50 to 100 real questionsTake them from search logs, support tickets, sales calls and pilot users. Keep the original wording, typos and vague phrasing included.
- Label expected answers and source passagesFor each question a domain expert records the correct answer and the document passages that contain it. The passages are what make retrieval measurable.
- Log every stage of a runFor each question store the retrieved chunks with their ranks and scores, the final prompt, the answer, latency and token cost. You can't score retrieval separately if you only keep the answer.
- Score retrievalCompare the retrieved chunks with the labeled passages: recall@k, precision@k and a rank metric. No language model is needed for this part, so it is fast, cheap and repeatable.
- Score answersCheck faithfulness to the retrieved context, relevance to the question and correctness against the expected answer, using an LLM judge that you have calibrated against human labels.
- Read the failures, not just the averagesSort the failed questions into categories: missing document, wrong chunk, ignored context, outdated source. The biggest category is your next piece of work.
- Automate it and set a gateRun the set in CI, compare with the last accepted baseline and block releases that score lower.
- Feed production back inAfter launch, sample real conversations every week and add each new failure to the test set.
Build the test set from real questions
The test set decides what your scores mean. A set of tidy questions written by the engineers who built the pipeline will give high scores and tell you little.
Size
Start with 50 to 100 questions. That finds the large failures and is small enough to label carefully. Plan to reach 200 to 500 before you treat score changes as release decisions, because small sets are statistically noisy. At 100 questions and an 80% pass rate, the 95% margin of error is about 8 percentage points in each direction. A change from 80% to 84% is inside the noise. At 400 questions the margin shrinks to about 4 points.
Sources and mix
Real user wording beats invented wording. Cover the mix your product will actually face:
- Simple lookups answered by one passage.
- Multi-part questions that need two or more passages, often from different documents.
- Questions with exact terms: product codes, error numbers, names. Vector search alone often misses these.
- Unanswerable questions where your documents have no answer. The expected behavior is to say so. Without these, you never measure the failure that hurts trust most: a confident invented answer.
- Permission cases, if access differs by user: the same question asked by two roles should retrieve different documents.
A model can draft questions from your documents to get you started. According to the Ragas documentation, its test set generator builds a knowledge graph from your documents and produces single-hop and multi-hop queries from it. Treat generated questions as drafts for an expert to review. They tend to reuse the wording of the source text, which makes retrieval look better than it will be on real traffic.
Who writes the expected answers
Someone who would be trusted to answer the question without the system: a support lead, a compliance officer, a senior engineer. Engineers can prepare the candidates (pull the passages, draft an answer), but the expert approves. For each item, store the question, the expected answer, the IDs of the passages that support it, and a note on anything the answer must or must not include. Version the set in the repository next to the code, and re-check labels when the underlying documents change.
Evaluate retrieval and generation separately
Retrieval sets the ceiling. If the passage with the answer never reaches the prompt, no model or prompt can produce a grounded correct answer. So measure retrieval first, on its own, and fix it before you spend time on prompts.
Retrieval metrics
| Metric | What it asks | How to read it |
|---|---|---|
| Recall@k | Of the passages that should have been found, what share is in the top k results? | The most important retrieval number. Low recall means the answer was not available to the model. |
| Precision@k | Of the k results returned, what share is relevant? | Low precision means noise in the prompt: higher cost, and more chances for the model to pick up the wrong fact. |
| MRR | How high does the first relevant result rank? (Average of 1 divided by its rank.) | Rank 1 scores 1.0, rank 2 scores 0.5, rank 5 scores 0.2. Good for tuning a reranker. |
| nDCG@k | Are the most relevant results at the top of the list, compared with an ideal ordering? | Use it when relevance is graded (highly, partly, not relevant) rather than yes or no. |
Set k to the number of chunks you actually pass to the model. Recall@20 is of little comfort if only the top 5 go into the prompt. It also helps to measure recall at two points, before and after reranking, so you can tell whether the first-stage search or the reranker lost the passage.
Answer metrics
| Metric | What it asks | Needs an expected answer? |
|---|---|---|
| Faithfulness (groundedness) | Is every claim in the answer supported by the retrieved passages? Usually computed by splitting the answer into claims and checking each one against the context. | No |
| Answer relevance | Does the answer address the question that was asked, without drifting or padding? | No |
| Correctness | Does the answer agree with the expected answer on the facts that matter? | Yes |
| Refusal accuracy | For unanswerable questions, did the system decline instead of inventing an answer? For answerable ones, did it avoid declining? | Yes (a label that the question is unanswerable) |
Faithfulness and correctness are not the same thing, and the gap between them is diagnostic. A faithful but wrong answer means the retrieved document was wrong or outdated: a data problem. A correct but unfaithful answer means the model used its training data instead of your sources: it happened to be right this time, and it will not be when your facts differ from the public ones. Our guide to RAG hallucinations covers how to reduce both kinds.
LLM-as-judge and its limits
Answer metrics need something that reads text and makes a judgment. People do this best and slowest. An LLM judge is a second model prompted to grade the output: given the question, the retrieved passages and the answer, is each claim supported? It makes it practical to score hundreds of answers on every change.
The limits are real, and you should plan around them:
- A judge is a model, with model errors. It can pass a fluent, well-structured answer that is subtly wrong, and it lacks your domain knowledge unless you give it the expected answer.
- Scores drift. The same answer can get different grades across runs, and a new judge model version can shift every score at once. Old and new numbers stop being comparable.
- Wide scales are unreliable. "Rate accuracy from 1 to 10" produces numbers nobody can interpret. A narrow question with a pass or fail answer ("Is the refund period stated in the answer the same as in the expected answer?") is far more repeatable.
- It costs money and time. Each judged answer is one or more extra model calls. Claim-by-claim faithfulness checks multiply that.
Have domain experts label 50 to 100 answers as pass or fail. Run the judge on the same answers and count the disagreements. Read each one and adjust the judge prompt until agreement is high enough for the decision you are making. Pin the judge model version, set its temperature to zero where the provider allows it, and repeat the comparison whenever the judge prompt or model changes.
Keep a human review sample
Calibration is not a one-time event. Each week, pull a sample of scored answers, for example 20 to 50, and have an expert grade them without seeing the judge's verdict. Weight the sample toward what matters: answers the judge failed, answers users rated down, high-risk topics, and a random slice of passes to catch what the judge lets through. Track judge-versus-human agreement as a metric of its own. When it drops, fix the judge before you believe any other number.
Regression checks in CI
A RAG pipeline changes in more ways than ordinary code: prompts, chunk size, embedding model, reranker, the answering model, and the documents themselves. Any of these can fix ten questions and break five others. A regression run makes that visible before release.
- Two tiers. Retrieval metrics need no model calls beyond embedding the questions, so run them on every pull request. Run the full suite with judged answer metrics before each release and on a nightly schedule.
- Compare with a baseline, not an absolute target. Store the scores of the last accepted version. Fail the build when a metric drops by more than the noise you measured for your set size.
- Diff per question. An unchanged average can hide five new failures and five new passes. List the questions that flipped and read them.
- Hard-fail a critical subset. Mark the questions where a wrong answer is unacceptable (pricing, legal, safety, permissions). Any failure there blocks the release regardless of the average.
- Pin what you can. Model versions, judge prompt, test set version and index snapshot. Otherwise you can't tell whether your change or someone else's moved the score.
- Track cost and latency in the same run. A change that adds two points of recall and doubles response time is a trade-off someone should approve.
Evaluation libraries
Retrieval metrics take a few dozen lines to write yourself. For answer metrics, open-source libraries give you judge prompts to start from; check them against your human labels like any other judge. Three common ones, as their own documentation describes them:
| Library | What its documentation offers for RAG |
|---|---|
| Ragas | RAG metrics including Context Precision, Context Recall, Faithfulness, Response Relevancy and Noise Sensitivity. Faithfulness is the share of claims in the response supported by the retrieved context, scored 0 to 1. Context Recall has an LLM-based variant that uses a reference answer and non-LLM variants that compare against reference contexts or their IDs. Includes a test set generator. |
| DeepEval | Retriever metrics (Contextual Relevancy, Contextual Precision, Contextual Recall) and generator metrics (Answer Relevancy, Faithfulness), plus G-Eval for custom criteria. Metrics return a score from 0 to 1 with a reason and pass against a threshold. It plugs into Pytest, and evals run with the deepeval test run command. |
| TruLens | Open-source, OpenTelemetry-native evals and tracing. Its documentation frames RAG quality as the "RAG triad": context relevance, groundedness and answer relevance. Includes leaderboard and compare views for app versions. |
Online signals after launch
An offline test set measures the questions you thought of. Production shows you the ones you didn't. Log every request with its retrieved chunks and answer, and watch:
- Explicit feedback: thumbs up and down with an optional comment. Few users leave it, so treat it as a source of examples, not a rate you can trust.
- Rephrased questions: a user asking the same thing again in different words usually means the first answer failed.
- Escalations: handoffs to a human agent or a support ticket opened right after an answer.
- No-answer rate: how often the system declines. A sudden rise points to a broken index or ingestion job; a rate near zero on a narrow knowledge base suggests it is answering things it shouldn't.
- Reference-free judge scores: faithfulness and answer relevance on a sample of live traffic, tracked as a trend.
- Latency and cost per answer, split into retrieval and generation.
Every confirmed production failure goes into the test set with an expected answer and source passages. That is how a set of 80 questions becomes 400 that reflect real use.
Built by Gilzor
Results we’ve shipped
Talk to the people who build it. Tell us about your project and get a free estimate of scope, timeline and cost.
Symptom, likely cause, fix
When scores are bad, the pattern across the two scorecards usually names the cause:
| Symptom | Likely cause | Fix to try first |
|---|---|---|
| "I don't know" although the document exists; low recall@k | The passage is not retrieved: chunks split the answer, or the query wording differs from the document's | Check the document is indexed; adjust chunking; add keyword search next to vector search (hybrid); rewrite queries |
| Recall is good at k=20, answers are still wrong | The right passage ranks low and is cut off or buried in noise | Add or tune a reranker; pass fewer, better chunks; track MRR or nDCG |
| Right passage in the prompt, wrong answer; low faithfulness | The model ignores or misreads context, or the prompt allows outside knowledge | Tighten the prompt to answer only from sources; reduce context noise; try a stronger model |
| Confident answer to a question the documents don't cover | No refusal path; weakly related chunks are treated as evidence | Instruct the model to decline without support; add a relevance threshold; add unanswerable questions to the test set |
| Faithful to the source but factually outdated | Stale or duplicate document versions in the index | Fix ingestion freshness; remove superseded versions; filter by date or version |
| Multi-part questions are half answered | One search can't cover two topics | Split the question into sub-queries and merge results |
| Numbers from tables or PDFs come out wrong | Parsing flattened the table or lost the layout | Use a parser that keeps table structure; chunk tables whole |
| Offline scores are high, users complain | The test set doesn't match real questions, or the judge is too lenient | Rebuild the set from production logs; recalibrate the judge against human labels |
| Scores dropped and nobody changed the code | New documents in the index, or the provider updated a model | Pin model versions; run evals nightly; evaluate after each large ingestion |
FAQ
How do I run AI evals on my RAG pipeline?
How many questions does a RAG test set need?
What is the difference between faithfulness and correctness?
Can I use an LLM to grade RAG answers?
Which RAG evaluation metrics matter most?
Can I generate the test set synthetically?
How often should RAG evals run?
Where Gilzor fits
If you have a RAG pipeline and no reliable way to say how good it is, that is the work we offer under RAG evaluation services: a test set built from your real questions with your experts, separate retrieval and answer scorecards, a calibrated judge, and a regression run in your CI. We build custom RAG systems and add RAG to existing products as well, so the findings can go straight into fixes. A 2-week fixed-price Discovery Sprint is the usual way to start. Tell us what your system answers, where it goes wrong and what a wrong answer costs you.
No sales pitch
Get a straight answer for your project
Tell us what you’re building. We’ll reply with options, a rough cost and timeline. If we’re not the right fit, we’ll say so.
CTO of Gilzor. Responsible for architecture and the engineering standards our teams work by.
LinkedIn →Gilzor · AI/ML Solutions partner
Need a team for your AI product?
Services
AI/ML SolutionsComputer vision, NLP, predictive analytics, automation.→By company type
Selected projects
The team behind them
