What is a good hit rate?

Aim as high as practical on your own questions, and investigate every miss. A hit rate that looks high on easy questions can hide weak spots, so include hard and ambiguous ones.

When should I add a reranker?

When the right passage is usually among the top 20 candidates but not the top 3. Retrieving more and reranking often improves precision at a modest cost in latency.

RAG Step 5: Evaluation and Production Best Practices

Introduction

A RAG demo that works on three questions says little about quality. To improve a system and trust it, you need to measure retrieval and answers separately and watch it after launch. This final lesson of the RAG course covers evaluation methods, common improvements, and production concerns.

Prerequisites

All earlier RAG lessons
A list of realistic questions with known answers

Step-by-Step Guide

Step 1: Build an evaluation set

Write 30 to 100 questions people would really ask. For each, record the source passage that contains the answer and, if you can, a reference answer. Include questions the documents cannot answer, so you can check that the system declines.

eval_set = [
    {"question": "How many vacation days do new employees get?",
     "source": "handbook.pdf", "page": 12},
    {"question": "What is the CEO's favorite color?",
     "source": None, "page": None},   # not answerable from the documents
]

Step 2: Measure retrieval separately

A wrong answer is often a retrieval problem. Hit rate at k measures how often the right source appears in the top k results. Check it before looking at generation, because the model cannot use a passage it was never shown.

def hit_rate(store, eval_set, k: int = 4) -> float:
    answerable = [c for c in eval_set if c["source"]]
    hits = 0
    for case in answerable:
        results = store.search(case["question"], k=k)
        if any(r["source"] == case["source"] and r.get("page") == case["page"] for r in results):
            hits += 1
    return hits / len(answerable)

print("Hit rate@4:", hit_rate(store, eval_set))

Step 3: Measure answer quality

Judge whether answers are correct, complete, and faithful to the sources, meaning every claim is supported by the retrieved text. Score a sample by hand first. A model can grade at larger scale, but calibrate it against your human scores. Track how often the system correctly declines unanswerable questions.

Step 4: Improve the weakest link

If retrieval is weak, try better chunking, metadata, query rewriting, hybrid search that combines keyword and vector scores, or a reranker that re-scores a larger candidate set with a more accurate model. If retrieval is good but answers are poor, improve the prompt or use a stronger model. Change one thing at a time and re-measure.

Step 5: Prepare for production

Plan how documents are updated and re-indexed, enforce per-user access control before retrieval, log questions, retrieved chunks, and answers, and monitor latency and cost. Treat retrieved documents as untrusted input, since a document can contain text that tries to instruct the model. Collect user feedback and add real failures to your evaluation set.

FAQs

What is a good hit rate?: Aim as high as practical on your own questions, and investigate every miss. A hit rate that looks high on easy questions can hide weak spots, so include hard and ambiguous ones.
When should I add a reranker?: When the right passage is usually among the top 20 candidates but not the top 3. Retrieving more and reranking often improves precision at a modest cost in latency.

Summary

Evaluate retrieval and generation separately with a realistic question set, fix the weakest link, and prepare for production with access control, logging, re-indexing, and defenses against untrusted document content.