Evaluating RAG Systems: Metrics, Benchmarks, and Continuous Quality Monitoring
Building a RAG system is step one. Proving it works reliably — and detecting when it degrades — requires systematic evaluation.
1. Core RAG Evaluation Metrics
| Metric | What It Measures | How to Compute |
|---|---|---|
| Faithfulness | Does the answer match the retrieved context? | LLM-as-judge: "Is this answer supported by the context?" |
| Answer Relevance | Does the answer address the question? | LLM-as-judge: "Does this answer the question?" |
| Context Precision | Are retrieved docs actually relevant? | % of retrieved docs that are relevant |
| Context Recall | Did we retrieve all relevant docs? | % of relevant docs that were retrieved |
2. Automated Evaluation Pipeline
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision
results = evaluate(
dataset=test_set, # Questions + ground truth answers + retrieved contexts
metrics=[faithfulness, answer_relevancy, context_precision],
llm=evaluation_llm # Use a strong model as judge
)
print(results)
# {'faithfulness': 0.89, 'answer_relevancy': 0.92, 'context_precision': 0.78}
RAG evaluation is not a one-time activity — it's a continuous monitoring system that catches quality degradation before users notice it.
