Your RAG system works but you can't tell if it's getting better. How do you evaluate it?
Shipping RAG is easy; evaluating it is where real engineers differ from demoers.
Decompose: retrieval quality (are the right docs found?) vs generation quality (does the answer use them correctly?) vs end-to-end (is the user happy?).
Retrieval metrics: recall@k (did any gold doc make it into top-k), MRR (where in the list), NDCG (graded relevance). Requires a labeled query→gold-doc set.
Answer metrics: faithfulness (claims supported by retrieved context), relevance (answer addresses the question), groundedness (no hallucinations). LLM-as-judge with rubric works well.
End-to-end: human eval on representative queries, thumbs-up rate in production, escalation rate.
Eval datasets: build from real user queries (sampled + labeled). Synthetic evals miss long-tail reality.
Evaluating RAG means separating retrieval failures from generation failures — they need different fixes. For retrieval I label a few hundred real queries with gold documents and track recall@k and MRR; if recall@5 is low, the embedding model or chunking is broken. For generation quality I use LLM-as-judge with a rubric (faithfulness, relevance, completeness) — judge another model against the retrieved context. End-to-end I measure thumbs-up rate, escalation rate, and regression against a frozen golden eval set every release. A common trap: high BLEU/ROUGE but low faithfulness — the model hallucinates confidently. Faithfulness as a separate metric catches that. I build my eval dataset from production queries (sampled and labeled) — synthetic Q&A pairs systematically miss the weird ways users actually phrase things.
LLM-as-judge drifts across model versions — re-run with the same judge when comparing runs.
Faithfulness matters more than fluency. A polite wrong answer is worse than a hesitant right one.
HireMe AI (即答侠) includes a sample rubric for each metric — interviewers see you've run evals before, not just talked about them.
Use sparingly — they correlate with surface similarity, not correctness. Faithfulness + relevance are more informative.
A few hundred labeled examples catch most regressions. For production, automated tracking on 10K+ production queries captures drift.
HireMe AI listens to your interview in real-time, detects questions, and generates instant answer suggestions — so you never freeze up again.
Try HireMe AI Free