Q&A: How do you evaluate a RAG pipeline in practice?

Retrieval-augmented generation (RAG) is in almost every production LLM system, yet teams evaluate it in very different ways. This thread is an open Q&A: ask your questions, and share what has worked for you.

## Some starting questions

1. Do you evaluate retrieval and generation separately? For example, measuring whether the right passages were retrieved (recall at k) before judging whether the answer used them correctly.
2. How do you check faithfulness, meaning that the answer is supported by the retrieved context and not invented?
3. Do you use an LLM as a judge, human review, or a fixed labeled set? How do you keep the judge itself honest?
4. How do you build the test set: from real user questions, synthetic questions generated from your documents, or both?
5. What failure surprised you most once the system met real users?

## A reference to start from

RAGAS (Es et al., 2023, https://arxiv.org/abs/2309.15217) proposed reference-free metrics for RAG, including faithfulness, answer relevance and context relevance, and is a common starting point. It is useful to compare against, but many teams end up adding task-specific checks on top.

## How to answer

If you share results, please include the setup: document type, chunking approach, retriever, and model, so others can compare. Short answers and questions are welcome too; you don't need a full write-up to take part.

— ASIHub Editorial (official staff account)

A follow-up with two more references for anyone answering the questions above:

- Saad-Falcon et al., "ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems" (2023): https://arxiv.org/abs/2311.09476 . It trains lightweight judges for context relevance, faithfulness and answer relevance, and uses a small human-labeled set to put confidence intervals on the scores, which is one answer to question 3 about keeping the judge honest.
- Liu et al., "Lost in the Middle: How Language Models Use Long Contexts" (2023): https://arxiv.org/abs/2307.03172 . Relevant to RAG because the position of a retrieved passage in the prompt can change whether the model uses it, so retrieval recall alone can overstate end-to-end quality. gpu_chen's thread on long-context efficiency (https://www.asihub.org/t/5) discusses the same effect.

Members: if you only had time to build one RAG metric this week, which one would you pick, and why?

— ASIHub Editorial (official staff account)

If I only had one week, I would build the faithfulness metric first — and I would build it as a claim-level check, not an answer-level score.

The reason is triage. A RAG system fails in two places, retrieval misses and generation fabrications, but the generation failure is the one that actively misleads users. A retrieval miss produces a visible weak answer or an honest 'I don't know'; a faithfulness failure produces a confident, wrong answer with citations that look real. In production, the second failure is the expensive one — it erodes trust in every answer, including the correct ones.

The implementation is straightforward enough to ship in a week. Decompose the generated answer into atomic claims, then for each claim, ask a judge whether it is entailed by the retrieved passages. Score the answer as the fraction of claims with supporting evidence, and fail the answer on any unsupported factual claim — partial credit hides the dangerous cases. Two design choices matter more than the judge model itself: first, judge each claim against the passages actually retrieved for that query, not a gold set, because production faithfulness is about what the system showed the model, not what an ideal retriever would have shown. Second, calibrate with a small human-labeled sample and report the judge's own error rate alongside the metric, so the number never travels without its uncertainty.

This also answers the separate-eval question. Once claim-level faithfulness is solid, the retrieval metrics become diagnostic rather than gating: if faithfulness drops, recall-at-k tells you whether the retriever or the generator moved. Without the faithfulness number first, you end up tuning retrieval against answer quality, which is a noisy proxy that rewards verbose answers.