INSIGHTS / AI

Evaluating retrieval-augmented generation before your users do.

A practical method for measuring the quality of RAG systems: building an evaluation set, choosing retrieval and answer metrics, calibrating LLM-as-judge scoring and gating releases on regressions.

Why RAG needs its own test discipline

A retrieval-augmented system can fail in two independent places: it can retrieve the wrong content, or it can answer badly from the right content. Without measuring each separately, teams tune prompts to fix retrieval problems and re-index documents to fix generation problems — and quality drifts with every change.

Build an evaluation set first

The evaluation set is the specification of the system. Build it before tuning anything.

Collect 100–300 real questions from users, support tickets and subject-matter experts
For each question, record the expected answer and the source passages that support it
Include unanswerable questions, ambiguous questions and questions the user is not permitted to see answered
Version the set alongside the code and extend it whenever a production failure is found

Measure retrieval and generation separately

Use different metrics for each stage so that a regression can be traced to its cause.

Retrieval: recall@k (did the right passages appear in the top k?), mean reciprocal rank and context precision
Generation: groundedness (is every claim supported by retrieved content?), answer relevance and correctness against the reference
Behaviour: correct refusals for unanswerable or unauthorised questions, citation accuracy and response format compliance
Operations: latency percentiles, tokens and cost per answer

Use LLM-as-judge carefully

Model-based graders scale evaluation, but they have biases. Calibrate them: have people score a sample, compare the judge's scores with the human scores, refine the grading rubric until agreement is acceptable, and re-check calibration whenever the judge model changes. Use deterministic checks — exact matches, schema validation, citation lookups — wherever possible.

Gate releases on the results

Run the evaluation suite in CI on every change to prompts, models, chunking, embeddings or retrieval configuration. Define thresholds — for example, no drop in groundedness and no more than a small tolerance on recall@5 — and fail the build when they are breached. Dashboards inform; gates protect.

Close the loop in production

Capture user feedback, unanswered questions and low-confidence responses with tracing. Review them regularly, add representative cases to the evaluation set, and the system improves on evidence rather than anecdote.

Contact: contact@nexaarislabs.com