RAG Eval Drift: When Our Retriever Outpaced the Generator

The Incident: Green Metrics, Red Customers

We shipped a RAG system for a Tier-1 investment bank in Mumbai to answer regulatory and compliance queries. The retriever was a dense embedding model (all-MiniLM-L6-v2) plus BM25 hybrid. The generator was a fine-tuned Llama-3-8B on legal Q&A pairs. Our eval set had 500 hand-written queries sampled from pre-launch user interviews. Offline metrics looked green: NDCG@10 = 0.78, Recall@100 = 0.91.

Week 1 post-launch, support tickets spiked from legal analysts. Users reported: "The bot used to find the right clause, now it just makes stuff up." Offline eval scores stayed green. We pulled a production query sample (n=2000) and found 35% new query types not in the eval set. New queries were long Slack copy-pastes, multi-hop regulatory cross-references, and "compare X and Y" tasks. The retriever still fetched relevant docs, but the generator hallucinated on unseen query shapes.

The Setup: Legal Knowledge Assistant for a Financial Client

We built the system at Agentic Academy Labs in Sikar, India, working with a 12-person SaaS team in Pune. The retriever used all-MiniLM-L6-v2 embeddings with BM25 hybrid search. The generator was fine-tuned Llama-3-8B on legal Q&A pairs. The eval set had 500 hand-written queries from pre-launch interviews.

Offline metrics were green: NDCG@10 = 0.78, Recall@100 = 0.91. We deployed to production on a Tuesday morning IST.

The Drift: What Actually Happened

By Thursday, support tickets spiked. Legal analysts said the bot used to find the right clause but now made stuff up. Offline eval scores stayed green.

We pulled a production query sample (n=2000) and found 35% new query types not in the eval set. New queries were long Slack copy-pastes, multi-hop regulatory cross-references, and "compare X and Y" tasks.

The retriever still fetched relevant docs, but the generator hallucinated on unseen query shapes.

What We Tried and What Failed

Attempt 1: Blame the Generator

Hypothesis: Llama-3 fine-tune overfit to short, well-formed eval queries.

We tried prompt engineering with "answer only from context" guardrails. No improvement. Hallucinations persisted.

Root cause missed: eval set was short questions (avg 7 words), prod was long dumps (avg 42 words).

Attempt 2: Retrain the Retriever

Hypothesis: embedding model could not handle long, noisy queries.

We switched to multi-qa-MiniLM-L8 cosine and added query expansion. NDCG@10 improved to 0.81 on eval, but user complaints unchanged.

Root cause missed: eval queries and prod queries occupied different regions of embedding space.

Attempt 3: Refresh the Eval Set

Hypothesis: stale eval set no longer representative.

We sampled 500 new queries from last 30 days of prod traffic. NDCG@10 dropped to 0.62, but now correlated with user satisfaction (r=0.73).

This was the first honest signal.

The Working Approach: Drift Detection in Production

The Query-Distribution Drift Test

We implemented a 40-line drift test comparing eval and prod query distributions:

import numpy as np
from scipy.spatial.distance import cosine
from scipy.stats import wasserstein_distance
from sklearn.linear_model import LogisticRegression

# 1. Embedding-space centroid distance
eval_centroid = np.mean(eval_embeddings, axis=0)
prod_centroid = np.mean(prod_embeddings, axis=0)
centroid_drift = cosine(eval_centroid, prod_centroid)

# 2. Length distribution (Earth Mover Distance)
length_drift = wasserstein_distance(
    [len(q.split()) for q in eval_queries],
    [len(q.split()) for q in prod_queries]
)

# 3. Intent-class distribution (Total Variation Distance)
intent_dist_eval = np.bincount(eval_intent_labels, minlength=5) / len(eval_intent_labels)
intent_dist_prod = np.bincount(prod_intent_labels, minlength=5) / len(prod_intent_labels)
intent_drift = 0.5 * np.sum(np.abs(intent_dist_eval - intent_dist_prod))

# Alert thresholds: centroid > 0.3, length > 5.0, intent > 0.15
if centroid_drift > 0.3 or length_drift > 5.0 or intent_drift > 0.15:
    print(f"DRIFT DETECTED: centroid={centroid_drift:.3f}, length={length_drift:.3f}, intent={intent_drift:.3f}")

Continuous Monitoring Pipeline

We deployed the drift test as a daily cron job on /prod/queries/ (last 24h window). Eval embeddings stored in /models/eval_set_embeddings.npy. Production query embeddings logged to /data/prod_query_embeddings/. Intent classifier was zero-shot 5-class (definition, comparison, troubleshooting, configuration, compliance). Alert channel was #rag-evals Slack bot with drift breakdown.

Per-Cluster Gating for Releases

We clustered the eval set into 5 intent clusters using K-Means on embeddings. For each release, we required NDCG@10 >= 0.70 on every cluster with n >= 5. We blocked release if any cluster regressed by more than 1 point. Reports included aggregate NDCG@10 plus per-cluster breakdown in /reports/release_eval/.

Pitfalls We Would Warn an Intern About

Pitfall 1: Aggregate Metrics Hide Slice Regressions

NDCG@10 = 0.75 looks great until you see it is 0.92 on definitions and 0.41 on comparisons. Always break down by intent cluster, especially for new query types.

Pitfall 2: Length Blindness in Eval Design

Short eval queries (5-10 words) do not represent long prod queries (30-90 words). Retriever behavior changes dramatically with query length. Eval must match.

Pitfall 3: Stale Baselines Gate Against Themselves

"Beating the baseline" means nothing if the baseline was measured on stale data. Always compare against a recent baseline (last 7 days of prod), not the original launch eval.

Pitfall 4: Retrieval Metrics Do Not Catch Generation Failures

Retriever returns correct docs (Recall@100 = 0.95), but generator contradicts them. Must measure faithfulness: fraction of answer sentences supported by retrieved context. Use ragproof.generation.faithfulness or ClaudeJudge for entailment-grade scoring.

Pitfall 5: No Continuous Monitoring Means Silent Degradation

Knowledge base changes (new docs, restructured content) degrade retrieval silently. Track reranker top-1 score distribution over a moving window. Alert on mean drop > 0.05.

What We Would Do Differently Next Time

Ship Drift Detection Before the Model

Integrate the 40-line drift test into CI/CD from day one. Block deployment if drift exceeds thresholds against the current eval set. Make drift detection a prerequisite, not an afterthought.

Design Eval Sets for Distribution Matching

Sample eval queries from the same time window as prod traffic (rolling 30-day window). Ensure length distribution matches: if prod avg is 42 words, eval should be too. Include all intent classes proportionally, not just the easy ones.

Add a Generation Evaluation Layer

Measure faithfulness and answer relevance on every release. Use ragproof.generation.faithfulness (embedding-support heuristic) as default. Add ClaudeJudge for high-stakes queries where hallucination costs money.

Implement Per-Cluster Gating

Never ship a change that regresses any intent cluster with n >= 5 by more than 1 NDCG point. Report per-cluster metrics in every release eval, not just aggregate. Use cluster gating to catch regressions on niche but critical query types.

Track Retrieval Quality Continuously in Production

Log reranker top-1 scores for every prod query to /data/reranker_scores/. Compute moving average over 1000-query windows. Alert when mean drops below baseline - 0.05 (set at launch). Run full retrieval eval (hit@k, MRR, nDCG, recall) hourly on a sample of prod queries.

Close the Evals Blind Spot with Layered Evaluation

Layer 1: Retrieval eval (context recall, context precision, nDCG@10) runs before generation. Layer 2: Generation eval (faithfulness, answer relevance) runs after generation. Layer 3: Drift detection runs continuously on prod traffic. If context recall < 0.7, do not debug generation. Fix retrieval first.

The headline is this: your eval scores can be high while your product is broken. The likeliest reason is not your retriever or your model. It is that your eval set has drifted away from the queries your users are actually sending Your RAG Eval Set Is Probably Wrong. The Test That Catches It.

The 40 lines above will not fix the drift. They will tell you it happened, the day it happened, before customer-success forwards the screenshot Eval drift, when retrieval metrics climb while users complain.

Closing the evals blind spot requires two things: a retrieval evaluation layer that runs before generation, and a continuous monitoring layer that runs in production. Neither is optional if the system output has downstream consequences Why Your RAG System Cannot Tell When It Is Wrong.

We now ship drift detection before the model, design eval sets for distribution matching, and add a generation evaluation layer on every release. The legal team stopped filing complaints. The dashboards stayed green for the right reasons.