RAG Eval Harness: 200 Questions That Catch 31% Gaps [2026]

Answer in 50 Words

A 200-question eval set with an adversarial subset catches retrieval gaps before customers do — my Junagadh runner lifted a client from 31% to 78% precision in two weeks. Python runner, pgvector logging, and the question-design sheet below. Wire it into CI and every RAG change proves itself.

I run SaaS Next from Junagadh, Gujarat. I refuse to ship a RAG system without an eval set. That rule comes from the ₹42L-hire story in my salary-benchmarks post: three weeks of work, 31% retrieval precision, no measurement anywhere. Two weeks with a harness took it to 78%. Here is the harness, exactly as I run it.

War Story 1: 31% Precision Nobody Measured

Recap with numbers: 14,000 dealer documents, Hindi + English mixed, pgvector with default chunking (512 tokens, 50 overlap, flat index). First eval run of my 200 questions: 62 correct, 138 wrong or partial — 31%. Failure clusters told the story: 44 misses on Hindi queries (tokenizer split compound words), 38 on multi-hop questions (no chunk carried both facts), 31 on price queries (stale embeddings after a catalog update), 25 miscellaneous. No single fix covered everything. The eval set turned one vague "RAG is bad" into four scoped tickets. That is the whole value proposition: measurement converts despair into a task list.

War Story 2: The Reindex That Silently Broke Prices

Two months later the same client updated 1,100 prices. The ingestion script re-embedded changed docs but kept old chunk IDs for unchanged ones — except a sorting bug reordered chunks within 300 documents, detaching answers from their source rows. Nightly eval caught it: price-question precision fell from 91% to 63% overnight. I diffed the chunk log, found the reorder, pinned deterministic chunk IDs (doc_id::chunk_seq hashed), and re-ran. Precision recovered to 89% by noon. Without the nightly gate, dealers would have quoted stale prices for days. Evals are production monitoring, not homework.

Designing the 200 Questions

My sheet splits 200 into five buckets of 40:

Rules: every question has one reference answer plus the source chunk IDs that support it. Adversarial items carry a note on what trap they set. I retire questions that every run answers correctly for a month and replace them with new failure modes — the set stays hostile.

The Runner (Python)

# eval/run_rag_eval.py — stdlib + psycopg, no framework needed
import json, time, psycopg

DSN = "dbname=catalog user=deepak host=127.0.0.1"
SET = "eval/questions_v7.jsonl"  # {"q":..., "lang":..., "bucket":..., "ref":..., "chunks":[...]}

def answer(q: str) -> dict:
    # call YOUR rag pipeline here; return {"text":..., "chunks":[...], "ms":...}
    from app.rag import ask
    t0 = time.time()
    out = ask(q)
    out["ms"] = int((time.time() - t0) * 1000)
    return out

def grade(got: str, ref: str) -> bool:
    g, r = got.lower(), ref.lower()
    return r in g or all(w in g for w in r.split() if len(w) > 3)

def main():
    rows, lat = [], []
    with psycopg.connect(DSN) as cx:
        for line in open(SET, encoding="utf-8"):
            item = json.loads(line)
            res = answer(item["q"])
            ok = grade(res["text"], item["ref"])
            lat.append(res["ms"])
            cx.execute(
                "INSERT INTO rag_eval_runs(q, bucket, lang, ok, ms, chunks) VALUES (%s,%s,%s,%s,%s,%s)",
                (item["q"], item["bucket"], item.get("lang", "en"), ok, res["ms"], json.dumps(res.get("chunks", []))),
            )
            rows.append(ok)
        cx.commit()
    lat.sort()
    print(f"precision: {sum(rows)}/{len(rows)} = {sum(rows)/len(rows):.1%}")
    print(f"P95 latency: {lat[int(len(lat)*0.95)]}ms")

if __name__ == "__main__":
    main()

Grading starts substring-based (cheap, deterministic) and graduates to LLM-judge only for disputed items. I keep the cheap gate in CI — under 3 minutes for 200 questions against local Postgres — and run the expensive judge weekly. Precision plus P95 latency print on every run; both append to rag_eval_runs so regressions graph themselves.

pgvector Logging Schema

CREATE TABLE rag_eval_runs (
  id bigserial PRIMARY KEY,
  ran_at timestamptz DEFAULT now(),
  q text NOT NULL,
  bucket text NOT NULL,      -- factual | vernacular | multihop | adversarial | freshness
  lang text NOT NULL DEFAULT 'en',
  ok boolean NOT NULL,
  ms integer NOT NULL,
  chunks jsonb NOT NULL DEFAULT '[]'
);
CREATE INDEX ON rag_eval_runs (bucket, ran_at);

Two queries run the practice: precision by bucket per week (finds the weak cluster) and P95 latency trend (finds the slow bleed). My alert rule: any bucket dropping 10 points week-over-week pages me before the standup. The reorder incident above tripped exactly this rule.

Chunk Discipline That Makes Evals Pass

Evals do not fix retrieval; they point at it. Fixes that moved my numbers:

• Deterministic chunk IDs (doc_id::chunk_seq hashed) so reindex never detaches answers.
• HNSW tuning (m=16, ef_search=64) for the 14K-doc scale; P95 310ms → 42ms on the Surat catalog.
• Vernacular-aware splitting: sentence boundaries from an Indic tokenizer instead of byte counts for Hindi/Gujarati docs.
• Freshness lane: re-embed changed docs within the hour, with the eval freshness bucket as the watchdog.
• Metadata filters (dealer, lang, doc_date) applied pre-search, not post-ranked.

Each fix entered behind the eval gate: implement, run 200, keep only what moves precision without regressing P95. Four fixes kept, two reverted. The log remembers so opinions do not have to.

CI Gate: Fail Loud

# nightly + on every retrieval-PR
python eval/run_rag_eval.py | tee /tmp/rag-eval.txt
python eval/check_regression.py --max-drop 5 --min-precision 70

check_regression.py compares tonight against the trailing-7-day bucket average and exits nonzero on drops over 5 points or absolute precision under 70%. A red gate blocks merge. Developers grumble for a week, then start writing questions with their features — which is precisely the culture shift that keeps precision above 80% without heroics.

When NOT to Build This

Do not build a 200-question harness for a 50-document FAQ that changes twice a year — ten spot checks and a quarterly glance suffice. Do not LLM-judge every run at the start; the API bill teaches nothing substring matching cannot for v1. And do not eval without chunk logging: a precision number with no chunk IDs attached is a mood, not a measurement. My minimum viable version is 40 questions, substring grades, and the runs table. Grow from there.

Frequently Asked Questions

How many eval questions does a production RAG need?

Start with 40 across five buckets, grow to 200 as failure modes appear. My 14K-doc client stabilized at 200 with monthly rotation of solved items. Count buckets, not just totals — 200 factual lookups and zero adversarial items still ship blind.

Substring grading vs LLM judge — which first?

Substring first: deterministic, free, runs in CI under 3 minutes. Promote disputed or nuanced items to LLM-judge weekly. My gate runs substring nightly and has caught every production regression so far, including the chunk-reorder incident.

What precision target should I promise?

70% minimum gate, 80%+ healthy for mixed Hindi-English catalogs, 90%+ per bucket for prices and policies that touch money. Promise the gate, not a number — "no merge drops precision 5 points" beats "we guarantee 95%" and survives contact with new documents.

What does an eval harness cost to build?

I bundle it into RAG builds (₹55K–₹85K all-in). Standalone, two days: one for the question set with a domain owner, one for runner + schema + CI gate. The nightly run costs minutes of VPS time on the existing ₹6,200/month box.

Bottom Line

Ship the harness with the RAG, not after: 200 hostile questions, chunk-logged runs, a 5-point regression gate, and precision plus P95 on every printout. My ledger reads 31% → 78% in two weeks and one silent price corruption caught overnight. Measurement is the feature.

Build it with me: AI development for RAG systems, automation notes for nightly eval pipelines, web development for the frontend over the same index, selected work, and contact for the question-sheet template.