| name | hsg-rag-validation-and-qa |
| description | What counts as evidence in HSG_RAG and how to produce it. Load when you need to run or interpret the test suites (offline pytest, RUN_LLM_EVAL 34-case fact eval, RUN_UAT_LLM_JUDGE), decide which gates a change must pass before merge, add a new test / eval case / UAT conversation, judge whether a failing case is a flake or a regression, or understand the uat-results golden inventory and the CI gates (pr-llm-eval.yml, main-smoke.yml). Not for debugging failures (use hsg-rag-debugging-playbook) or changing the gates themselves (use hsg-rag-change-control). |
HSG_RAG Validation and QA
All facts date-stamped as of 2026-07-07. This skill defines what "proven" means
in this repo: which suite provides which evidence, which gates block which change,
and how to extend the evidence base without weakening it.
Machine convention (dev Mac): the repo venv has no pytest. Run pytest with
/opt/anaconda3/bin/python -m pytest; run the app with venv/bin/python.
In CI, plain python -m pytest works (pytest installed explicitly). The
two-interpreter trap is owned by hsg-rag-build-and-env — see it before
assuming the anaconda path exists on your machine.
1. The test pyramid as it actually exists
| Layer | Command | Cost | What it proves |
|---|
| Offline suite (default) | /opt/anaconda3/bin/python -m pytest -q | free, ~16 s | invariants, parsers, consent, prompts, facts-file sanity |
| LLM fact eval (34 cases) | RUN_LLM_EVAL=1 /opt/anaconda3/bin/python -m pytest tests/test_llm_fact_eval.py -v | paid, ~3 min | live chain answers facts correctly, no cross-programme contamination, latency cap |
| UAT LLM judge | RUN_UAT_LLM_JUDGE=1 /opt/anaconda3/bin/python -m pytest tests/test_uat_llm_judge.py -v -s | paid, ~10 min | full conversations meet quality bar (avg ≥ 8.5, every case ≥ 8.0) |
| Live probes (streaming, smoke over public domain) | see hsg-rag-diagnostics-and-tooling | free/cheap | production actually behaves as the code claims |
Verified baseline 2026-07-07: offline suite = 286 passed, 1 skipped, 12 deselected in 15.6 s.
The 12 deselected are network/integration-marked tests excluded by
pytest.ini (addopts = -m "not network and not integration").
1.1 Offline suite map (tests/)
| File / package | Guards |
|---|
test_verified_facts.py | facts-file invariants (fees unique per programme, early < final fee), prompt rendering of the facts block, local language-detection heuristics, config sanity |
test_stream_parser.py | incremental JSON response-field stream parser incl. fuzz across chunk boundaries, escapes, unicode |
test_pricing_prompts.py | pricing rules in prompts (current-fee-first, deadline awareness) |
test_tone_and_handover.py | lead-prompt tone rules, user-led booking, advisor routing content |
test_language_handling.py | DE/EN detection, clarification and fallback messages |
test_input_handler_gibberish.py, test_invalid_input_escalation.py | garbage input → NOT_VALID_QUERY_MESSAGE / escalation paths |
test_scope_guardian_offtopic.py | off-topic queries stay refused |
test_main_happy_path.py | app-level happy path with a fake agent (no LLM) |
test_programme_tagging_strategy.py, test_safe_import_reconciliation.py, test_master_transfer_integrations.py | import pipeline strategies and reconciliation |
consent/ (4 files) | consent constants (texts, links, DE+EN coverage), consent state machine, consent logger format, agent-chain session handling |
scraping/ (5 files) | scraper, chunking, resume behaviour, URL utils (some files network-marked) |
test_weaviate_connection.py, test_weaviate_restore_schema.py, test_programme_positioning_real_agent.py, test_reply_speed_real_agent.py | marked network/integration — excluded by default |
tests/conftest.py auto-skips whole files when optional imports are missing
(pytest_ignore_collect + TEST_DEPENDENCIES map). Consequence: a wrong/renamed
dependency makes tests silently vanish, not fail — after dependency changes,
compare the collected count against the baseline above. (Known drift: the map
still lists tests/test_chatbot_improvements.py, which no longer exists.)
1.2 LLM fact eval — the 34/34 release gate
File: tests/test_llm_fact_eval.py. Opt-in via RUN_LLM_EVAL=1
(module-level skipif). Needs OPEN_ROUTER_API_KEY + WEAVIATE_* in .env
(the module docstring still says "30 questions" and "OPENAI_API_KEY" — both
stale; there are 34 cases (31 until 2026-07-08, +3 deadline-state cases) and the chain runs on OpenRouter).
Mechanics you must preserve when touching it:
- Expected values are computed at collection time from
data/database/programme_facts.json (_fee(), _start_year(), _ects()),
so the nightly facts regeneration never breaks the suite. Never hardcode a
price or date into a case.
- Each case:
expect_any = list of token groups; at least one token of
every group must appear in the normalized answer. forbid = tokens that
must NOT appear — this is the cross-programme contamination guard, the
test encoding of the historic wrong-price bug (see
hsg-rag-failure-archaeology). _normalize() lowercases and strips thousand
separators so 77'500, 77,500, 77 500 all match 77500.
- Case taxonomy (34): pricing 8 (the hotspot; includes 3-programme comparison
and deadline logic), deadlines 3, starts 3, duration 3, language/format 4,
advisors 3 (with forbid-guards against wrong advisor), grounding/honesty 3
(no invented facts, no "six-figure" vagueness), conversational 4 (fit
questions, booking intent, overview).
- Latency gate: every turn must finish <
MAX_TURN_SECONDS = 25.0 —
catches gross regressions like an accidental reasoning-model switch.
- Flake absorption:
FLAKE_RETRIES = 1 — one retry with a fresh chain and
session (commit 8d0136d). A real regression fails both attempts.
Single case: RUN_LLM_EVAL=1 ... -k "de_price_emba".
1.3 UAT LLM judge
UAT = User Acceptance Testing (the term is used unexpanded across this skill
library; this suite is its home). File: tests/test_uat_llm_judge.py. Opt-in
via RUN_UAT_LLM_JUDGE=1.
Scenarios come from tests/fixtures/UAT.xlsx (one sheet per case; sheets
About, TestProtocol, Reporting are skipped). Each UATCase carries
persona, user turns, expected behaviour, success criteria, red flags, expected
language, greeting expectation.
- Judge model:
openai/gpt-4o-mini via OpenRouter by default (consistent
with the app; opt out with UAT_USE_OPENROUTER=0, override with
UAT_JUDGE_MODEL).
- Two thresholds, both required (commit
532baba): average score
≥ UAT_AVERAGE_MIN_SCORE (default 8.5) AND every case
≥ UAT_CASE_MIN_SCORE (default 8.0). Gate test:
test_uat_average_score_passes_llm_judge.
- Judge JSON can arrive malformed (observed 2026-07-06, 1 of 11 cases):
JUDGE_PARSE_ATTEMPTS = 3 re-judges instead of scoring 0 (commit c48b2c1);
persistent malformed JSON still raises. Unit-tested via monkeypatch in the
same file (those tests run offline).
- Results are written per case to
UAT_RESULTS_DIR (default uat-results/).
- The file also contains offline tests (workbook structure, deterministic
deadline-status enrichment
test_tuition_deadline_status_enrichment_is_deterministic,
date-independent hard-facts test — commit ca11dec).
2. Golden inventory — uat-results/TC-*.json
11 judged transcripts from the last accepted UAT run: TC-EMBA-01..03,
TC-IEMBA-01..02, TC-EMBAX-01, TC-EDGE-01..05. Each file: case_id,
title, score, thresholds, passed, verdict, issues,
criteria_missed, full judgement (strengths/weaknesses) and transcript
payload. Uses:
- Reference evidence of the accepted quality bar — compare a new run's
verdicts against these before arguing a score change is noise.
- Regenerated on every judged run (files are overwritten); commit them when a
run is accepted as the new baseline, per change control.
3. Acceptance discipline — which gate blocks what
CI (verified in .github/workflows/):
| Gate | Trigger | What runs | Threshold |
|---|
pr-llm-eval.yml ("PR UAT LLM Judge") | every PR to main | UAT judge suite | avg ≥ 8.5, every case ≥ 8.0 |
main-smoke.yml ("Main Smoke") | every push to main | 34-case fact eval | all pass; asserts EXPECTED_SMOKE_TESTS=34 collected |
deploy.yml | push to main / after facts run | deploy + health checks | /health local + public |
Caveat: main-smoke.yml and deploy.yml run in parallel — the deploy is
NOT blocked by the smoke result. A red smoke on main means production may
already be running the bad commit: treat as incident, fix or roll back
(hsg-rag-run-and-operate).
Minimum evidence per change class (details and rationale in
hsg-rag-change-control):
| Change | Offline suite | 34/34 fact eval | UAT judge | Live probe |
|---|
| Prompts / model / chain / middleware | required | required (PR merge triggers it on main anyway — run locally first) | required (runs on PR) | streaming + latency probe |
Facts pipeline / programme_facts.json schema | required | required (soll-values must stay dynamic) | — | alert-chain awareness |
| UI / consent / booking | required (consent package) | — | recommended | smoke over public domain |
| Scraping / import | required (scraping package) | — | — | object counts before/after |
| Docs only | — | — | — | — (deploy skips doc-only pushes) |
Numbers decide, not impressions: a quality claim without a full fact-eval run and UAT
scores is an opinion.
4. How to add evidence
A new offline test: put it under tests/ (or the matching subpackage);
mark it @pytest.mark.network / @pytest.mark.integration if it needs the
outside world, otherwise it must pass with no network and no .env keys. If it
imports optional dependencies, add the file to TEST_DEPENDENCIES in
tests/conftest.py. Verify: run the offline suite and check it collects (the
silent-skip trap above).
A new fact-eval case: append a dict in build_cases() — id prefixed
de_/en_, lang, query, expect_any (token groups), forbid
(contamination tokens for every pricing/advisor case). Pull every volatile
value through _facts()/_fee()/_start_year()/_ects(); never literal
prices/dates. Then update EXPECTED_SMOKE_TESTS in
.github/workflows/main-smoke.yml (it pins the collected count — forgetting
this fails Main Smoke by design) and the full-eval references per
hsg-rag-docs-and-writing.
A new UAT conversation: add a sheet to tests/fixtures/UAT.xlsx following
an existing case sheet (persona, user turns, expected behaviour, success
criteria, red flags; language cues like "Websiteaufruf auf Deutsch" drive
expected_language). Sheet names in SKIPPED_SHEETS are ignored. Run the
judge suite once locally; commit the new uat-results/TC-*.json with the
change.
5. Flake policy
Legitimate, already built in — do not stack more retries on top:
- One fresh-session retry per fact-eval case (sampling noise;
FLAKE_RETRIES=1).
- Up to 3 judge re-calls on malformed judge JSON (infrastructure, not quality).
Not legitimate: raising FLAKE_RETRIES, lowering UAT thresholds, widening
expect_any groups, or deleting forbid tokens to get CI green. A case that
fails twice with a fresh session is a regression. Contamination and latency
failures are never flakes. Threshold changes are change-control decisions with
before/after numbers, not test fixes.
When NOT to use this skill
- Diagnosing why a case fails or prod misbehaves →
hsg-rag-debugging-playbook
- Changing gates/thresholds or classifying a change →
hsg-rag-change-control
- Measuring latency, streaming, live smoke probes →
hsg-rag-diagnostics-and-tooling
- What the correct programme facts are →
emba-domain-reference
- Past failed approaches behind these guards →
hsg-rag-failure-archaeology
- Deploy/rollback mechanics →
hsg-rag-run-and-operate
Provenance and maintenance
Written 2026-07-07 from repo state on main (post PR #69). Re-verify volatile claims:
/opt/anaconda3/bin/python -m pytest -q
/opt/anaconda3/bin/python -m pytest --collect-only -q | tail -3
grep -c "dict(id=" tests/test_llm_fact_eval.py
grep "EXPECTED_SMOKE_TESTS" .github/workflows/main-smoke.yml
grep -E "MIN_SCORE|PARSE_ATTEMPTS" tests/test_uat_llm_judge.py | head -4
ls uat-results/
cat pytest.ini
