| name | ch12-feedback-lab |
| description | Guides the Chapter 12 (Feedback stage) lab of the book "Applied AI in Cyber Threat Intelligence" (Packt): the follow-on QIR generation pipeline and "Chap 12 Simulations.ipynb" (cell ids gs-run, gt-run, judge-run). Use when running chapter-12/ code or evals, protecting the shipped ground_truth.csv canon from overwrite, or interpreting the zero-recall Code Publication Impact scenario hiding inside the passing averages. |
Chapter 12 — Feedback Stage (QIR-generation pipeline + eval notebook)
Three-agent loop turning stakeholder feedback into template-validated follow-on
QIRs, plus Chap 12 Simulations.ipynb. This is the healthiest chapter in the
repo: the README's per-cell map matches real notebook cell ids (gs-run,
gt-run, judge-run...), generated artifacts are labeled "not shipped", and
both judge-metric views are reconciled in prose.
Lab checklist
- [ ] venv + both requirements files; .env in chapter-12/
- [ ] adk web with the README's paired QIR+feedback prompt (Figs 10-11 = target)
- [ ] Paid (no cell labels here — see README cost note): demo-run, judge-run (10 calls), gt-run (5 scenarios), gs-run (66 runs, optional)
Missing key fails fast (~28 s) with the key URL. A spurious
Task exception was never retrieved ... _async_httpx_client traceback may
print on top of the real error — ignore it, read the RuntimeError beneath.
Canon protection
ground_truth.csv ships pre-filled with the recorded run. gt-summary
overwrites it by scenario-name lookup, and a partial gt-run followed by
gt-summary silently mixes reader and recorded rows. Copy aside first or
restore with git checkout -- ground_truth.csv. There is no resume logic —
gt-run always runs fresh.
Reading the recorded results
Recorded GT passes both gates (avg Recall 80% ≥ 60%, avg Scope 80% ≥ 70%) —
but only because averaging absorbs one scenario ("Code Publication Impact")
that scores 0.0 on every metric while the pipeline's own verdict on it was
PASS. That zero-row is the judge-leniency failure mode the Discussion warns
about, sitting in the shipped data; point readers at it rather than letting
them treat it as their own error. Judge calibration: 20/21 checks (95%) and
9/10 cases (90%) — both pass; the one miss (compound_not_split) is named in
the README.
Minor notes
session_service.py's persistent: bool = True parameter is decorative —
it always returns an in-memory service; the chapter creates no session state
files needing cleanup.- A vestigial
GS_CHECKPOINT constant in gs-imports is unused (disclosed in
Known Limitations as forward-compat).
Directions for improvement (from the chapter)
The shipped data itself names the two targets:
- The zero-recall scenario ("Code Publication Impact": 0.0 on every metric
while the pipeline's own verdict was PASS) is the Discussion's lenient-judge
warning made concrete. Exercise: improve the Transformation agent's
prompts/templates (
my_agent/requirement_templates.py) until that scenario
produces scoring QIRs, and consider a rule check that refuses a PASS verdict
on an empty QIR set — then re-run gt-run/gt-summary (canon copied aside)
and check the per-row Recall, not just the passing averages.
- The judge's one named miss (
compound_not_split: PARTIAL where FAIL —
compound stakeholder questions must be split into separate QIRs): tighten
the judge instructions until that case scores FAIL, holding the other nine;
targets are the chapter's own gates (≥ 80% both views; recorded 95%/90%).
- Running
gs-run (66 paid runs) legitimately generates the "not shipped"
best_config.json — the reader-completed artifact this chapter labels
honestly.
Repo vs printed book
The repo contains post-print fixes; where a printed listing differs slightly
from repo code, the repo version is the corrected one. The "frozen in print"
issues above are the deliberate exceptions kept to match the book — guide the
reader around them at runtime; never edit them in the repo.
API keys are free-tier at https://ai.google.dev/gemini-api/docs/api-key; check
the chapter README's "Cost expectations" note before any paid cell.
