ch09-analysis-lab

name

ch09-analysis-lab

description

Guides the Chapter 9 (Analysis stage) lab of the book "Applied AI in Cyber Threat Intelligence" (Packt): the SAT-scaffolding pipeline and "Chap 9 Simulations.ipynb". Use when running chapter-09/ code or evals — ALWAYS read before the ground-truth eval: the shipped ground_truth.csv resume trap silently replays recorded results with zero API calls, and the write-back can corrupt the CSV. Also covers grid search, 14-case judge calibration, and adk web setup.

Chapter 9 — Analysis Stage (SAT pipeline + eval notebook)

Three-agent LoopAgent pipeline scaffolding Structured Analytic Techniques (14-SAT catalog), plus Chap 9 Simulations.ipynb with grid search (optional), 5-scenario ground-truth eval, and 14-case judge calibration.

The most important thing to know — the resume trap

ground_truth.csv ships with its result columns pre-filled with the authors' recorded run, and the GT eval cell treats any row with a non-empty F1 as "already complete". Consequence: running the eval as shipped replays the recorded results with zero API calls ("Resuming — 5 scenario(s) already complete") — a reader can believe they reproduced numbers they never ran.

• To run a real eval: first blank the result columns (F1, Verdict, etc.) in a copy-protected way — e.g. cp ground_truth.csv ground_truth.canon.csv first, then clear the result cells and run.
• To just study the recorded results keylessly: run the loader/table cells as-is; that's a legitimate path.

Write-back corruption — fixed in the repo (history note)

The repo's cell 17 now writes back the resumed entries' true counts (r.get("n_pipeline", ...) / r.get("n_matched", ...)), so the replayed path writes back exactly what it read — the historical bug that zeroed Pipeline_Findings/Overlap on all rows is fixed here. Two cautions remain:

• If a reader's copy of the notebook predates this fix, the old cell 17 corrupts on the resumed path — symptom: those two columns all 0 with F1/Verdict intact. Recover with git checkout -- ground_truth.csv and update the notebook to the current repo version.
• The general canon rule still applies: the write-back overwrites ground_truth.csv with whatever gt_results holds — copy the CSV aside before real (fresh) eval runs.

Other canon files that get overwritten

my_agent/judge_eval_results.json ships pre-filled with the recorded 12/14 run; executing the judge calibration (cell 20 or judge_eval.py) overwrites it. Same recovery: git checkout -- my_agent/judge_eval_results.json.

Lab checklist

- [ ] venv + both requirements files (ch9's include jupyter + jinja2); .env in chapter-09/
- [ ] adk web from chapter-09/ (agent constructs keylessly; key needed at first chat)
- [ ] Notebook: keyless study path or real evals per above

Cost + doneness

Grid: 450 runs (optional — "left as an exercise"; best_config.json ships hand-authored). GT: 5 scenarios, avg F1 ≥ 70% (recorded: 77.9%, 3/5 pass individually — disclosed). Judge: 14 calls, ≥ 80% (recorded: 12/14 = 85.7%, the two misses are named in the README with rationale). This chapter's recorded results PASS their thresholds and its canonicity labeling is the most honest in the repo — the GT screenshot showing per-row "PASS" verdicts beside sub-threshold F1s is the pipeline judge's verdict, not the F1 check (the Discussion cell explains).

Directions for improvement (from the chapter)

• The grid search is explicitly "left as an exercise" (README, on best_config.json): running it (450 paid runs) fills the header-only config_search_results.csv and writes a machine-generated best_config.json — the intended reader-completed artifact.
• Recalibrate the judge: the README names the two recorded misses (eval_01, eval_11 — judge over-strict) with rationale. Exercise: adjust the judge instructions in my_agent/judge_eval.py's prompt so those pass without regressing the other twelve; target > 12/14. The shipped judge_eval_results.json is the before-picture (git-recoverable).
• Close the two sub-threshold GT scenarios (F1 0.667 / 0.571 in the recorded run): improve SAT-selection instructions, blank the result columns properly (see the resume-trap section above), and re-run the 5-scenario eval against the ≥ 70% avg F1 bar.

Repo vs printed book

The repo contains post-print fixes; where a printed listing differs slightly from repo code, the repo version is the corrected one. The "frozen in print" issues above are the deliberate exceptions kept to match the book — guide the reader around them at runtime; never edit them in the repo. API keys are free-tier at https://ai.google.dev/gemini-api/docs/api-key; check the chapter README's "Cost expectations" note before any paid cell.