| name | skeptic-review |
| description | Run Dev Skeptic and Platform Skeptic reviews on enriched findings to challenge scores, verify exploitability, and populate TL;DR executive summaries (Phase 5). |
Run both skeptic reviewers over enriched findings for an experiment.
Full agent guidance is in Agents/DevSkeptic.md and Agents/PlatformSkeptic.md.
Prerequisites
Phase 4 (LLM enrichment) must be complete. Verify:
python3 Scripts/Experiments/triage_experiment.py resume
All findings to review must have llm_enriched_at IS NOT NULL.
Running skeptic reviews
python3 Scripts/Utils/run_skeptics.py --experiment <id>
Key flags
| Flag | Purpose |
|---|
--experiment <id> | Experiment ID — required |
--repo <name> | Limit to a single repository (optional) |
--reviewer dev|platform|security|all | Run specific reviewer(s) (default: all) |
The script is idempotent: it checks skeptic_reviews for existing (finding_id, reviewer_type) rows and skips already-reviewed findings. Only re-run if new findings were added or context changed.
What each reviewer checks
Dev Skeptic (reviewer_type = 'dev')
Focuses on code correctness and realistic exploit paths:
- Data source tracing — is the vulnerable input user-controlled or internal-only?
- Actual mitigations present — ORM, parameterized queries, template auto-escaping, input validators
- Test-only dependency scope — does the vulnerable package only deploy to test environments?
- Framework-level controls — does the framework neutralise the finding?
Platform Skeptic (reviewer_type = 'platform')
Focuses on cloud/IaC controls and platform constraints:
- Compensating controls in IaC — WAF, APIM JWT validation, private endpoints, VNet integration
- SKU/tier requirements — does the recommended fix require a paid tier or redeploy?
- Network constraints — VPN, jump hosts, internal-only routes that affect exploitability
- Only credit defenses with evidence — cite the IaC file and line, not assumptions
Security Skeptic (reviewer_type = 'security')
Runs last. Synthesises the Dev and Platform reviews into a final adjusted score:
- Reads the Dev and Platform reviews already written to DB
- Sets the authoritative
adjusted_score that becomes the finding's final score
- Writes the TL;DR executive summary when all three reviews are present
- Run
--reviewer security alone if Dev/Platform are already done and only the final synthesis is needed
Scoring rules
- Final score = Security skeptic's
adjusted_score (incorporates Dev and Platform inputs). Falls back to average of all three scores if the Security reviewer did not run.
score_adjustment = float (negative = downgrade, positive = escalate)recommendation = confirm | downgrade | dismiss | escalate- CRITICAL: Score based on actual exploitable damage with proven defenses, not principle violations alone
Correct example
"Down to 5/10 — APIM JWT validation confirmed in terraform/apim_policies.tf blocks exploitation"
Wrong example
"Keep 9/10 — violates defence-in-depth even though APIM validates JWTs"
TL;DR population trigger
After both Dev and Platform reviews are complete for a finding, populate ## 📊 TL;DR — Executive Summary in the finding MD:
- Final score with adjustment tracking (Security → Dev → Platform)
- Top 3 priority actions with effort estimates
- Material risks summary (2–3 sentences)
- Reason for any score change
DB output per review
reviewer_type: 'dev' or 'platform'
score_adjustment: float
adjusted_score: int 1–10
confidence: float 0.0–1.0
reasoning: full analysis
key_concerns: comma-separated
mitigating_factors: comma-separated
recommendation: confirm | downgrade | dismiss | escalate
Score history row also written to risk_score_history automatically.
After skeptic reviews
python3 Scripts/Generate/render_finding.py --experiment <id> --all
python3 Scripts/Generate/generate_diagram.py <id>
Then proceed to report generation: experiment-run Phase 6, or cloud-summary skill.
Agent review logic
Full reviewer checklists, data source classification, evidence citation rules:
Agents/DevSkeptic.md · Agents/PlatformSkeptic.md
