summarize-assessment

name

summarize-assessment

description

Use after the napkin_math pipeline has produced parameters/bounds/scenarios/montecarlo JSON to generate a plan assessment (assessment.md) — a thin interpretation layer over the intermediary artifacts. Emits a JSON manifest, a provenance map, gate verdicts (Critical / Fragile / Marginal / Robust), failure drivers, confidence and trust boundaries, scenario sanity check, and suggested next actions. The artifact is a navigation/judgment file, not a copy of the raw simulation data.

Summarize napkin_math outputs into a plan assessment

Overview

A thin wrapper around experiments/napkin_math/summarize_assessment.py. The script reads the pipeline artifacts and emits assessment.md next to them. The output is an interpretation layer: it tells the next reader of this directory what the simulation tested, which gates fail or pass, which inputs drive the result, which assumptions remain unvalidated, and what to inspect next. The raw distributions live in montecarlo.json; assessment.md references them via the provenance map rather than reproducing them.

When to Use

• The Monte Carlo stage has just produced montecarlo.json and the user wants a verdict
• The user asks "is this plan in trouble?" or "what does the simulation say?"
• After any iteration on bounds or calculations, to see how the gate signals moved

Not for: producing the simulation itself (monte-carlo), running scenarios (run-scenarios), or extracting parameters from a report.

Workflow

1. Locate the inputs. Required: parameters.json. Optional but recommended: bounds.json, scenarios.json, montecarlo.json. validation.json and montecarlo_settings.json are picked up automatically from the same directory if present. The script degrades gracefully if any optional file is missing — it just omits that section. If parameters.json is missing, ask.

2. Invoke the script. Requires Python 3.11+ (no extra deps):

   /opt/homebrew/bin/python3.11 experiments/napkin_math/summarize_assessment.py \
     --parameters   <path>/parameters.json \
     --bounds       <path>/bounds.json \
     --scenarios    <path>/scenarios.json \
     --montecarlo   <path>/montecarlo.json \
     [--output      <path>/assessment.md]
   

   Default output: <dir-of-parameters>/assessment.md. The script prints the output path on stdout.

3. Report back. Tell the user the output path. If the user asks for a verdict in-conversation, read assessment.md and quote the gate-verdict rows and critical-findings bullets verbatim — don't paraphrase.

How verdicts are decided

Verdicts come from the user's own threshold definitions in the Monte Carlo settings. No identifier-string or unit-string interpretation — domain-bias-free.

Each threshold has an operator (>=, <=, etc.) and a value. The user wrote the threshold because they want it to pass. The verdict is the pass probability in the simulation:

Band	Pass probability	Verdict	Note
≥ 80%	strong majority	Robust	passes in the strong majority of runs
50–80%	uncomfortable	Marginal	passes more often than not but uncomfortably close
20–50%	minority pass	Fragile	fails in the majority of runs
< 20%	rarely passes	Critical	rarely passes under current bounds

Any output classified Critical or Fragile also gets a "bottom line" callout at the top of the report.

The script does not invent thresholds for outputs the user did not declare. To get a verdict on an output, declare a threshold on it in the Monte Carlo settings file.

Audience and tone

assessment.md is written to be consumed by the next program or process that touches this directory — a downstream pipeline stage, a planning loop, a follow-on extractor, a future invocation of this same workflow. A human can also read it, but the writing optimises for token-density of useful signal over engagement hooks. The output describes what the file is (an interpretation layer over the simulation artifacts); it does not label its audience.

What that means concretely:

• The file leads with a machine-readable JSON manifest. Every prose section after it is a structured form of those same signals.
• Stable retrievable section names (## Critical findings, ## Gate verdicts, ## Failure drivers, ## Suggested next actions). No colorful or narrative-flavoured headings.
• No reader-engagement prefixes ("If you read nothing else, read this", "Stop and pay attention", "Important:"). The structural markers and verdict labels carry that weight.
• No filler sentences whose only job is to motivate the next sentence. Lead with the substantive claim.
• Keep substantive explanations (what a verdict label means, what a column shows, what makes an item belong in a section). Those are signal, not filler.
• Don't apologise for or hedge the bad news. State it.

Writing rules — apply to the script's output AND to anything you say back to the user about the assessment

These are not stylistic preferences. They are how this skill is meant to communicate.

1. Critical findings first. After the artifact contract, machine summary, provenance map, modelling frame, and simulation settings, the first interpretation section is ## Critical findings. It consolidates every signal that the plan does not survive its own assumptions: Critical and Fragile thresholds, scenario warnings, numbers the model could not compute, and inputs the plan does not supply at all. If nothing qualifies, the section is omitted entirely — silence is the only acceptable form of good news.

2. No sugar-coating. A 5% pass probability is "rarely passes under current bounds", not "shows some challenges". A base-scenario value on the wrong side of a declared threshold is "the gate fails at the plan's own central assumptions", not "may warrant further attention". Use the strongest accurate language without overclaiming epistemic certainty — never "the math says it certainly will" — but never soften the result either; if the script's wording softens a result, fix the script.

3. No sycophancy. Never start a paragraph with "Great plan, but..." or "The team has done strong work; one concern is...". The downstream consumer has the plan available. It does not need praise from the report.

4. No hedging phrases. Banned in both the script's emitted text and in conversational reporting:

   • the honest read is, frankly, to be fair, in fairness, candidly, let's be real, look, the truth is
   • rhetorical "I'll be honest with you" / "to put it bluntly" / "if I may"

   These imply the default mode is dishonest or evasive. State the claim directly.

5. Hedges about data vs hedges about the speaker. Hedges that point at the underlying simulation are fine: "the simulation shows", "based on the bounds we have", "within the assumed ranges". Hedges that point at the writer's posture are not.

6. Quote the verdicts; don't paraphrase. When the script emits **Critical** — rarely passes under current bounds, report it as Critical — rarely passes under current bounds. Don't summarise it as "this one is concerning". The verdict bands are precise and load-bearing.

Sections in the generated assessment.md

Order is deliberate. Stable section names — programmatic consumers retrieve by heading text, so the headings stay the same regardless of plan domain:

• # Assessment: <plan name> — title plus a 2-line frontmatter (type, primary goal).
• ## Artifact contract — declares what this file is (an interpretation layer over the simulation artifacts) and what it is not (a copy of the raw simulation data, an external feasibility proof, a probability calibration).
• ## Machine summary — a JSON code block with the compact manifest: assessment_schema_version (currently 6), artifact_type, plan_name, artifact_set (version / plan_slug / relative_dir — the portable identifier), source_plan_dir (absolute path; local-only), primary_model_result (a structured object: overall_risk_band ∈ doom/fragile/marginal/viable/unknown, basis — a one-line disclaimer that the band reflects the worst declared gate's pass-rate band and is not a calibrated whole-plan probability, reason, worst_gate, worst_gate_pass_rate), validation_status, simulation (n_runs/seed/distribution_default), primary_failed_gates, primary_uncertainty_drivers, known_unmodelled_existential_gates (flat list of ids the extractor flagged as existential but unmodelable; empty when none), assessment_scope_warning (string when the unmodelled list is non-empty, null when empty — a one-line warning to programmatic consumers that the simulation is partial), do_not_treat_as, schema_notes (allowed enums for overall_risk_band, verdict, basis, threshold_basis, plus the primary_model_result_semantics disclaimer). The basis_enum is intentionally wider than what the current pipeline emits (report_derived, model_assumption) — it reserves report_explicit, report_inferred, external_reference, manual_override, unknown for future provenance types. JSON, not YAML — that is intentional.

Common Mistakes

Mistake	Fix
Running before montecarlo.json exists	Gate verdicts, failure drivers, and the machine summary's primary_model_result all depend on simulation output. Run the Monte Carlo stage first.
Reading the markdown and paraphrasing the gate verdicts	Quote them. The cutoff bands and phrasing are deliberate.
Treating the machine summary as authoritative without reading the prose	The JSON manifest is a compact pointer, not a proof. The aggregation warning, trust boundaries, and failure-driver rows are load-bearing context.
Treating a Marginal verdict as good news	Marginal means "passes in 50–80% of runs" — that's the same as "fails up to 50% of the time".
Inventing a threshold to make a number look good	Thresholds reflect the user's success criteria. Don't fabricate them after the fact.

Reference

• Script (authoritative): experiments/napkin_math/summarize_assessment.py
• Companion skills: ../monte-carlo/SKILL.md, ../run-scenarios/SKILL.md, ../generate-bounds/SKILL.md, ../extract-parameters-from-full/SKILL.md, ../extract-parameters-from-digest/SKILL.md
• Example output: any assessment.md under experiments/napkin_math/output/<version>/<plan>/