summarize-assessment

name

summarize-assessment

description

Use after the napkin_math pipeline has produced parameters/bounds/scenarios/montecarlo JSON to generate a plan assessment (assessment.md) — a thin interpretation layer over the intermediary artifacts. Emits a JSON manifest, a provenance map, gate verdicts (Critical / Fragile / Marginal / Robust), failure drivers, confidence and trust boundaries, scenario sanity check, and suggested next actions. The artifact is a navigation/judgment file, not a copy of the raw simulation data.

Summarize napkin_math outputs into a plan assessment

Overview

A thin wrapper around experiments/napkin_math/summarize_assessment.py. The script reads the pipeline artifacts and emits assessment.md next to them. The output is an interpretation layer: it tells the next reader of this directory what the simulation tested, which gates fail or pass, which inputs drive the result, which assumptions remain unvalidated, and what to inspect next. The raw distributions live in montecarlo.json; assessment.md references them via the provenance map rather than reproducing them.

When to Use

• The Monte Carlo stage has just produced montecarlo.json and the user wants a verdict
• The user asks "is this plan in trouble?" or "what does the simulation say?"
• After any iteration on bounds or calculations, to see how the gate signals moved

Not for: producing the simulation itself (monte-carlo), running scenarios (run-scenarios), or extracting parameters from a report.

Workflow

1. Locate the inputs. Required: parameters.json. Optional but recommended: bounds.json, scenarios.json, montecarlo.json. validation.json and montecarlo_settings.json are picked up automatically from the same directory if present. The script degrades gracefully if any optional file is missing — it just omits that section. If parameters.json is missing, ask.

2. Invoke the script. Requires Python 3.11+ (no extra deps):

   /opt/homebrew/bin/python3.11 experiments/napkin_math/summarize_assessment.py \
     --parameters   <path>/parameters.json \
     --bounds       <path>/bounds.json \
     --scenarios    <path>/scenarios.json \
     --montecarlo   <path>/montecarlo.json \
     [--output      <path>/assessment.md]
   

   Default output: <dir-of-parameters>/assessment.md. The script prints the output path on stdout.

3. Report back. Tell the user the output path. If the user asks for a verdict in-conversation, read assessment.md and quote the gate-verdict rows and critical-findings bullets verbatim — don't paraphrase.

How verdicts are decided

Verdicts come from the user's own threshold definitions in the Monte Carlo settings. No identifier-string or unit-string interpretation — domain-bias-free.

Each threshold has an operator (>=, <=, etc.) and a value. The user wrote the threshold because they want it to pass. The verdict is the pass probability in the simulation:

Band	Pass probability	Verdict	Note
≥ 80%	strong majority	Robust	passes in the strong majority of runs
50–80%	uncomfortable	Marginal	passes more often than not but uncomfortably close
20–50%	minority pass	Fragile	fails in the majority of runs
< 20%	rarely passes	Critical	rarely passes under current bounds

Any output classified Critical or Fragile also gets a "bottom line" callout at the top of the report.

The script does not invent thresholds for outputs the user did not declare. To get a verdict on an output, declare a threshold on it in the Monte Carlo settings file.

Audience and tone

assessment.md is written to be consumed by the next program or process that touches this directory — a downstream pipeline stage, a planning loop, a follow-on extractor, a future invocation of this same workflow. A human can also read it, but the writing optimises for token-density of useful signal over engagement hooks. The output describes what the file is (an interpretation layer over the simulation artifacts); it does not label its audience.

What that means concretely:

• The file leads with a machine-readable JSON manifest. Every prose section after it is a structured form of those same signals.
• Stable retrievable section names (## Critical findings, ## Gate verdicts, ## Failure drivers, ## Suggested next actions). No colorful or narrative-flavoured headings.
• No reader-engagement prefixes ("If you read nothing else, read this", "Stop and pay attention", "Important:"). The structural markers and verdict labels carry that weight.
• No filler sentences whose only job is to motivate the next sentence. Lead with the substantive claim.
• Keep substantive explanations (what a verdict label means, what a column shows, what makes an item belong in a section). Those are signal, not filler.
• Don't apologise for or hedge the bad news. State it.

Writing rules — apply to the script's output AND to anything you say back to the user about the assessment

These are not stylistic preferences. They are how this skill is meant to communicate.

1. Critical findings first. After the artifact contract, machine summary, provenance map, modelling frame, and simulation settings, the first interpretation section is ## Critical findings. It consolidates every signal that the plan does not survive its own assumptions: Critical and Fragile thresholds, scenario warnings, numbers the model could not compute, and inputs the plan does not supply at all. If nothing qualifies, the section is omitted entirely — silence is the only acceptable form of good news.

2. No sugar-coating. A 5% pass probability is "rarely passes under current bounds", not "shows some challenges". A base-scenario value on the wrong side of a declared threshold is "the gate fails at the plan's own central assumptions", not "may warrant further attention". Use the strongest accurate language without overclaiming epistemic certainty — never "the math says it certainly will" — but never soften the result either; if the script's wording softens a result, fix the script.

3. No sycophancy. Never start a paragraph with "Great plan, but..." or "The team has done strong work; one concern is...". The downstream consumer has the plan available. It does not need praise from the report.

4. No hedging phrases. Banned in both the script's emitted text and in conversational reporting:

   • the honest read is, frankly, to be fair, in fairness, candidly, let's be real, look, the truth is
   • rhetorical "I'll be honest with you" / "to put it bluntly" / "if I may"

   These imply the default mode is dishonest or evasive. State the claim directly.

5. Hedges about data vs hedges about the speaker. Hedges that point at the underlying simulation are fine: "the simulation shows", "based on the bounds we have", "within the assumed ranges". Hedges that point at the writer's posture are not.

6. Quote the verdicts; don't paraphrase. When the script emits **Critical** — rarely passes under current bounds, report it as Critical — rarely passes under current bounds. Don't summarise it as "this one is concerning". The verdict bands are precise and load-bearing.

Sections in the generated assessment.md

Order is deliberate. Stable section names — programmatic consumers retrieve by heading text, so the headings stay the same regardless of plan domain:

• # Assessment: <plan name> — title plus a 2-line frontmatter (type, primary goal).
• ## Artifact contract — declares what this file is (an interpretation layer over the simulation artifacts) and what it is not (a copy of the raw simulation data, an external feasibility proof, a probability calibration).
• ## Machine summary — a JSON code block with the compact manifest: assessment_schema_version (currently 6), artifact_type, plan_name, artifact_set (version / plan_slug / relative_dir — the portable identifier), source_plan_dir (absolute path; local-only), primary_model_result (a structured object: overall_risk_band ∈ doom/fragile/marginal/viable/unknown, basis — a one-line disclaimer that the band reflects the worst declared gate's pass-rate band and is not a calibrated whole-plan probability, reason, worst_gate, worst_gate_pass_rate), validation_status, simulation (n_runs/seed/distribution_default), primary_failed_gates, primary_uncertainty_drivers, known_unmodelled_existential_gates (flat list of ids the extractor flagged as existential but unmodelable; empty when none), assessment_scope_warning (string when the unmodelled list is non-empty, null when empty — a one-line warning to programmatic consumers that the simulation is partial), do_not_treat_as, schema_notes (allowed enums for overall_risk_band, verdict, basis, threshold_basis, plus the primary_model_result_semantics disclaimer). The basis_enum is intentionally wider than what the current pipeline emits (report_derived, model_assumption) — it reserves report_explicit, report_inferred, external_reference, manual_override, unknown for future provenance types. JSON, not YAML — that is intentional.
• ## Provenance map — table listing every intermediary file with its role and "open when" guidance. The first row points at extract_parameters_input.md, then parameters/bounds/calculations/scenarios/scenario_outputs/montecarlo_settings/montecarlo/validation.
• ## Modelling frame — the source plan's own statement of what the model is testing, lifted verbatim from parameters.plan_summary.modelling_frame. When parameters.unmodelled_gates is non-empty, this section gains a bold Note caveat naming the count of unmodelled existential gates and telling the reader the gate-verdict pass rates are conditional on those gates holding.
• ## Known unmodelled existential gates — table of gates the extractor flagged as existential to the plan but unmodelable by deterministic Python (legal authorization, political reversal, AML/banking compliance, external-actor commitments). Columns: Gate, Why it matters, Source anchor (which source section names it), Consequence if false. Section omitted entirely when parameters.unmodelled_gates is empty or absent.
• ## Simulation settings — n_runs, seed, distribution_default, validation status.
• ## Critical findings — bullets in severity order. When parameters.unmodelled_gates is non-empty, a SCOPE WARNING bullet leads the section naming the unmodelled gate labels and pointing at the dedicated section. Then: Critical gates, Fragile gates, scenario warnings, numbers the model could not compute (≥5% blank runs), still-missing inputs. Section omitted entirely when nothing qualifies.
• ## Gate verdicts — every declared threshold, worst-first, with the min marker on aggregate gates. Columns: marker, output, condition, threshold basis (report_explicit / report_inferred / model_defined / unknown — derived from the corresponding key_value's value_type), pass rate, verdict, meaning. Includes an ### Aggregation warning sub-section when the thresholds use incompatible units and the plan declares no min() aggregate.
• ## Decision implications — one row per gate with verdict in Critical/Fragile/Marginal. Five columns: Gate, Verdict, Planning consequence (templated by verdict), Structural lever (the top driver from quartile_analysis with the direction implied by its sign of Δ-pp), Gate meaning (the gate's own rationale lifted from parameters.recommended_first_calculations[].why_first or derived_questions[].why_it_matters, plus the threshold parameter the formula tests against). The Gate-meaning column surfaces plan-specific framing without inventing tactical advice; concrete revisions should be derived by reading the source report and the relevant intermediary artifacts.
• ## Failure drivers — one row per failing gate (Critical or Fragile): top driver from quartile_analysis (max abs Δ-pp) and the conditional input restriction from required_input_thresholds that would lift the gate to 80%. Binding-gate frequencies for aggregates appear as bullets below the table.
• ## Missing inputs ranked by impact — the missing_value_priority table. The Basis column translates the bounds.json source label (data → report_derived, assumption → model_assumption) so it isn't mistaken for empirically observed real-world data.
• ## Confidence and trust boundaries — Validated (a one-line list of validation.json checks_performed), Not validated (a canonical list: real-world accuracy of bounds, independence assumptions, external feasibility, factual truth of source claims), Per-output confidence (HIGH/MEDIUM/LOW grade table from model_confidence). The grade-table column is Declared-source inputs — the share of input bounds anchored in the source report's narrative; the rest are modelling assumptions. Neither is empirical real-world data.
• ## Scenario sanity check — short low/base/high deterministic comparison table. Columns: Low inputs / Base inputs / High inputs, matching the keys in scenarios.json.
• ## Suggested next actions — five imperatives for whatever consumes this file next. Phrased as "To answer X, lead with Y; to audit Z, open W".
• ## Open questions for next analysis pass — five standing audit questions the simulation can't answer on its own (bound width/bias, gate independence, hard vs soft gates, missing-input remediation, unmodelled gates).

Common Mistakes

Mistake	Fix
Running before montecarlo.json exists	Gate verdicts, failure drivers, and the machine summary's primary_model_result all depend on simulation output. Run the Monte Carlo stage first.
Reading the markdown and paraphrasing the gate verdicts	Quote them. The cutoff bands and phrasing are deliberate.
Treating the machine summary as authoritative without reading the prose	The JSON manifest is a compact pointer, not a proof. The aggregation warning, trust boundaries, and failure-driver rows are load-bearing context.
Treating a Marginal verdict as good news	Marginal means "passes in 50–80% of runs" — that's the same as "fails up to 50% of the time".
Inventing a threshold to make a number look good	Thresholds reflect the user's success criteria. Don't fabricate them after the fact.

Reference

• Script (authoritative): experiments/napkin_math/summarize_assessment.py
• Companion skills: ../monte-carlo/SKILL.md, ../run-scenarios/SKILL.md, ../generate-bounds/SKILL.md, ../extract-parameters-from-full/SKILL.md, ../extract-parameters-from-digest/SKILL.md
• Example output: any assessment.md under experiments/napkin_math/output/<version>/<plan>/