swmm-end-to-end

name	swmm-end-to-end
description	Top-level orchestration skill for agentic SWMM modelling. Use when an agent needs one entrypoint that decides which module tools to run, in what order, and when to stop, for example to build, run, QA, and optionally calibrate a SWMM case from prepared or partially prepared inputs.

SWMM End-to-End Orchestration

Part of Agentic SWMM — install the project first for the executable toolchain (aiswmm CLI, SWMM solver, MCP servers).

What this skill provides

• A top-level orchestration contract for the agent runtime.
• A stable handoff point for Agentic AI project memory in agent/memory/.
• A deterministic execution order across the existing module skills:
  • swmm-anywhere (entry skill for data-scarce regions — no real pipe data)
  • swmm-gis
  • swmm-climate
  • swmm-params
  • swmm-network
  • swmm-builder
  • swmm-runner
  • swmm-design-review
  • swmm-plot
  • swmm-calibration
  • swmm-uncertainty
  • swmm-lid-optimization
  • swmm-experiment-audit

Routing rule — real-data path vs synth-data path

The orchestrator MUST inspect the user's inputs before choosing the entry skill:

• If the request includes any of .shp, .csv, network.json, a CAD file, or an existing .inp path → the user has real data; route to swmm-network (or swmm-builder if the INP is already prepared).
• If the request includes only a bbox or a location name with no pipe-network file attached → the user is in data-scarce mode; route to swmm-anywhere, which produces a synth .inp that downstream skills (swmm-runner, swmm-experiment-audit, swmm-plot) then consume identically to a real-data INP.
• If both signals appear (bbox and a SHP) → prefer the real-data path (swmm-network); only fall back to swmm-anywhere if the user explicitly asks for a synth baseline for comparison.
• Clear stop conditions so the agent does not pretend a full model was built when critical inputs are still missing.
• A minimal real-data fallback path for Tod Creek via scripts/real_cases/run_todcreek_minimal.py.
• A mandatory audit handoff that consolidates artifacts, metrics, QA, comparison records, and default Obsidian audit notes after success or failure.

When to use this skill

Use this skill when the user asks for:

• one agent-facing entrypoint for SWMM modelling,
• end-to-end build + run + QA,
• an agent to decide which SWMM module comes next,
• a real-data dry run before full automation is ready, or
• a bounded orchestration layer without rewriting the underlying scripts.

Do not use this skill when the user clearly wants only one module in isolation, such as only rainfall formatting or only calibration metrics.

Recommended public memory preload

Before using this skill in Codex, OpenClaw, Hermes, or another compatible runtime, load the Markdown files in agent/memory/:

1. identification_memory.md
2. soul.md
3. operational_memory.md
4. modeling_workflow_memory.md
5. evidence_memory.md
6. user_bridge_memory.md

Those files define the public project identity, agent posture, evidence boundaries, and first-run user behavior. This skill remains the execution contract; the memory files should shape decisions and communication, not replace tool calls or depend on the maintainer's private local workspace.

Supported operating modes

Mode 0: MCP-first framework smoke test mode

Use this when the user is testing the Agentic SWMM framework itself, especially with prompts like "test the end-to-end framework", "test skill and MCP tool calls", "from raw data to INP", or "automatic modeling smoke test".

This mode tests orchestration behavior before scientific model quality. It must:

• trigger this swmm-end-to-end skill first;
• choose the smallest real-data subset that can exercise the chain;
• call the relevant MCP tool contracts as the primary path;
• Do not bypass MCP tool contracts by calling the underlying Python scripts as the primary path;
• use temporary script edits only for missing framework adapters that do not yet have MCP coverage;
• record every temporary script fallback as missing framework capability, not as a completed MCP feature.

The run manifest for this mode must include:

• tool_transport: mcp when the MCP server was called through the protocol, script_fallback when the tool contract exists but the MCP transport was bypassed, or temporary_script when no MCP contract exists yet;
• mcp_tool_calls: ordered records of the intended or actual tool calls, including server, tool name, input paths, output paths, and status;
• missing_or_fallback_inputs: explicit data gaps and assumptions such as missing soil, nonnumeric pipe diameter fallback, inferred outfall, inferred invert elevation, or shortened rainfall window;
• framework_gaps: MCP/skill gaps discovered during the run that should be implemented later.

Use scripts/mcp_stdio_call.py when Codex needs to verify the MCP transport directly from this repository. The helper initializes a server over stdio, checks tools/list, calls one tool with JSON arguments, and stores the raw MCP response as an artifact. Use repo-root relative paths in the JSON arguments; the helper resolves those to absolute paths before sending the tool call so server-local working directories do not corrupt path resolution.

For a raw-data to INP smoke test, prefer this MCP call order when inputs are present:

1. swmm-gis-mcp.qgis_area_weighted_params for land-use/soil area-weighted params, or record a missing-input fallback if soil is absent.
2. swmm-climate-mcp.format_rainfall for event rainfall.
3. swmm-climate-mcp.build_raingage_section when an explicit raingage artifact is needed. 4a. swmm-network-mcp.prepare_storm_inputs for raw municipal shapefiles (clip pipes + manholes to basin, fill mapping from skills/swmm-network/templates/city_mapping_raw_shapefile.template.json). Skip when the source is already a structured CAD export with explicit from/to nodes. 4b. swmm-network-mcp.snap_pipe_endpoints — heal sub-millimetre to centimetre vertex drift between adjacent pipe segments. Reasonable starting tolerance is 0.5–3 m. Reports clusters merged + any pipes dropped as self-loops. 4c. swmm-network-mcp.infer_outfall (mode endpoint_nearest_watercourse when a watercourse layer is available; else lowest_endpoint). 4d. swmm-network-mcp.reorient_pipes to BFS-flip flow direction. 4e. swmm-network-mcp.import_city_network to assemble network.json.
4. swmm-network-mcp.qa. With the 4a–4d steps above, no_outfall_path warnings should be limited to genuinely disconnected sub-graphs (e.g. trunk pipes that fall outside the basin clip). 5b. swmm-network-mcp.assign_subcatchment_outlets — REQUIRED when the subcatchments came out of basin_shp_to_subcatchments (which seeds every subcatchment's outlet to the literal outfall). This step rewrites each subcatchment's outlet to a real upstream junction (mode=nearest_junction reads network.json's junctions list; or mode=manual_lookup for an explicit mapping). Without it the pipe network is in the .inp but receives no surface runoff. Pass the rewritten CSV (not the original) to build_inp in step 6.
5. swmm-builder-mcp.build_inp.
6. swmm-runner-mcp.swmm_run.
7. swmm-runner-mcp.swmm_continuity.
8. swmm-runner-mcp.swmm_peak.

Stop and report if a required MCP contract is absent. Continue with a temporary script only when the user explicitly asked to test the framework and the run manifest records the gap under framework_gaps.

Mode A: Full modular build

Use this when the required explicit inputs already exist or can be produced safely:

• subcatchment polygons or builder-ready subcatchments.csv
• network input that can become network.json
• land use and soil inputs
• rainfall input

Execution order:

1. swmm-gis
2. swmm-params
3. swmm-climate
4. swmm-network
5. swmm-builder
6. swmm-runner
7. QA checks
8. swmm-design-review (reference-free plausibility review — no observed data needed)
9. optional swmm-plot
10. optional swmm-calibration
11. optional swmm-uncertainty
12. optional swmm-lid-optimization
13. swmm-experiment-audit

Exact MCP call chain for the full modular path. Two parallel branches (GIS / hydrology vs network) merge at build_inp. Region-agnostic: substitute the user's basin shapefile, pipe shapefile, watercourse shapefile, landuse layer, soil layer, and rainfall input wherever the inputs are referenced.

GIS / hydrology branch (subcatchments + params)

1. swmm-gis-mcp.basin_shp_to_subcatchments — basin shp → subcatchments.geojson + .csv. (Or gis_preprocess_subcatchments if a pre-attributed subcatchment shp + DEM exist.)
2. swmm-gis-mcp.qgis_area_weighted_params — intersect subcatchments × landuse × soil → weighted_params.json. Inspect the warnings array for any landuse classes that fell through to DEFAULT.
3. (alternative when a richer params chain is wanted) swmm-params-mcp.map_landuse → map_soil → merge_params.

Climate branch (rainfall) 4. swmm-climate-mcp.format_rainfall — rainfall CSV (or .dat via inputDatPaths) → rainfall.json + timeseries.txt. 5. swmm-climate-mcp.build_raingage_section (only when an explicit raingage artefact is needed).

Network branch (pipe topology) 6. swmm-network-mcp.prepare_storm_inputs — clip pipes (+optional manholes) shp to the basin, fill mapping.json from templates/city_mapping_raw_shapefile.template.json. (Skip when the source is already a structured CAD export with explicit from/to nodes.) 7. swmm-network-mcp.snap_pipe_endpoints — heal sub-millimetre vertex drift between pipe segments so import_city_network can infer connected junctions. (BACKLOG.md B8; skip pre-B8.) 8. swmm-network-mcp.infer_outfall — pick a single outfall point (mode endpoint_nearest_watercourse when a watercourse shp is available, else lowest_endpoint). 9. swmm-network-mcp.reorient_pipes — BFS-from-outfall flow-direction fix. 10. swmm-network-mcp.import_city_network — assemble network.json from the prepared geojsons + mapping. 11. swmm-network-mcp.qa — topology + required-attribute QA.

Wiring + assembly 12. swmm-network-mcp.assign_subcatchment_outlets — REQUIRED if subcatchments came from basin_shp_to_subcatchments (which writes outlet=OUT1 as a placeholder). Rewrites the CSV so each subcatchment drains into a real upstream junction; without this the pipe network sits idle. 13. swmm-builder-mcp.build_inp — assemble model.inp + builder manifest.

Run + QA + plot 14. swmm-runner-mcp.swmm_run — omit the node arg to auto-detect the first [OUTFALLS] entry from the .inp. 15. swmm-runner-mcp.swmm_continuity 16. swmm-runner-mcp.swmm_peak — pass the actual outfall name explicitly (no longer defaults to O1). 16b. aiswmm review --run-dir runs/<case> --rules skills/swmm-design-review/rulebooks/synth_plausibility.yaml — reference-free plausibility review (no observed data); writes 11_review/design_review.json + .md. Direct review_run tool / CLI verb, not an MCP server. Use the synth_plausibility rulebook for swmm-anywhere runs; use gb50014_template (or a local rulebook) for real-data design checks. 17. optional swmm-plot-mcp.plot_rain_runoff_si — paired rain + flow figure for any node. 14. optional calibration tools:

• swmm-calibration-mcp.swmm_sensitivity_scan
• swmm-calibration-mcp.swmm_calibrate
• swmm-calibration-mcp.swmm_calibrate_search
• swmm-calibration-mcp.swmm_calibrate_sceua
• swmm-calibration-mcp.swmm_calibrate_dream_zs
• swmm-calibration-mcp.swmm_validate

15. optional uncertainty / sensitivity analysis tools (use instead of or in addition to calibration):

• swmm-uncertainty-mcp.swmm_sensitivity_oat — one-at-a-time sensitivity (parameter ranking)
• swmm-uncertainty-mcp.swmm_sensitivity_morris — Morris elementary-effects screening
• swmm-uncertainty-mcp.swmm_sensitivity_sobol — Sobol' first-order + total-effect indices
• swmm-uncertainty-mcp.swmm_uncertainty_source_decomposition — integrate raw ensemble outputs
• uncertainty propagation (hydrograph envelopes, not parameter ranking) has no MCP tool; use the scripts:
  • python3 skills/swmm-uncertainty/scripts/uncertainty_propagate.py ... (fuzzy alpha-cut)
  • python3 skills/swmm-uncertainty/scripts/monte_carlo_propagate.py ... (Monte Carlo)

16. optional LID scripts:

• python3 skills/swmm-lid-optimization/scripts/entropy_lid_priority.py ...
• python3 skills/swmm-lid-optimization/scripts/lid_scenario_builder.py ...

17. swmm-experiment-audit via CLI:

• aiswmm audit --run-dir runs/<case> — canonical path; adds backup/MOC/memory-hook. Pass --obsidian to also copy the note to the local Obsidian vault.
• python3 skills/swmm-experiment-audit/scripts/audit_run.py --run-dir runs/<case> — direct script path; Obsidian export is on by default (suppress with --no-obsidian).

Mode B: Prepared-input build

Use this when subcatchments.csv, network.json, params JSON, and rainfall references already exist.

Execution order:

1. swmm-builder
2. swmm-runner
3. QA checks
4. swmm-design-review (reference-free plausibility review)
5. optional plotting / calibration
6. optional uncertainty / LID scenario analysis
7. swmm-experiment-audit

Exact MCP call chain for prepared inputs:

1. swmm-builder-mcp.build_inp
2. swmm-runner-mcp.swmm_run
3. swmm-runner-mcp.swmm_continuity
4. swmm-runner-mcp.swmm_peak 4b. aiswmm review --run-dir runs/<case> [--rules <rulebook.yaml>] — reference-free plausibility review; writes 11_review/. Defaults to the gb50014_template rulebook; pass --rules skills/swmm-design-review/rulebooks/synth_plausibility.yaml for synthesized networks.
5. optional swmm-plot-mcp.plot_rain_runoff_si
6. optional calibration tools
7. optional uncertainty / LID scripts where a runnable base INP exists
8. swmm-experiment-audit via CLI:
   • aiswmm audit --run-dir runs/<case> — canonical path; adds backup/MOC/memory-hook. Pass --obsidian to also copy the note to the local Obsidian vault.
   • python3 skills/swmm-experiment-audit/scripts/audit_run.py --run-dir runs/<case> — direct script path; Obsidian export is on by default.

Mode C: Minimal real-data Tod Creek fallback

Use this only when the user wants a real-data run but the full modular path is not ready because there is no trustworthy multi-subcatchment + network input yet.

Script:

• scripts/real_cases/run_todcreek_minimal.py

Audit command after the script returns:

• aiswmm audit --run-dir runs/real-todcreek-minimal --workflow-mode "minimal real-data fallback" — canonical path with backup/MOC/memory-hook.

Characteristics:

• one subcatchment
• simple junction/outfall/conduit layout
• real DEM / land use / soil / rainfall inputs
• useful as a real-data smoke test, not as the final watershed architecture

This fallback is a script path, not a module-MCP path. The agent should choose it only when:

• the user explicitly accepts a simplified real-data smoke test, or
• full modular inputs are incomplete and the goal is to verify the repo can run with real data.

Required decisions before execution

The orchestrator should decide these items explicitly:

• Are we doing a full modular build or a minimal real-data fallback?
• Do we already have network.json or equivalent network source files?
• Do we already have subcatchment polygons or builder-ready subcatchment CSV?
• Is the user asking for:
  • build only,
  • build + run + QA, or
  • build + run + QA + calibration / uncertainty / LID scenarios?

If the answer is unclear, prefer:

• build + run + QA
• no calibration
• full modular build only when required inputs are real and explicit

Input completeness rules

For full modular build

The run should stop and report missing inputs if any of these are absent:

• network information that can be converted to network.json
• subcatchment geometry or builder-ready subcatchment table
• rainfall input
• land use / soil inputs or an accepted pre-merged params artifact

For minimal Tod Creek fallback

The run requires:

• data/Todcreek/Geolayer/n48_w124_1arc_v3_Clip_Projec1.tif
• data/Todcreek/Geolayer/landuse.shp
• data/Todcreek/Geolayer/soil.shp
• data/Todcreek/Rainfall/1984rain.dat
• data/Todcreek/outlet_candidate.geojson

Execution policy

• Prefer existing module scripts and MCP tools over ad hoc one-off code.
• Keep every stage artifact under runs/<case>/... or another explicit run directory.
• Preserve machine-readable JSON outputs where available.
• Fail fast on missing critical inputs.
• Do not silently invent a drainage network for a supposed full build.
• If full modular inputs are incomplete but Tod Creek real-data fallback is available, say so explicitly and switch only if that matches the user’s intent.
• Always call swmm-experiment-audit after the attempt, even when the run fails or stops early. The audit record should reflect partial evidence rather than invent missing outputs.

Artifact handoff contract

The top-level skill should pass artifacts between MCP tools using explicit run-local paths.

Recommended stage layout:

• runs/<case>/01_gis/subcatchments.csv
• runs/<case>/01_gis/subcatchments.json
• runs/<case>/02_params/landuse.json
• runs/<case>/02_params/soil.json
• runs/<case>/02_params/merged_params.json
• runs/<case>/03_climate/rainfall.json
• runs/<case>/03_climate/timeseries.txt
• runs/<case>/03_climate/raingage.json
• runs/<case>/03_climate/raingage.txt
• runs/<case>/04_network/network.json
• runs/<case>/04_network/network_qa.json
• runs/<case>/05_builder/model.inp
• runs/<case>/05_builder/manifest.json
• runs/<case>/06_runner/model.rpt
• runs/<case>/06_runner/model.out
• runs/<case>/06_runner/manifest.json
• runs/<case>/07_qa/continuity.json
• runs/<case>/07_qa/peak.json
• optional runs/<case>/08_plot/...
• optional runs/<case>/09_calibration/...
• optional runs/<case>/09_uncertainty/...
• optional runs/<case>/09_lid/...
• runs/<case>/09_audit/experiment_provenance.json
• runs/<case>/09_audit/comparison.json
• runs/<case>/09_audit/experiment_note.md

Preflight

Before running any operating mode, every MCP server under mcp/<server>/ must have its node_modules installed. node_modules/ is .gitignored, so a fresh clone (or any server added later) needs an install step:

scripts/install_mcp_deps.sh                  # install for all mcp/*/ servers
scripts/install_mcp_deps.sh swmm-calibration # install for one server

The script loops over every mcp/*/package.json and runs npm install. Exit code is non-zero if any install fails. Servers that fail to install will not respond to tools/list, so a cold-start agent that skips this step will see ambiguous "MCP server exited before response" errors deep into Mode 0 instead of a clean install failure up front.

After install, verify with a tools/list probe per server, for example:

python3 skills/swmm-end-to-end/scripts/mcp_stdio_call.py \
  --server-dir mcp/swmm-calibration --tool __probe__ \
  --arguments-json '{}' --out-response /tmp/_.json

The harness will print the Available tools: [...] for that server in its error output (a dedicated --list-tools flag is on the framework backlog).

MCP execution notes

GIS stage

• Use gis_preprocess_subcatchments only when a subcatchment polygon dataset already exists.
• Do not claim GIS preprocessing can replace watershed delineation or pipe-network generation.

Params stage

• map_landuse and map_soil should target the same subcatchment ID universe.
• merge_params should be treated as the single params handoff into build_inp.

Climate stage

• format_rainfall should create both JSON metadata and timeseries text.
• build_raingage_section should run after rainfall formatting so series_name or rainfall_json stays consistent.

Network stage

• Use import_network only when raw conduits/junctions/outfalls exist.
• If network.json already exists, skip import and run qa directly.
• If no trustworthy network source exists, stop the full modular path.

Builder stage

• build_inp is the only handoff into swmm_run.
• Treat builder validation failures as hard stops for the full modular path.

Runner and QA stage

• swmm_run creates the canonical manifest.json.
• swmm_continuity and swmm_peak are mandatory QA steps, not optional metrics.
• Prefer the fixed swmm_peak parser path that reads the correct summary block.

Calibration stage

• Do not enter calibration unless the user requested it or the workflow explicitly includes it.
• Require an observed flow file before any calibration tool is called.

Audit stage

• Run swmm-experiment-audit after success, failure, or early stop.
• Use the run directory as the single audit input.
• Pass --compare-to <baseline-run-dir> when the user requests baseline/scenario or before/after comparison.
• The audit must write 09_audit/experiment_provenance.json, 09_audit/comparison.json, and Obsidian-compatible 09_audit/experiment_note.md inside the run directory.
• The audit should also use the default Obsidian export unless the user explicitly asks for --no-obsidian.
• The default Obsidian vault is ~/Documents/Agentic-SWMM-Obsidian-Vault, with 10_Memory_Layer for durable lessons and 20_Audit_Layer for run-level evidence.
• Do not include chat transcripts or conversational content in audit outputs.

Agent prompt-level instruction

When an agent uses this skill, it should:

• choose one operating mode first,
• announce the chosen mode,
• create a case run directory,
• call only the MCP tools required for that mode,
• stop immediately on missing critical inputs instead of hallucinating replacements,
• call swmm-experiment-audit on the run directory,
• summarize which concrete artifacts and audit records were produced.

QA gates

Minimum QA checks for a successful run:

• builder validation has no critical missing sections
• SWMM return code is zero
• .rpt and .out exist
• continuity metrics can be parsed
• peak metric can be parsed from the correct summary block

For calibration mode, also require:

• observed flow file parses successfully
• time overlap between observed and simulated series is adequate

Output contract

At minimum, the orchestrator should leave behind:

• built .inp
• run .rpt
• run .out
• manifest.json
• a short machine-readable QA summary
• 11_review/design_review.json + 11_review/design_review.md (reference-free plausibility review)
• 09_audit/experiment_provenance.json
• 09_audit/comparison.json
• Obsidian-compatible 09_audit/experiment_note.md

If plotting is requested, also produce:

• rainfall-runoff figure artifact

If calibration is requested, also produce:

• ranking / summary JSON
• chosen parameter set or best-params output

Recommended agent behavior

• Use this skill as the only top-level SWMM skill.
• Treat module skills as subordinate implementation skills.
• Keep reasoning at the orchestration layer and calculations at the script layer.
• When a run fails, report the failing stage and the missing or invalid input rather than guessing.

Known limitations

• This skill does not remove the need for swmm-network; it only coordinates it.
• Full watershed automation still depends on real subcatchment + network preparation.
• The Tod Creek real-data fallback is intentionally simplified and should not be confused with the final multi-subcatchment production workflow.