benchmark-lightflows

# Benchmarking Lightflow Workflows Use `benchmarks/run_benchmark.py` to measure and verify **Arm A (`Lightflow DAG`)** against **Arm B (`Steelmanned Traditional Skill`: `benchmarks/baseline_skills/<name>/SKILL.md` + `examples/<name>/actions.py`)** across all 7 domain workflow pairs (`hn_digest`, `usgs_seismic_alert`, `pypi_upgrade_guard`, `async_job_watcher`, `blue_green_release`, `incident_db_failover`, `tenant_gitops_onboarding`) plus the `create_lightflow` authoring workflow. -------------------------------------------------------------------------------- ## Commands ```bash # Run the unified A/B benchmark (Markdown report) python3 benchmarks/run_benchmark.py # Run N=5 independent trials to verify multi-run consistency python3 benchmarks/run_benchmark.py --trials=5 # Emit machine-readable JSON metrics python3 benchmarks/run_benchmark.py --trials=5 --json ``` -------------------------------------------------------------------------------- ## What Is Verified on Every Run 1. **Ground-Truth Correctness & Invariants**: - `order_match`: Completed stages in `passport.json` match the exact expected topological sequence. - `no_duplicate_side_effects`: Every stage completes at most once (`1x`), including across mid-run failures and `lightflow resume` recoveries (`pypi_upgrade_guard`, `blue_green_release`, `incident_db_failover`). - `gate_discipline`: Every `operator_action` stage records a `PAUSED` stamp (`exit 2`) before its `COMPLETED` stamp. - `external_state_valid` & `baseline_external_state_valid`: Both Arm A and Arm B produce the exact expected on-disk artifacts, intermediate rollbacks, and cleanups. 2. **Context & Code Generation Cost (`1 token ≈ 4 chars`)**: - **Skill Loaded**: Universal `skills/run_lightflows/SKILL.md` (`~1,097 tok` once per session) vs. per-workflow `benchmarks/baseline_skills/<name>/SKILL.md` (`~7,075 tok` across the 7 domain workflows). - **Agent Command Output (`cmd`)**: Compact declarative `lightflow start / resume` CLI calls (`~795 tok` across 7 workflows) vs. inline `python3 -c` orchestration snippets (`~2,992 tok` across 7 workflows). - **Out-of-Context State**: `passport.json` bytes persisted on disk (`52,958 chars` / `~13,240 tok`) and `lightflow.yaml + actions.py` source bytes avoided (`115,321 chars` / `~28,830 tok`, **83% warm zero-context savings**).