rl-daily-dsm

name

rl-daily-dsm

description

LIVE EnergyPlus daily six-zone RL on Lakeside A04. rleplus Gym/runner backend. SB3 PPO/DQN. One step = one weather day; one EnergyPlus process per multi-day episode (EnergyPlusContinuityPlant). Reward contract v2. No Ray, no Amphitheater IDF. Long campaign forbidden until a physics champion exists. Without a champion, only labeled A04 research-poc / research-long are allowed (cannot set long_campaign_allowed). Use when training, evaluating, launching research-long, interpreting campaign manifests, or deciding Terminal A/B/C readiness.

RL daily DSM (A04 + rleplus)

Claim: ENERGYPLUS DSM OPTIMIZATION SCREENING / RETROSPECTIVE REPLAY.

Do not start a 20–30 hour PPO/DQN campaign while the model is not transient-validated. A04-v2 produced no champion (MODEL_DEVELOPMENT_INCOMPLETE_NO_CHAMPION). Long RL remains blocked. hp67 child models also failed champion gates — keep RL on A04 parent with Terminal B labels until a future child passes full physics gates.

Hard boundaries (fail closed)

• Campaigns must set live_energyplus=true via EnergyPlusContinuityPlant. FakeContinuityPlant is a bookkeeping test double only.
• BACnet command authority = 0. Never invent command writes.
• Vibe19 / public Streamlit is out of scope unless the human explicitly asks to touch it.
• Never raise ENGINEERING_MARGIN or soften W2A scored-runtime gates to pass.
• Never treat training mean reward as a validation leader.
• Never compare absolute $ across FLAT_PLUS_DEMAND and ILLUSTRATIVE_TOU_PLUS_DEMAND.
• Never say “17/17 ready” for school readiness. Non-school days auto-pass the readiness gate and are not school readiness success.

Action / observation / reward (current research stack)

Use contracts/reward_contract_v2.json and eplus_gym/rl/reward_v2.py for multi-day work. Do not reinterpret published operator_pay_2x_v1 smoke artifacts.

Piece	Locked value
Action contract for finished research-long	research_action_contract_v3 (Box10 + DQN table; post_occupancy_extension_minutes 0–180)
School occupancy	Immutable calendar; extension must not invent holiday occupancy
Cooling	Approximately fixed (~74/85); cooling_action_space=false
Observation	obs_schema v4, dim 206 for finished campaigns
Readiness	Steps 30–31, band 68–74°F, all six zones, checked school days only
Recovery	recovery_lead_minutes is the linear ramp duration ending at DualSP start
PPO continuous	setback_depth < 0.25°F → CONTINUOUS_CONDITIONING_THERMOSTATIC
DQN	Unique post-clamp schedules; indices do not wrap
Policy artifact	SB3 .zip is canonical; do not write a v2/dim-19 daily_policy.pkl

recovery_lead_minutes values 60/120/180 must differ in schedule proof. Schedule proof is required on every step for action-contract v3.

Baseline contract (do not retcon)

Finished research-long campaigns used observed_bas_incumbent_v2:

• Heating ≈ 68°F occupied / 64°F unoccupied, scheduled DualSP transitions
• Cooling ≈ 74°F occupied / 85°F unoccupied
• Not A04 native SCH_HtgSP; not Gym DualSP 70/65

Field note (unresolved): reported actual BAS may use continuous 68/74. That conflict was not resolved before the campaigns. Do not relabel historical baselines or claim modeled deltas as verified savings vs actual BAS.

Contract: contracts/observed_bas_incumbent_v2.json. Terminology: contracts/baseline_terminology_v1.json. Verified BAS status: contracts/verified_bas_incumbent_v1.json (UNRESOLVED).

Forbidden cross-experiment claims

Never claim “DQN/grid reduced 285 kW to 211/220 kW”. January ~285 kW is utility / A04 native calibration weather+schedule; December ~200–234 kW peaks used observed_bas_incumbent_v2 on Dec 15–31. Use eplus_gym/rl/paired_comparison.py fail-closed gates. Jan 26 bridge: scripts/vibe22_cold_day_bridge.py (diagnostic, not holdout).

Physics / Track status (scientific record)

W2A quality gates on scored-runtime only: runtime = total - warmup - sizing.

Track B two-pass LIVE EnergyPlus ran. Superseded two-pass tree scored 3,780. Later LIVE matrix (37 reports) scored 2,106 / 5,332 warmup on the first child — still not a champion. CLI instrumented day: 738 scored / 4,657 warmup plus active invalid-domain 759. A04 heating capacity in the eio is User-Specified 149430 W/zone; airflow is Design Size. EnergyPlus 26.1 eio names are uppercase.

Track C sequential (not a matrix): C1 ~603 kW scored W2A 822; C2 hard-size 800 kW scored 2016. C3 skipped (unverified A04 curves). Do not promote a topology merely because warnings drop.

Frozen ramp stays 2.651 °F / 15 min. A04 remains immutable.

Research CLI (Terminal B)

research-poc and research-long are separate subcommands (not aliases of campaign / operator-pay --mode).

• Missing confirm flags → exit 4
• Labels: A04_RESEARCH_POC_NOT_TRANSIENT_VALIDATED, RESEARCH_LONG_ALLOWED
• SIMULATION_TRAINING_READY and OPERATIONAL_DSM_READY stay false
• long_campaign_allowed stays false (research contract cannot set true)
• Future agents: inspect IDF/RDD with EnergyPlus MCP first; MCP cannot rewrite W2A banks

EnergyPlusContinuityPlant.reset() consumes schedule index 0; remaining lookback uses indices 1..95. A04 3-day continuity gallery: n_process_starts==1 per arm.

Future vibe22_rl.py campaign constructs MultiDayDailyEnv, not DailySixZoneGymEnv (legacy diagnostic only).

Finished research-long campaigns (2026-08-20)

Authoritative SITE_ROOT runs (read-only; never rewrite manifests):

Experiment	Run root under $SITE_ROOT/reports/eplus_gym/rl/
PRIMARY FLAT_PLUS_DEMAND	research_long_flat_plus_demand_20260820T132506Z
SECONDARY ILLUSTRATIVE_TOU_PLUS_DEMAND	research_long_illustrative_tou_plus_demand_20260820T210304Z

Scale (do not conflate counters):

• Train: 44 days (2025-11-01..2025-12-14)
• Validation: 17 days (2025-12-15..2025-12-31)
• Checked school days in validation: 5 (Dec 15–19); unchecked: 12
• target_transitions 8192 × PPO/DQN seeds 0–1
• Eval: 11 arms × 17 days = 187 rows
• actual_energyplus_process_launches: null in manifests — publish “not recorded”; do not invent

Publication language:

• Say validation leader in markdown/figures/slides
• Keep raw eval field name winner only inside JSON provenance
• Leaders are readiness-constrained deterministic costs — not training mean reward

Headline outcomes (modeled; simulation-only):

• PRIMARY validation leader trained_ppo_seed0 ≈ +$5.26 vs incumbent, higher peak — did not reduce peak or total cost
• SECONDARY validation leader trained_dqn_seed1 ≈ −$63.23 illustrative; energy down, demand/peak up; TOU not verified utility pricing
• Incumbent checked-school readiness was 0/5 on PRIMARY (auto-pass non-school must not be spun as ready)

December billing floor: all Dec 15 rows had opening_mtd_kw = 0.0. Disclose, do not re-score (no arm-specific Dec 1–14 facility series for offline MTD repair without re-running EnergyPlus).

Published pack: docs/results/ via scripts/vibe22_publish_rl_poc_results.py. Exhaustive discrete DQN-table LIVE screen completed as fixed-policy grid comparator — see docs/results/grid_search/ and skill grid-search-comparator. Two-month replay: docs/results/two_month_policy_replay/ + skill two-month-policy-replay. Nightly identical-state compute: docs/results/nightly_grid_compute/ + skill nightly-grid-compute (verdict NIGHTLY_GRID_FEASIBLE_WITHIN_15_MIN on test hardware).

Commands

python scripts/a04_live_multiday_continuity.py --site-root $env:SITE_ROOT
python scripts/a04v2_trackb_two_pass.py --site-root $env:SITE_ROOT
python scripts/vibe22_rl.py operator-pay-experiment --mode smoke --reward-name operator_pay_2x_v1 --run-id oppay2x_smoke_20260816 --site-root $env:SITE_ROOT
python scripts/vibe22_rl.py research-poc --confirm-simulation-only-physics-limits --max-wall-hours 6 --site-root $env:SITE_ROOT
python scripts/vibe22_rl.py research-long --confirm-simulation-only-physics-limits --confirm-a04-not-transient-validated --micro-gate --site-root $env:SITE_ROOT
python scripts/vibe22_rl.py research-long --confirm-simulation-only-physics-limits --confirm-a04-not-transient-validated --obs-schema v4 --tariff-mode FLAT_PLUS_DEMAND --action-contract research_action_contract_v3 --execute-live --heartbeat $env:SITE_ROOT/reports/eplus_gym/rl/research_long_flat_plus_demand_heartbeat.json --site-root $env:SITE_ROOT
python scripts/vibe22_publish_rl_poc_results.py --site-root $env:SITE_ROOT
py -3.12 scripts/vibe22_two_month_policy_replay.py --site-root $env:SITE_ROOT --resume
py -3.12 scripts/vibe22_nightly_grid_compute.py --site-root $env:SITE_ROOT --stage all --resume

Long campaign --n-days 100 is prohibited.

Mega v3 program notes

Do not treat scaffold JSON or unit tests as completed experiment phases. Phases that require live EnergyPlus or training stay pending until real evidence exists. hp67 v2 champion failed. Three-day pilot passed 2026-08-19 and authorized research-long on A04 fallback (Terminal B).

Audits:

• docs/audits/2026-08-18-vibe22-final-physics-and-rl-poc.md
• docs/audits/2026-08-18-vibe22-research-long-launch.md
• docs/audits/2026-08-19-vibe22-launch-readiness-second-research-long.md
• docs/audits/2026-08-20-vibe22-action-space-tariff-experiments.md
• docs/results/vibe22_rl_poc_results.md (finished PoC pack)

../../vibe22_agent_spec/CONTRIBUTING_RL.md · ../../vibe22_agent_spec/MEGA_V3_PHASES.md · ../../vibe22_agent_spec/RESULTS_PUBLICATION.md · ../../vibe22_agent_spec/TWO_MONTH_POLICY_REPLAY.md · ../../vibe22_agent_spec/NIGHTLY_GRID_COMPUTE.md

Honor NO_PRISTINE_LOCKED_TEST_AVAILABLE.