| name | plenum-research-methodology |
| description | The discipline that turns a hunch into an accepted result in the Plenum repo — the evidence bar (one mechanism must explain every observation, hypotheses must predict numbers before you measure, adversarial refutation), the hunch→issue→sandbox-experiment→PR→docs lifecycle, how to run a systematic audit like the #280–#305 wave, experiment hygiene (both-directions tests, timezone matrix, restart-mid-state, simulated-vs-real HVAC), and a result-acceptance checklist. Load when you have a theory about a bug or a design idea and need to know what proof this project demands before it ships. |
Plenum research methodology: hunch → accepted result
This skill is the process discipline. It does not tell you what to research
(load plenum-research-frontier), how to compute a specific analysis (load
plenum-proof-and-analysis-toolkit), how to gate a change through review
(load plenum-change-control), or how to pull numbers out of a running
system (load plenum-diagnostics-and-tooling). Load it when you are about
to claim "I know why X happens" or "we should build Y" and want that claim to
survive review in this repo.
When NOT to use this skill
- You need a symptom→cause triage table right now → plenum-debugging-playbook.
- You want the specific SQL/log/metric to measure something → plenum-diagnostics-and-tooling.
- You want analysis math with worked examples → plenum-proof-and-analysis-toolkit.
- You want what's worth researching next → plenum-research-frontier.
- You're past the result and shipping it → plenum-change-control, plenum-validation-and-qa.
- You want the history of a specific settled battle → plenum-failure-archaeology.
Jargon used below (see hvac-zoning-reference for full definitions):
deadband = tolerance band around a target temperature; overshoot =
conditioning past target; short-cycling = compressor starting/stopping too
fast; cycle = one engine-managed conditioning run; golden = committed
reference screenshot for visual regression.
1. The evidence bar
A root-cause claim is accepted here when it clears three tests. All three,
every time — the project's history shows what each one costs when skipped.
1a. One mechanism must explain ALL observations — including the negatives
A mechanism that explains the headline symptom but not the side observations
is at best half the story. Two canonical cases:
#281 — unit auto-detection had TWO independent root causes. Symptom:
"leave temperature_unit blank to auto-detect" always resolved to °F. Root
cause 1: ha_client.py compared HA's unit_system — which is an object
({"temperature": "°C", ...}) — against the string "metric", so detection
always returned "F". Root cause 2: run.sh defaulted a blank option to
'F' and exported it, which scheduler.py treats as a hard override lock
that disables the detection task entirely. Fixing only the string comparison
would have left detection dead on every HAOS install, because the run.sh lock
meant the fixed code never executed. The tell that one cause wasn't enough:
the fix for cause 1 predicts "detection now returns C on metric HA," but the
observation "the detection task never even starts on HAOS" remains
unexplained. Rule: after you find a cause, re-walk every observation and
ask "does my mechanism produce exactly this?" Any leftover observation means
a second cause, or the wrong cause. (Corollary from the same incident:
tests can mask a bug by mocking the wrong shape — test_ha_client_internals
mocked {"unit_system": "metric"}, a payload HA never sends. A passing test
is an observation too; ask what it actually exercised.)
#86 — the surface hunch was wrong. Symptom: schedule-triggered cycles ran
for hours after every room reached target. The surface hunch — "the schedule
keeps the rooms active, so the cycle can't end" — was plausible and even
partially true, but it doesn't explain the decisive observation: the engine's
own logs showed all_at_target flipping back to False after every vent had
closed. The actual mechanism was physical + logical: once a vent closes, the
room's temperature naturally drifts back across the target (post-closure
drift), and _monitor_rooms had no vent_closed_at guard on its
elif not at_target branch, so a served room re-blocked termination forever.
The timeline in the issue (vents closed 1:33 AM, cycle still running at
2:21 AM, engine=running in reconcile logs throughout) is what killed the
surface hunch — the schedule theory predicts a re-started cycle, not one
continuous never-terminating cycle. Rule: build the event timeline before
theorizing. The observation your hunch cannot reproduce is the one that
matters.
1b. Hypotheses must predict NUMBERS before you run anything
Vague predictions ("the value will be too high") are unfalsifiable — almost
any bug produces "too high." Compute the exact number your mechanism implies,
then measure:
- Double conversion (#231): if the frontend converts 16 °C → 60.8 °F and
the backend converts again treating 60.8 as °C, the DB must hold exactly
60.8 × 9/5 + 32 = 141.44 °F. Not "over 100" — 141.44. If you measure
140.9, your mechanism is wrong (or there's a second conversion/rounding
step you haven't found).
- Absolute-vs-delta confusion (#291): converting a 2 °F deadband with the
absolute formula predicts
(2−32) × 5/9 = −16.7 °C — a negative number for
a width. The delta formula predicts 2 × 5/9 = 1.1 °C. One measurement
discriminates the two instantly.
- Missing °C→°F normalization (#280/#182's 158 °F bug): a raw 21 °C read
as °F predicts readings ~48° low; a 70 °C setpoint sent raw predicts HA
clamping to the entity max (~35 °C → displayed 158 °F after correct
back-conversion:
70 × 9/5 + 32 = 158). The signature offsets — ×1.8,
+32, −32, ~48° — each fingerprint a distinct wrong code path (the
discrimination table lives in plenum-debugging-playbook).
- Timezone bugs (#301, #294, #26, #65): predict the shift exactly: a
naive-UTC timestamp rendered as local is off by your current UTC offset,
changing at DST. "Off by 5 hours in winter, 4 in summer, in this direction"
is a prediction; "times look wrong" is not.
Write the predicted number in the issue before running the measurement.
If the measurement matches to the decimal, you have the mechanism. If it's
merely "in the same direction," you don't — keep digging.
1c. Adversarial refutation — try to kill your own mechanism first
Before accepting a mechanism, construct the observation that would falsify
it, then go collect that observation. Concretely:
| If your mechanism is… | The falsifying observation to go look for |
|---|
| "frontend double-converts" | a °F-mode install showing the same corruption (double conversion is a no-op in °F — if °F is also wrong, the cause is elsewhere) |
| "schedule keeps the cycle alive" (#86's rejected hunch) | one continuous cycle ID across the whole window instead of repeated restarts |
| "the detection code has a parsing bug" (#281 cause 1 alone) | a debug log proving the detection task ran at all on HAOS (it didn't — cause 2) |
| "post-closure drift blocks termination" (#86 accepted) | a cycle where temps held exactly at target after closure and it still didn't terminate |
| "vents thrash because monitor re-closes them" (#237) | a hold period with zero open/close events in the event log |
Each row is cheap to check with the event log / cycle history / a SQL query
(recipes in plenum-diagnostics-and-tooling). A mechanism you have tried
and failed to refute is worth ten mechanisms that merely fit the first
symptom. This is also the reviewer's job in change control — do it before
they do.
2. The idea lifecycle in this project
hunch ──► issue with predicted numbers ──► sandboxed experiment
│
measured result vs prediction
match?│
yes ◄─────────────┴─────────────► no
│ │
PR through change control revise mechanism,
│ or retire with a
docs/ page (user-visible written post-mortem
features) + CHANGELOG (plenum-failure-
archaeology entry)
Step 1 — Hunch → issue. File the issue before the fix. House style
(visible in #280–#305, #86, #237): Summary, Severity, exact file:line
"Where", predicted numeric impact, "Suggested fix", and reproduction
conditions. One issue per independent mechanism — #281's two causes lived in
one issue only because both gated the same user-visible feature and had to
ship together, and the issue explicitly says "two independent root causes;
both must be fixed."
Step 2 — Experiment in a sandbox, never on live HVAC first. Three
sandboxes exist, in increasing realism. Know exactly what each does and does
not give you:
| Sandbox | What it IS (verified in code/docs) | What it is NOT |
|---|
System Off (system_enabled flag) | Semantics owned by plenum-run-and-operate (engine gate is system_enabled OR dev_mode, scheduler.py:477): with Dev Mode also off, engines skip their ticks; any running cycle is aborted immediately (vents restored, setpoint released). Plenum keeps monitoring HA state and serving the UI with zero service calls to HA once that one-time in-flight abort completes (abort nuance: plenum-run-and-operate). | Not an experiment mode on its own — with Dev Mode off, nothing computes, so you observe HA passively but learn nothing about engine decisions. With Dev Mode ON, engines still tick even while System is Off (writes intercepted) — that combination is Dev Mode's shadow mode, not a quieter System Off. |
Dev Mode (developer_mode flag) | Pure write interception: every HA write in ha_client.py (set_thermostat_temperature, set_thermostat_hvac_mode, set_thermostat_temperature_range, open_cover, close_cover, position/tilt/toggle) checks self.dev_mode and logs a [DEV] Would … line to the event log instead of calling HA. The engine runs fully: real 60s ticks, real sensor reads, real room selection and setpoint math. See docs/system-modes.md. | No fake clock, no simulated physics, no simulated cycles. Reads are live and time is wall-clock. Because commands never reach HA, rooms never respond to the "conditioning" — so closed-loop behavior (reaching target, post-closure drift, min-runtime holds ending naturally) will NOT play out as it would live. Dev Mode validates decisions, not outcomes. Cycles started in Dev Mode also historically lingered ACTIVE across toggles (#54) — check cycle state after toggling. |
| Compose test stack / fake HA | The E2E docker-compose stack (see plenum-build-and-env) and backend/tests/integration/fake_ha.py give scripted HA responses — you control the physics. | Not real thermal mass or compressor behavior (§4d). |
For data-shape experiments (migrations, metric queries, trim/pagination
logic), work against a copy of the production DB, never the live file:
sqlite3 /path/to/app.db ".backup '/tmp/experiment.db'"
(app.db lives in DATA_DIR, default /data — locations per install mode
in plenum-run-and-operate.) Point a local dev backend at the copy via
DATA_DIR, or query it directly with the recipes in
plenum-diagnostics-and-tooling.
Only after the mechanism survives a sandbox do you touch a live system — and
even then, System Off → Dev Mode → live, watching the Logs page at each step.
Step 3 — Measured result vs prediction. Paste both into the issue: the
number you predicted and the number you measured. #86's issue body is the
model — a minute-by-minute event table that any reviewer can re-derive.
Step 4 — PR through change control. No shortcuts: classification,
required tests, evidence expectations are owned by plenum-change-control.
If your fix adds a temperature field, config option, or UI change, the add-a-
knob / parity checklists apply (plenum-config-and-flags,
plenum-validation-and-qa).
Step 5 — Docs or documented retirement. A shipped user-visible behavior
gets a docs/ page (house rules in plenum-docs-and-writing — e.g.
docs/overflow-conditioning.md came out of #237). A dead idea gets a
written record too: the refuting observation and the rejected approach go
into the issue before closing, so the next person finds the corpse instead of
re-running the experiment. The settled-battle chronicle is
plenum-failure-archaeology; if your retirement is load-bearing (someone
will plausibly re-propose it), add it there.
3. Where good ideas historically came from — four generators
Evidence: the 48 closed bugs and 58 closed features. Four repeatable
generators produced nearly all of them.
3a. Production symptom → design change
The best features here were extracted from bug timelines, not imagined:
- #237: vent thrashing during minimum-runtime holds (open→close→open
churn, verified against
_monitor_rooms code paths) → the fix half
(persisted hold state) plus a design insight: excess conditioning during
the hold could be redirected into presence-idle rooms → overflow
conditioning (docs/overflow-conditioning.md, follow-ups #254, #268,
#305).
- #86: the post-closure-drift investigation established that room
temperature measurably drifts after vent closure — a physical fact the
engine now respects (the
vent_closed_at termination guard, and drift
compensation via the room temp_offset field — the #86→temp_offset
lineage is inferred from the briefing history, not traceable in git log).
Method: when closing a bug, ask "what did the timeline teach us about the
physical system that the design didn't know?" That delta is a feature
candidate.
3b. Systematic audits — how to run one (the #32/#48 and #280–#305 method)
Two engine audits (#32, #48) and one pre-launch full-stack audit (~25 issues,
#280–#305) found the highest-severity latent bugs in the project's history —
runaway-HVAC-grade (#280), safety-monitoring gaps (#286), and every °C
sibling of #231. The repeatable method:
- Scope one subsystem end-to-end, not a grep pattern: "the engine's
per-tick flow," "every temperature that crosses the API," "everything the
WS reconnect path can do." #32's issue opens by documenting how the
system works today — write that description first; the bugs fall out of
the places where the code refuses to match your description.
- Read adversarially. For each statement of intent (docstring, CLAUDE.md
invariant, issue promise), ask "what input makes this false?" #280 came
from reading the unit contract ("all temps are °F internally") and then
checking every read site against it — climate-entity attributes were raw.
#281 came from distrusting a test fixture's payload shape.
- One issue per independent finding, each self-contained: severity,
file:line, mechanism, predicted numeric impact, suggested fix, and a
provenance footer (
_Found during full front-end + back-end logic audit._). Never batch unrelated findings — they get triaged, fixed, and
reverted independently.
- Severity-triage before fixing anything. The #280–#305 wave marked
HIGH the ones with physical-world consequences (runaway HVAC, suspended
safety monitoring) and fixed those first; UI paper cuts (#263's missing
Apply button) queued behind them.
- Audit the tests too — a mock that pre-normalizes data (#280's
test_vacation_mode.py comment claiming HA returns °F) is itself a
finding.
3c. Test-review sweeps (#268–#270)
A cheaper audit variant: read the test suite against the feature list and
file an issue per untested guard. #268 (min-runtime hold gate — the #237
anti-thrash fix — had zero coverage), #269 (in-tick system-disabled and
vacation abort guards untested), #270 (nice-to-fix engine test items). The
question that generates these: "which safety-critical branch would a mutation
in production leave green in CI?" Also #212 — cycle timeout had no
end-to-end test.
3d. Safety reviews (#208–#213)
A checklist review against domain knowledge rather than code: enumerate the
ways real HVAC equipment gets damaged (short-cycling, dead-heading, running
AC in cold weather, acting on stale sensors) and check the engine has a guard
for each, on by default (#213 specifically reviewed defaults). Domain
checklist source: hvac-zoning-reference. Safety guards, once shipped, are
one-way ratchets — never weakened to fix comfort (inferred project norm; see
plenum-change-control).
4. Experiment hygiene specific to Plenum
4a. Any symmetric contract needs both-directions tests
The generalized #231 lesson: a contract with two symmetric sides passes
trivially when both sides make the same mistake or when you only exercise the
identity direction. °F mode made double conversion a no-op — only the °C leg
of the E2E matrix could catch it, and per CLAUDE.md it is the only test
that would have. Apply the same rule to every symmetric pair:
- °F/°C: run both matrix legs; assert exact values, not "roundtrips."
- restart-resume: test stop-mid-state → restart → resume, not just clean
start (see 4c).
- enable/disable: #54 (dev-mode cycles linger ACTIVE across toggles) and
#51 (system stop leaves cycle rows open) both came from testing only the
enable direction. Toggle both ways mid-cycle and assert the state table.
- connect/disconnect: #283 (frontend WS reconnects after intentional
close) and #297 (backend reconnect busy-loop) — test the teardown
direction of every connection path.
4b. Timezone matrix
Time bugs here recur (#26, #65, #294, #301). Any experiment involving
timestamps must run at minimum under: UTC, a negative-offset zone
(America/Chicago), and ideally across a DST boundary. Predict the exact
shift (§1b). Naive-UTC-rendered-as-local is the house pathology; the tz
helpers live in smart_vent/backend/tz.py.
4c. Restart-mid-state survival
The engine persists cycle state and restores it on startup. Any experiment
that changes cycle/hold/override state must include: kill the backend
mid-cycle → restart → verify the state restored and the invariant still
holds. Precedents: #300 (restore leaked overflow-room state because
ON CONFLICT dropped role/joined_at), #237's fix persisting
min_runtime_until specifically so holds survive restarts, #51 (rows left
"Active/running" after stop). A result that only holds within one process
lifetime is not accepted.
4d. Simulated pass is necessary, not sufficient
Every sandbox in §2 has fake physics. Real systems differ in ways that have
bitten this project:
- Thermal mass: real rooms keep changing temperature after a vent closes
(#86's post-closure drift) — a simulator that freezes temp at target would
never have shown the bug.
- Compressor minimum runtimes and off-times: real equipment cannot start/
stop per-tick; the short-cycle guards (#208) exist because of it, and #237's
thrash only manifests during a real minimum-runtime hold.
- HA entity quirks: clamping (a 69.8 °C setpoint silently becomes ~35 °C,
#280), unsupported cover services (#57), entities going unavailable
mid-cycle (#267).
So the ladder is: unit test → integration/fake-HA → Dev Mode against real
sensors → System On on live equipment, watched. Claim success only at the
rung you actually reached, and say which rung in the PR.
4e. Don't let the instrument lie
Two recurring traps: mocks that encode the wrong reality (#280, #281 — the
test asserted the bug), and CI env vars that mask the production path (#281:
compose stacks always pass TEMPERATURE_UNIT explicitly, so the blank-auto
path never ran in CI). Before trusting a green run, ask: does any fixture,
env var, or mock differ from production in the exact dimension under test?
5. Result acceptance checklist (copy into the issue/PR)
## Result acceptance — <issue #>
### Mechanism
- [ ] One stated mechanism, in one paragraph, at file:line granularity.
- [ ] It reproduces EVERY observation in the timeline, including negatives
(things that did NOT happen) and passing tests.
- [ ] Independent co-causes ruled out or separately filed (#281 precedent).
### Prediction vs measurement
- [ ] Numeric prediction written BEFORE measuring (exact value, not a bound).
- [ ] Measured value pasted next to it; matches to the decimal.
- [ ] The falsifying observation was constructed and checked (§1c) — state
what it was and what you found.
### Experiment provenance
- [ ] Sandbox rung reached: unit / fake-HA / Dev Mode / live (state which).
- [ ] DB experiments ran against a `.backup` copy, never live app.db.
- [ ] No live-HVAC action before a sandbox pass.
### Hygiene
- [ ] Symmetric contract? Both directions tested (F/C, resume, disable,
disconnect).
- [ ] Timestamps involved? Non-UTC zone tested; shift predicted exactly.
- [ ] State involved? Restart-mid-state survival verified.
- [ ] Fixtures/mocks verified against real payload shapes (#280/#281 trap).
- [ ] If only simulated: limitations vs real HVAC stated (thermal mass,
min runtimes, entity clamping).
### Landing
- [ ] Change-control class + required evidence met (plenum-change-control).
- [ ] Regression test that fails on the old code, at the mechanism (not a
symptom snapshot).
- [ ] docs/ page updated or created if user-visible; CHANGELOG entry.
- [ ] If retired instead: refuting observation written into the issue;
archaeology entry if re-proposable.
Provenance and maintenance
Facts verified against the repo on 2026-07-05 (v0.22.1):
- Dev Mode = write interception only, no fake clock/physics:
grep -n "dev_mode" smart_vent/backend/ha_client.py (interception sites)
and docs/system-modes.md.
- System Off semantics (skip ticks unless Dev Mode is on, abort running
cycle, zero HA calls after the one-time abort): engine gate
system_enabled OR dev_mode at scheduler.py:477; details owned by
plenum-run-and-operate. Note docs/system-modes.md (~line 26) states
the precedence backwards — a known doc bug; the code wins.
- Conversion helpers live in
smart_vent/backend/units.py (to_f,
delta_to_f, from_f, from_f_delta) — consolidated by #251.
Pre-2026-07-05 CLAUDE.md copies said _to_f/_delta_to_f in routes.py;
corrected in PR #388 — if docs and repo disagree again, the repo wins
(see plenum-architecture-contract).
Re-verify: grep -n "def " smart_vent/backend/units.py.
- 141.44 double-conversion arithmetic:
python3 -c "print((16*9/5+32)*9/5+32)".
app.db path: smart_vent/backend/main.py:42-43 (DATA_DIR default
/data, app.db). Re-verify: grep -n "DATA_DIR" smart_vent/backend/main.py.temp_offset field: smart_vent/backend/models.py:44. Its origin in #86
is inferred from project history, not traceable in git log.- Issue narratives (#32, #48, #86, #231, #237, #280–#305, #208–#213,
#268–#270, #51, #54, #57, #267, #283, #297, #300, #301) summarized from
the closed-issue record; re-verify with
gh issue view <n> --repo dhruvb14/smart-thermostat-with-vents.
- Line numbers cited inside issue quotes (e.g.
cycle_engine.py:994-1000)
are as-of the issue's filing date and will drift; search by symbol name.
