emd-harness

name

emd-harness

description

Design or refactor AI harnesses, agent workflows, eval-fix loops, and self-improving coding systems using the EMD feedback pattern -- a symbolic generalization of PID control: Error (current failure), Memory (leaky history of recurring failures), Delta (change since last iteration). Use when adding structured iteration feedback, recurring-failure memory, trend/regression signals, stagnation handling, anti-windup guardrails, or architecture-aware decision routing to an iterative AI workflow. Ships a standalone Python reference implementation (scripts/emd_state.py).

EMD Harness

Use EMD as a workflow-control pattern for AI harnesses. It is inspired by PID control but its state is symbolic and structured -- do not treat it as a literal numeric PID controller.

EMD means:

• Error (P): the current structured failure signal -- what is wrong now.
• Memory (I): leaky accumulated history of recurring failures and unresolved bias -- what keeps happening.
• Delta (D): change since the previous iteration -- regressions, improvements, and responsiveness to the last action.

When this applies

Any loop that iterates on an artifact against a checkable objective: controller synthesis, prompt optimization, agentic code-fix loops, eval-driven refactors, data-pipeline tuning. If the loop currently routes on a single pass/fail boolean or dumps full logs into the next prompt, EMD is the upgrade.

Workflow

1. Identify the loop: plant, sensors, actuators, objective, and iteration boundary (see the mapping table in references/emd-pattern.md).
2. Replace raw pass/fail with structured, normalized error vectors (Error / P).
3. Add a leaky-integral memory of repeated failures, with decay and bounds (Memory / I).
4. Add delta signals for score changes, metric regressions, and responsiveness (Delta / D).
5. Feed a compact EMD state (not raw logs) into planner, diagnoser, generator, reviser, and decision prompts.
6. Add guardrails for anti-windup, stagnation, rollback, and escalation.
7. Verify with tests that repeated failures and regressions actually change future decisions.

Use scripts/emd_state.py as the reference implementation for steps 2-6 rather than rewriting the state math; adapt the Metric/Observation field names to the harness.

Implementation rules

• Keep the EMD state deterministic and serializable before giving it to an LLM.
• Normalize metric errors against goals or thresholds so they are comparable.
• Store memory as structured counters, decayed scores, or concise hypotheses -- never long natural-language transcripts.
• Treat delta as a short-horizon trend signal, not as proof of causality.
• Use EMD to bias or route actions; never let it silently override hard safety, correctness, or user constraints.
• Preserve the actual revised artifact between iterations. A revision that the next generator overwrites is not a closed loop.
• Fix implementation/tool/sensor invalidity before tuning parameters (the harness can't tune through a broken sensor).

Resource map

Read the reference for the full pattern; read or run the scripts to implement it.

Resource	Purpose
references/emd-pattern.md	Full pattern: loop mapping, detailed schemas, per-term field guides, decision-routing rules, prompt block, persistence, verification checklist, anti-patterns. Read when designing the state or wiring the prompt.
scripts/emd_state.py	Standalone, stdlib-only reference implementation. Builds the EMD state (Error/Memory/Delta + guidance + compact prompt block) from a list of Observations. Adapt field names; reuse the math.
scripts/test_emd_state.py	Smoke test covering normalization, recurrence/decay, regression/improvement, guidance routing, and prompt stability. Run after edits.

Minimal usage

from emd_state import Metric, Observation, build_emd_state

history = [
    Observation(1, score=0.30, passed=False, failure_labels=["overshoot"],
                metrics=[Metric("overshoot_pct", actual=18.0, target=5.0, direction="max")]),
    Observation(2, score=0.42, passed=False, failure_labels=["overshoot"],
                metrics=[Metric("overshoot_pct", actual=13.0, target=5.0, direction="max")]),
]

state = build_emd_state(history, responsiveness="monotonic_wrong")
prompt_block = state["prompt"]          # compact text -> inject into the next LLM prompt
bias = state["guidance"]["primary_action_bias"]  # route the next action

responsiveness comes from the harness's own action->metric sensitivity probe (monotonic_correct, monotonic_wrong, none, noisy, ...); leave it "unknown" until such a probe exists.

Output expectations

When applying this skill to a harness, deliver:

1. A serialized EMD state per iteration (emd_report.json or similar) persisted in the run trace.
2. A compact prompt block wired into the decision/generation prompts -- not raw logs.
3. Decision routing that demonstrably changes after repeated/stagnant failures (covered by a test).
4. Guardrails for anti-windup, regression rollback, and escalation, with a note that EMD biases actions but never overrides hard constraints.

Recommended state shape:

{
  "error":   {"score": 0.0, "passed": false, "failure_labels": [], "metric_errors": {}, "dominant_error": {}},
  "memory":  {"decay": 0.65, "recurring_failures": [], "integral_metric_errors": {}, "stagnant_iterations": 0},
  "delta":   {"score_delta": 0.0, "metric_error_delta": {}, "regressions": [], "improvements": [], "responsiveness": "unknown"},
  "guidance":{"primary_action_bias": "", "action_biases": [], "guardrails": []}
}