harness-evolution-orchestrator

name	harness-evolution-orchestrator
description	Coordinate the slow-loop trigger conditions for harness evolution across sessions.

harness-evolution-orchestrator Skill

Purpose

Operate the slow loop. This skill decides when the harness should transition from observation into mutation based on repeated evidence rather than one-off friction.

Signals

• repeated governance failures: trigger when the same governance rule fails at least 3 times within 7 days, sourced from run_records.status, invariant IDs, and benchmark governance task results.
• recurring instruction drift: trigger when the same drift pattern appears in 2 consecutive sessions within 14 days, sourced from analyzer findings and benchmark hygiene task IDs.
• benchmark regressions in held-out tasks: trigger when held-out task success rate drops by more than 5 percentage points from the most recent promoted baseline, sourced from benchmark summaries.
• stale audit trail or state artifacts: trigger when HARNESS_EVOLUTION_STATE.md has no update for 30 days or run-artifacts/ has no new session manifest for 14 days.

scripts/harness_core/evals/trigger_evals.yaml entries must define:

• trigger_name
• type
• threshold
• metric_field
• comparison_operator
• timeframe
• enabled

Validation rules:

• comparison_operator must be one of >=, >, ==, <=, <.
• timeframe must be an ISO-8601 duration or an explicit day count such as 7d or P14D.
• Missing trigger_evals.yaml means "no triggers configured"; log that state and stop.
• Malformed entries must be reported with field-level errors and skipped rather than silently applied.

Signal mapping:

• governance failures use metric_field: governance_failure_count
• instruction drift uses metric_field: repeated_drift_count
• held-out regression uses metric_field: held_out_pass_rate_delta
• stale state uses metric_field: stale_days

Steps

1. Pre-flight validation: verify scripts/harness_core/evals/trigger_evals.yaml and HARNESS_EVOLUTION_STATE.md are readable, and confirm sub-skills harness-analyzer, harness-transformer, and harness-candidate-pr-loop are available before continuing. If any dependency is missing or unreadable, stop and emit a blocking error report.
2. Inspect scripts/harness_core/evals/trigger_evals.yaml and validate every entry against the schema above before evaluating trigger thresholds.
3. Review the recent ledger entries in state/harness-run-ledger.sqlite and the collected artifacts in run-artifacts/<session-id>/manifest.json for the last 14 days. Check run_id, status, updated_at, score_delta, held_out_pass_rate, artifact error counts, and missing-manifest gaps before deciding whether the slow loop should move.
4. If triggers are not met, stop and report which metrics stayed below threshold and the exact observation window used.
5. If triggers are met:
   • capture a pre-mutation snapshot of the current state file, relevant ledger rows, and latest artifact manifest
   • run harness-analyzer; treat analysis failures as hard stops, log the failure, and abort the chain
   • run harness-transformer; validate its output schema before proceeding, and abort with a logged validation error if the payload is malformed
   • hand off to harness-candidate-pr-loop; sanitize downstream inputs, retry only on configured transient failures, and surface terminal errors without swallowing them
   • run post-mutation verification by re-running trigger evaluation or a smoke benchmark; if verification fails, roll back using the captured snapshot and record the rollback
6. Update HARNESS_EVOLUTION_STATE.md with a complete state record using this schema:
   • timestamp
   • trigger_evaluations
   • decision_outcome
   • subskill_executions
   • verification_results
   • next_scheduled_evaluation
   • circuit_breaker

circuit_breaker must cap the system at 1 mutation attempt per 24 hours unless a human resets it. Increment it before a mutation handoff, reset it on success, and record rollback events explicitly when verification fails.

Constraints

• Do not mutate the harness in the same session that detected the issue.
• Prefer repeated objective evidence over subjective frustration.
• Escalate authority-boundary changes for human review. Authority-boundary changes are any changes that alter permissions, decision authority, safety guardrails, or compatibility contracts. Examples: permission model changes, trigger-threshold rewrites, ledger/state schema breaks, cross-plugin interface changes, or updates that weaken safety constraints. Escalate when the change broadens write or execution scope, changes promotion criteria, breaks stored state compatibility, or affects another plugin/repo contract.