TDD cycle scoring, self-improvement, and governance system for Hermes Cortex. Scores completeness/quality/progress per cycle with nomic embeddings, logs to SQLite, collects user feedback via CLI, auto-applies config patches with safety bounds, and integrates weekly evaluation + retention. Runtime config, test suite, setup/verify/update scripts, and auto-remediation health checks.
Instrucciones de origen · Vista previa de solo lectura
name
loop-governance
version
1.5.0
category
devops
description
TDD cycle scoring, self-improvement, and governance system for Hermes Cortex. Scores completeness/quality/progress per cycle with nomic embeddings, logs to SQLite, collects user feedback via CLI, auto-applies config patches with safety bounds, and integrates weekly evaluation + retention. Runtime config, test suite, setup/verify/update scripts, and auto-remediation health checks.
The universal loop controller. Every agent loop — whether TDD cycles, review→fix→review, or plan→execute→verify — needs three things: a way to score progress, a way to decide what to do next, and a bound that prevents infinite spinning.
Overview
Loop governance provides a scoring function and a decision matrix that any workflow skill can import. The scorer uses nomic-embed-text (local, cheap, ~50ms per call) for three independent measurements:
Dimension
What it measures
How (nomic)
Completeness (0-10)
How much of the goal is achieved
Embed test output vs spec → cosine similarity
Quality (0-10)
Code density, TODOs, docstrings
Embed code → magnitude + heuristic penalties
Progress (0-10)
Did this iteration change anything?
Embed prev output vs current → 1 - cosine similarity
These feed into a weighted composite score that drives the decision gate.
When to Load This Skill
Load loop-governance whenever you are running any multi-iteration loop:
After a change-test-loop cycle completes — decide whether to loop for more coverage
After a review→fix→review cycle in subagent-driven-development — decide whether to keep iterating
After a spike→evaluate→refine exploration loop
After a plan→execute→verify pipeline iteration
Anywhere you ask "should I loop again, stop, or escalate?"
A local SQLite vector cache (session-embeddings.db) stores embeddings from
sessions, loop DB cycles, and skills. It improves progress detection by comparing
current code against ALL stored good patterns, not just the immediate previous cycle.
Cache integration:loop_scorer._cache_boost() checks the cache during every
score_progress() call. If the current output is very similar (cosine > 0.85)
to a known-good pattern in the cache, progress gets a +2 boost. For moderate
similarity (>0.75), +1 boost. This means scoring improves as the cache grows.
Cron:agent-session-cache-build runs Monday 5am KST (weekly cache rebuild; the daily orch-skill-lifecycle pipeline at 4:34am reads whatever cache state exists).
Agent data: Agents contribute session data via agent-learning-collector Learning Reports (to inbox_orchestrator, every 6h) and the session cache. The cache is rebuilt weekly, incorporating new data.
Installation & invocation
The scorer module lives at scripts/loop_scorer.py in this skill directory (underscore so Python can import it directly). Run a demo from any directory:
from loop_scorer import score_progress, score_quality, score_completeness, composite_score
# Or for one-off scoring via shell
import json
result = json.loads(terminal(
'python3 -c "from loop_scorer import composite_score; import json; print(json.dumps(composite_score(9, 8, 9)))"'
)["output"])
Naming convention: Python modules use underscores (loop_scorer.py). CLI commands use hyphens (score-cycle, loop-feedback, auto-apply). This is deliberate — Python can't import hyphenated names, and CLI tools conventionally use hyphens.
What each dimension measures
Completeness
Measures how much of the spec/requirements the implementation fulfills. Uses a blended approach (60/40):
Test pass rate (60%) — the strongest single signal. If all tests pass, completeness gets at least 6/10. Maps via: 100%→10, 90%→8, 80%→6, 50%→4, below→linear.
Embedding similarity (40%) — cosine similarity between test output embedding and spec text embedding, mapped to 0-10.
The pass-rate anchor prevents false negatives when the test output boilerplate (pytest headers, PASSED/FAILED markers) dilutes the embedding signal. Always pass pass_pct when the test runner output is available.
5-7: Core behavior working, some edge cases or detail missing
2-4: Minimal test coverage, major spec gaps
0-1: No tests, all failing, or tests unrelated to spec
Quality
Embed the implementation code and compute a semantic density score. Apply heuristic penalties for TODO/FIXME markers, stub implementations, missing docstrings.
quality = score_quality(code_text)
# Returns 0-10
Scale:
8-10: Well-structured, documented, no stubs, proper error handling
5-7: Functional with minor quality gaps (light docs, some edge cases)
Embed the state from the previous iteration and the current iteration, then compute 1 - cosine_similarity. The more different the embeddings, the more progress.
Critical: compare CODE against CODE, not test output against test output.
Test runner output (pytest, etc.) contains heavy boilerplate — PASSED/FAILED markers, headers, timing info — that stays nearly identical across iterations even when the code changes substantially. Comparing test output produces artificially low progress scores (0-1/10). Comparing implementation code produces accurate progress detection.
# ✅ CORRECT — compare code
progress = score_progress(previous_code, current_code)
# ❌ WRONG — test output has too much boilerplate
progress = score_progress(previous_test_output, current_test_output)
Scale:
8-10: Completely different output — substantial change
5-7: Meaningfully different — real progress
2-4: Slightly different — minimal change
0-1: Nearly identical — no progress (trigger no-progress counter)
No-Progress Detection
The most common token waste in loops is spinning: the agent keeps iterating but produces the same output each time. No-progress detection prevents this.
Rule
# Track across iterations
no_progress_count = 0
previous_embedding = None
for iteration in range(max_iterations):
current_output = run_iteration(goal)
if previous_embedding:
progress = score_progress_from_embeddings(previous_embedding, embed(current_output))
if progress < 2.0: # threshold: less than 2/10 change
no_progress_count += 1
else:
no_progress_count = 0 # reset on genuine progress
previous_embedding = embed(current_output)
if no_progress_count >= 3:
return {"decision": "STOP ✗", "reason": "no-progress ≥ 3 iterations"}
Key point: Check progress at the semantic level (embeddings), not just text diff. A single comment change is not progress even though the text differs.
When to apply
Apply no-progress detection to:
Change-test-loop: between complete LEARN→RED→GREEN→REFACTOR cycles
Subagent-driven-development: between review→fix→review cycles (same finding appearing repeatedly)
Any exploration loop where successive iterations produce similar output
Bounded Iteration
Every loop must have at least one hard bound:
Bound
Default
When to adjust
Max iterations
5
Increase for complex multi-phase work, decrease for simple fixes
No-progress limit
3 consecutive
Tighten to 2 for cost-sensitive tasks
Token budget
None (opt-in)
Set when running against paid API — e.g. "200K tokens total"
Setting bounds
BOUNDS = {
"max_iterations": 5, # hard stop after this many cycles
"max_no_progress": 3, # stop after N iterations with < 2/10 progress
"max_retries_per_phase": 2, # per-phase retry limit (from change-test-loop)
"token_budget": None, # optional: max total tokens for this loop
}
No bounds → open loop. Open loops are only acceptable for continuous monitoring (cron jobs, watchdogs). Every code-workflow loop must be closed (bounded).
Fresh Context Strategy
After each loop iteration, the next iteration should start with fresh context — not the accumulated reasoning and tool output from previous rounds. This prevents context window degradation.
For subagent loops, this is natural: each delegate_task gets a clean agent. For in-session loops (change-test-loop), you can:
Summarize the current state into a compact prompt for the next iteration
Use the loop-scorer output as the steering signal — summarizing current scores and remaining gaps.
Data Capture & Self-Improvement
Every loop cycle generates data — scores, decisions, code, test results. Storing this data enables the system to improve itself over time by analyzing past decisions and tuning thresholds.
Every loop decision is a prediction — the scorer says "this is good enough to STOP" or "this needs another LOOP". User feedback provides the ground-truth label that turns predictions into a training dataset for meta-learning.
When to provide feedback
Provide feedback whenever you see a loop decision that was clearly right or clearly wrong:
Situation
Action
All tests pass, coverage good, quality acceptable → STOP was correct
loop-feedback accept <id>
Tests pass but edge cases remain → LOOP would have been better
loop-feedback override <id>
Spinning for 3+ cycles with no change → STOP (hard fail) was correct
loop-feedback accept <id>
System gave up too early (MOVE ON) but a better prompt would work
loop-feedback override <id>
System kept looping when goal was already met
loop-feedback override <id>
How to provide feedback
The loop-feedback CLI is available globally. Use --db <path> to point at a non-default database:
# List cycles that need feedback
loop-feedback list
# List recent cycles (with or without feedback)
loop-feedback list --all
loop-feedback list --limit 20
# Use a custom DB path
loop-feedback list --db /path/to/loop-governance.db
# Accept a decision (it was correct)
loop-feedback accept <cycle_id> --note "All tests pass, quality is high"
loop-feedback accept 42
# Override a decision (it was wrong — the scorer was wrong)
loop-feedback override <cycle_id> --note "Should have kept going, missing null case"
loop-feedback override 7
# Show feedback statistics
loop-feedback stats
# JSON output (for programmatic use)
loop-feedback list --json
loop-feedback stats --json
Interpretation
DB value
Meaning
user_overrode = NULL
No feedback yet (default)
user_overrode = 0
Accepted — user agrees the decision was correct
user_overrode = 1
Overridden — user disagrees with the decision
How feedback drives meta-learning
The weekly evaluation cron (loop_evaluator.py) reads user_overrode to compute:
Decision accuracy: what % of decisions are correct? Drives confidence in the system.
Threshold recommendations: if STOP decisions are frequently overridden at composite 8.0, maybe the threshold should be 8.5.
Weight recommendations: correlates each score dimension (completeness, quality, progress) with user acceptance, suggesting weight adjustments.
Config patches: generates JSON patches that could be auto-applied at high confidence.
Without user feedback, the evaluator reports: "No user feedback recorded yet. User feedback is the ground-truth label for meta-learning."
Auto-Apply (Phase 3 — Self-Tuning)
The auto-apply system reads the evaluator's config patch, validates every change against safety bounds, and applies low-risk modifications to the runtime config. This closes the loop: data → evaluation → recommendation → application → improved decisions.
Every proposed change is validated against hard limits (from loop-config.json):
Parameter
Bound
Rationale
min_confidence
0.7
Don't act on noisy data
max_threshold_delta
1.0
At most 1 point adjustment per week
max_weight_delta
0.10
At most 10% weight shift per week
Stop threshold range
[5.0, 10.0]
Must stay meaningful
Move-on threshold range
[1.0, 5.0]
Must stay meaningful
Weight range
[0.05, 0.80]
No single dimension can dominate or vanish
CLI usage
# Normal — skip if confidence < 0.7
auto-apply
# Preview without applying
auto-apply --dry-run
# Bypass confidence check (use with caution)
auto-apply --force
# JSON output for cron consumption
auto-apply --json
# Dry-run with JSON
auto-apply --dry-run --json
Runtime config
Thresholds and weights live in ~/.hermes-cortex/data/loop-governance-config.json and are read by composite_score() on every call:
# View current config
python3 ~/.hermes/skills/.../scripts/loop_config.py --show
# Set a value manually
python3 ~/.hermes/skills/.../scripts/loop_config.py --set weights.completeness 0.45
See references/config-format.md for the full config schema, field descriptions, and rollback procedure.
Cron schedule
All loop-governance crons managed via crons.json (versioned) + install-crons.py:
Cron
Schedule
Mode
Behaviour
agent-session-cache-build
Mon 5am KST
no_agent
Rebuilds embedding cache from sessions, DB, skills
local-agent-weekly-loop-eval
Mon 9am KST
LLM-driven
Per-host weekly loop evaluation (fleet-level eval absorbed into orch-skill-lifecycle 2026-08-02)
Versioning: Bump the version field in crons.json to trigger agent updates.
The install-crons.py script reads the template, removes stale crons (by name),
and creates fresh ones idempotently with the correct argument order (prompt must
come BEFORE --flags in hermes cron create). Called automatically by setup.sh.
New project setup: The agent-session-cache-build and local-agent-weekly-loop-eval crons run only on machines
with the loop-governance toolchain installed. Health monitoring (Ollama, DB, nomic
model, cycle count) runs inside system-alert.py every 10 minutes — no separate
health cron needed.
Verify: hermes cron list | grep -E '(loop|weekly|session-cache)'
Content sanitization
Code snapshots are sanitized before storage via LoopDB.sanitize_code():
API keys, tokens, passwords get redacted
Connection strings with credentials are masked
PEM private key blocks are removed
Evaluation Pipeline
A weekly analysis pipeline (loop_evaluator.py) reads the loop governance DB and produces a structured report every Monday at 9am via cron.
What it analyzes
Analysis
What it detects
Example output
Summary stats
Total cycles, unique tasks, avg scores
total_cycles: 47, avg_composite: 6.2
Decision distribution
STOP/LOOP/MOVE ON/HARD FAIL counts
STOP: 12, LOOP: 28, MOVE ON: 5
Score trends
Drift in each dimension (first vs second half)
completeness ↑ +1.2, quality ↓ -0.8
No-progress hotspots
Tasks with most no-progress cycles
feature-auth: 7 np cycles
Spinning tasks
3+ consecutive no-progress cycles
task-b-spinner: 5 consecutive
Decision accuracy
User feedback vs decision match rate
66.7% correct (2/3)
Threshold recommendations
Suggested threshold adjustments
Raise STOP to 8.5
Weight recommendations
Correlation-based weight adjustments
completeness: 40% → 45%
How to run manually
# Full report
python3 ~/.hermes/skills/devops/loop-governance/scripts/loop_evaluator.py
# JSON output (for programmatic consumption)
python3 loop_evaluator.py --json
# Config patch only
python3 loop_evaluator.py --config-patch
# Custom time window
python3 loop_evaluator.py --days 30
Cron schedule
A weekly evaluation runs every Monday at 9:00 AM KST via the local-agent-weekly-loop-eval cron job. It delivers the report automatically to the chat that created it.
Config patch
The evaluator can generate a recommended config patch — suggested changes to thresholds and weights based on historical data. Patches are reviewed by a human before application (included in the cron output).
Lock files and the immutable chain have failure modes beyond the skills marker. Hard-won lessons from P1-A (2026-07-31). Full detail: references/lock-lifecycle-race.md (mechanism + verification) and references/enforcement-update-surface.md (complete write-path audit + sanctioned-caller gate design).
Mid-write lock theft: every purge path — the enforcer's _has_governance_lock() pre-scan + Phase 2 scan, the MCP's _purge_stale_locks(), and purge-stale-governance-locks.py — used to delete any .governance-*.json that failed json.loads. _write_lock() wrote non-atomically (write_text), so a concurrent purge read mid-write caught partial JSON and deleted a FRESH lock. Symptom: begin_change succeeds, then seconds later check_lock says inactive and the file is gone. Fix (shipped): atomic writes (temp + rename, same fs) + purge loops skip unparseable files (never delete what you can't parse — it is being written). Locks now carry session_type (cron/bg/interactive); cron/bg sessions never purge interactive locks.
Sticky marker per governance lock — SUPERSEDED (2026-08-01): the sticky-marker
rule was the P1-A patch for the shared-file race: _check_skills_loaded_marker()
accepted a valid session:* marker when the session held an active governance lock.
It is now obsolete — markers are per-session files (state/skills-loaded/<session_id>),
so no concurrent session can steal the marker in the first place. The lock-based
exception and the daemon/subagent guards were removed with the shared file.
Session-identity marker race (.hermes-session-current.id) — FIXED (2026-08-02): the enforcer resolves the current session id from the shared fixed-path file ~/.hermes-cortex/state/.hermes-session-current.id, which ANY concurrent session (party subagents, background workers, parallel conversations) can overwrite. Symptom (confirmed 2026-07-31): your fresh .governance-<session>.json lock exists with a valid heartbeat, yet check_lock returns inactive and write tools block with "GOVERNANCE LOCK REQUIRED" — the enforcer looks up the lock under the stomped id. State dir shows two+ lock files (yours + a subagent's) and the marker holding a third, unrelated id. Fix (shipped in b26ea929): the enforcer now INJECTS args["session_id"] into every mcp__loop_governance__* call at pre_tool_call, and the MCP server's get_session_id(args) reads it as Priority 0 — beating the shared marker. The marker bridge remains only as legacy fallback for direct MCP callers. Recovery for pre-fix locks: re-acquire with begin_change() (creates the lock under the currently-resolved id), reload the 8 always-skills so the marker matches, then proceed. This compounds with Pitfall 3 (skills marker) — both live in the same shared state dir.
Enforcement update surface: to enforce "cortex-update is the ONLY way to update locked files", audit ALL write paths, not just the lock helper: the pre-commit DOGFOOD self-heal (auto-deploys the enforcer on commit — convert to a BLOCK), doctor remediation hints that say cp hooks directly (re-point to cortex-update.sh), the hermes-plugin-lock update self-update channel, and sudoers env_reset (callers must use explicit sudo CORTEX_UPDATE=1 cmd assignment — plain export does not survive sudo). macOS chflags needs no root → script-level gating is the only lever there.
Error Handling & Resilience
The scoring pipeline must not block development when a dependency is down. Every component degrades gracefully:
Failure point
Behaviour
Recovery
Ollama down (embedding unavailable)
embed() returns None — each scorer falls back to heuristic/pass-rate values. full_score() adds warnings that Ollama is unavailable. Cycle still logs to DB with logged: True.
system-alert.py (every 10 min) detects unresponsive Ollama and attempts restart (retry loop, 5x2s). Auto-remediation cron also handles it.
DB locked / write failure
Cycle scores proceed normally; result is returned with logged: false and log_error. JSON event still written to ~/.hermes-cortex/data/loop-events/.
Next cycle retries automatically. Remediation sensor checks DB writability.
Config file corrupt
get_config() falls back to hardcoded defaults. Warning logged but scoring continues.
Remediation sensor detects corrupt JSON and restores from config_history.
nomic model not pulled
Ollama returns 404 → fallback zero vector. User sees abnormally low scores.
setup.sh pre-pulls the model. Auto-remediation runs ollama pull nomic-embed-text on detection.
score-cycle CLI fails
Non-zero exit code + error message to stderr. No silent failures.
Logged to cron job error tracker. Auto-remediation applies known fixes.
Feedback DB query fails
loop-feedback returns error with cycle info. DB state unchanged.
Check DB permissions, disk space, schema version.
What gets NOTIFIED
The weekly cron report includes a health section showing:
DB size and row count
Last successful log timestamp
Any cycles logged with fallback (zero-vector scores)
Config file validity
Auto-remediation sends Telegram alerts for any pipeline component that fails 3+ consecutive checks.
Known verify.sh bugs (fixed)
Two bugs were found and fixed in verify.sh (all 3 copies: installed, skill, repo source):
set -euo pipefail breaks JSON mode: pass() and info() functions return the exit code of [[ "$JSON" != "1" ]] (exit 1 when JSON=1). With set -e, this kills the script on the first suppressed output. Fixed by adding return 0 to both functions.
JSON output mixed with human text: The header/summary text was printed to stdout, making JSON output unparseable by downstream scripts. Fixed by wrapping all non-JSON output in [[ "$JSON" != "1" ]] guards.
JSON Python string interpolation broken for multi-word entries: '${RESULTS[@]}'.split(' ') produced unterminated string literals when entries contained spaces. Fixed by writing results to a temp file and making Python read from the file.
Also fixed a quoting bug in loop-health-check.sh: the verify.sh flags were inside the double-quoted path string ("${INSTALL_DIR}/verify.sh --quick --json"), making bash treat them as part of the filename. Fixed by moving flags outside the quotes.
What gets AUTO-FIXED
The auto-remediation cron (every 5 minutes) handles these loop-governance issues:
Ollama down → restart via systemd/launchctl
Config file missing → recreate from defaults
DB file missing → recreate schema
Symlinks broken → re-link to canonical script paths
Resilience design principle
The scoring function is advisory, not critical. A failed score call must never block a build, a deploy, or a user's workflow. The system scores when it can, logs what it can, and reports what it can't — but it never halts.
File Organization
Every piece of the loop-governance system lives in a predictable location. This structure
lets any agent find the tools regardless of which Hermes profile or project it's working in.
Installation: Loop governance is installed automatically by cortex-update.sh (MCP server + plugin). The old CLI tools (score-cycle, loop-feedback) are deprecated.
Usage in a session:
mcp_loop_governance_begin_change(task_id="<name>", description="...") — start a change
Do the work (MCP server blocks write tools without a lock)
mcp_loop_governance_cycle_query(task_id="<name>") — find the cycle
mcp_loop_governance_feedback_accept(cycle_id=N, note="...") — score it (parameter is cycle_id, NOT id — passing id fails with "missing required argument")
mcp_loop_governance_end_change(task_id="<name>") — release the lock
Close-out is ENFORCED (2026-08-08, Luke directive):end_change() refuses
to release the lock while the task's cycle is unscored, and begin_change()
refuses a new task while this session still holds unscored PENDING cycles.
There is no warning-only path anymore — score every cycle (steps 3-4) BEFORE
end_change, and never start a new task until the previous one is scored and
closed. Hook-created cycles (precommit-* task ids, session_id NULL) do not
trip the gate.
Orphaned PENDING cycles from sibling sessions: the doctor's ❌ PENDING cycles failure often lists cycles you did NOT create — sibling/daemon sessions (background subagents, other CLI sessions, party agents) called begin_change and never scored. Enumerate with cycle_query(status="pending"), then feedback_accept(cycle_id=N, note="...") each verified-complete one. For cycles superseded by a later MOVE_ON cycle of the same task, cite the superseding cycle in the note (e.g. "cycle 2 (2152) verified deploy end-to-end").
PENDING vs scored-unreviewed — don't mass-accept hook cycles (2026-08-05): the pre-commit hook auto-logs one cycle per commit with a LOOP/MOVE_ON/STOP decision and user_overrode IS NULL. These accumulate by the hundred and look like "unreviewed" via cycle_query(unreviewed=true), but they are NOT failures: the doctor only fails on decision='PENDING' cycles. Only resolve genuinely PENDING cycles (fresh <24h, no live lock for that task). To tell them apart, query the DB directly: SELECT id, task_id, decision, user_overrode, timestamp FROM loop_cycles WHERE decision='PENDING' OR (user_overrode IS NULL) ORDER BY id — check decision per row, don't bulk-accept. Upstream doctor (63981498+) refines this further: a PENDING cycle whose task_id has NO active .governance-*.json lock is a LEAK (FAIL, blocks push); a PENDING cycle whose task holds a live lock is the current task (INFO, expected mid-session).
Lock STATUS field and the doctor's PENDING-cycle check (2026-08-08):begin_change writes the lock with status: "planning" when a plan is provided (loop-gov-mcp.py), "executing" otherwise. The doctor's leak check used to count ONLY status == "executing" locks as active — so every planned task's PENDING cycle was misreported as a LEAK (❌ FAIL) even while the task held its lock, and the doctor blocked the push mid-task. The lock's EXISTENCE is the active signal (end_change unlinks it on release); the doctor now accepts any non-terminal status (completed/cancelled excluded). If you see "❌ PENDING cycles" for a task whose lock you can see in ~/.hermes-cortex/state/.governance-*.json, check the deployed cortex_doctor/checks.py matches repo — a stale deployed copy with the old == "executing" filter produces exactly this false FAIL.
Or via the main installer: bash hermes-cortex/install.sh (loop-governance step removed in July 2026 — use cortex-update.sh instead).
Database Retention
Without a retention policy, the SQLite DB grows without bound. Implement a retention
schedule appropriate for your task volume:
from loop_db import LoopDB
db = LoopDB()
# Archive cycles older than 90 days to JSON events, delete from SQLite
db.conn.execute("""
DELETE FROM loop_cycles
WHERE timestamp < datetime('now', '-90 days')
""")
# Clean up orphaned content assets
db.conn.execute("""
DELETE FROM content_assets WHERE hash NOT IN (
SELECT spec_hash FROM loop_cycles WHERE spec_hash IS NOT NULL
UNION
SELECT code_hash FROM loop_cycles WHERE code_hash IS NOT NULL
UNION
SELECT test_output_hash FROM loop_cycles WHERE test_output_hash IS NOT NULL
)
""")
# VACUUM to reclaim space
db.conn.execute("VACUUM")
db.conn.commit()
print("Retention applied: deleted cycles older than 90 days")
db.close()
The weekly evaluation cron can be extended to include a retention step.
For high-volume environments, archive the JSON events directory (~/.hermes-cortex/data/loop-events/) to cold storage before deleting.
Test Suite
Every loop-governance module must have corresponding tests in scripts/tests/:
Test file
What it covers
test_loop_scorer.py
Each scoring function with known inputs/outputs, boundary conditions (empty string, very long code, negative values)
The remediation-sensor cron (every 5 minutes) should call verify.sh --quick --json
and check for:
Ollama not running → attempt restart
nomic-embed-text missing → pull model
Config file missing → recreate from defaults
DB missing → recreate schema
Symlinks broken → re-link
When auto-remediation fires, it should also send a Telegram notification so the
user knows the system self-healed.
Architecture Review Action Items
The most recent architecture review (2026-06-24, hc-party) produced these priority
action items for hardening the loop-governance system:
Priority
Item
Status
P1
Create tests/ directory with unit + integration tests (68 tests, all pass)
✅ Complete
P2
Graceful degradation: embed() returns None on failure, all scorers fall back to heuristics
✅ Complete
P3
Wire verify.sh --quick --json into 10-min loop-governance-health cron (auto-restarts Ollama)
✅ Complete
P4
DB retention: vacuum_old_cycles(days=90) added to loop_db.py, called weekly
✅ Complete
P5
Add "TDD scoring workflow" to AGENTS.md (non-negotiable rule #10)
✅ Complete
These items are tracked here for reference. Each should be implemented before the
next version bump.
Adoption Enforcement
The change-test-loop skill's Iron Law includes: "NO CYCLE WITHOUT SCORING AND LOGGING
TO THE GOVERNANCE DB." Every agent working on code in a hermes-cortex-managed repo
must:
Run score-cycle after every completed LEARN→RED→GREEN→REFACTOR cycle
Provide feedback via loop-feedback accept/override when a decision is controversial
Run verify.sh if any component (Ollama, scoring, DB) reports errors
The agent contract section of AGENTS.md enforces this as a non-negotiable rule:
"A cycle that isn't logged is invisible to the self-improvement system. It didn't happen."
Script Naming Conventions
All loop-governance scripts follow a strict naming convention for consistency and importability:
Why not one convention? Python modules must use underscores (import loop_scorer fails with a hyphen). CLI commands conventionally use hyphens (docker-compose, git push, not git_push). The symlinks bridge the two worlds.
When adding a new script:
Name the .py file with underscores in scripts/
Create a hyphenated symlink in ~/.local/bin/
Document both names in this skill
See references/macos-compatibility.md for full platform-specific details (grep patterns, timeout, launchd vs systemd, install flow).
Progress on test output is noisy
Test runner output (pytest, etc.) contains heavy boilerplate — PASSED/FAILED markers, timing info, file paths — that stays nearly identical across iterations even when the code changes substantially. Always compare implementation code for progress detection, not test output.
# Good
score_progress(previous_code, current_code)
# Bad — too much boilerplate, artificially low scores
score_progress(previous_test_output, current_test_output)
Short code samples produce weak embeddings
nomic-embed-text has 2048 token context and works best on content above 100 chars. Code snippets under 50 chars produce noisy, unreliable vectors. When scoring very short snippets, pad with surrounding context or score at the file level instead.
Pass rate beats embedding for completeness
Embedding similarity between test output and spec text is a weak signal — test output and spec are structurally different even when the implementation is perfect. The pass rate heuristic (60% weight) is far more reliable. Always pass pass_pct when available.
User feedback is the only ground truth
The scoring function can self-validate (tests pass → good), but the only real measure of decision quality is whether the user accepts or overrides the STOP/LOOP/MOVE ON decision. Write record_user_outcome() calls after every user-facing decision to build the training dataset for meta-learning.
Don't fine-tune nomic
The 137M parameter embedding model is adequate for semantic similarity at ~50ms per call. Fine-tuning it requires collecting a large, diverse, clean labeled dataset and risks embedding space drift that silently corrupts all future scores. Prefer heuristic augmentation (pass rate, diff size, lint count) over model fine-tuning.
Agent Roles & Orchestrator-Only Crons
The fleet has one orchestrator (Moses) and several agent machines (Titus, Gisu, Joseph,
Kustos). Only the orchestrator runs cross-agent tasks. Regular agents run local-only tasks
(agent-session-cache-build, local-agent-weekly-loop-eval).
Agent registry:~/hermes-cortex/ops/services/agent-registry.json defines each agent's role,
hostname, whether it's server-reachable (accessible), and whether it's the orchestrator
(is_orchestrator). The install-crons.py script reads this registry and compares the
local hostname against the orchestrator's hostname to decide which crons to install.
Orchestrator-only flag: Crons with orchestrator_only: true are skipped on non-orchestrator
machines. This prevents regular agents from running cross-agent health checks or other
orchestrator-specific tasks. Currently no crons are marked orchestrator-only — the flag is
ready for future use (e.g., cross-agent health monitoring).
Health monitor:agent-team-health-monitor.py polls server agents (those with health_url
in the registry) and skips client-only agents (Titus, accessible: false). It only runs on
the orchestrator machine — enforced by the cron installer.
See references/cron-management.md for the full orchestrator detection logic.
Integration with Other Skills
With change-test-loop
After each LEARN→RED→GREEN→REFACTOR cycle, use score-cycle to score progress and log to the database in one command:
Important: Always pass --prev-code-file (or --prev-code) with the code from the previous cycle, not test output. Test runner output has heavy boilerplate that makes progress detection meaningless.
Decision scoring is built in: score-cycle calls full_score() which runs all three scorers, applies the decision matrix, and logs to SQLite. Use the output to steer the loop:
LEARN → RED → GREEN → REFACTOR → [LOOP GOVERNANCE] → LEARN (or STOP)
With subagent-driven-development
After a review→fix→review cycle (spec review fails, implementer fixes, spec review runs again), use loop-governance to detect if this is making progress or spinning:
Same finding 3x → STOP ✗ (no progress, escalate to human)
Progress score ≥ 5 → LOOP 🔄 (keep iterating)
Progress score ≥ 8 → STOP ✓ (review passed, move to next task)
With orch-skill-lifecycle (successor to skill-miner)
skill-miner was removed 2026-08-02 — its collection role is absorbed into
the unified orch-skill-lifecycle pipeline (runs daily 04:34 KST). Fleet agents
now run agent-learning-collector (every 6h) which sends structured Learning
Reports to inbox_orchestrator; the orchestrator's lifecycle cron evaluates
them and patches skills/SOUL.md. See the orch-skill-lifecycle skill for the
full three-phase pipeline (collect → evaluate → upgrade).
With lesson-aware-agent
The loop-governance decision matrix pairs naturally with lesson search:
Before starting a loop, search lessons for known approaches
If a loop keeps failing with no progress, search lessons for the error pattern
Save unsuccessful loop attempts as lessons ("tried X approach, no progress after 5 iterations")
See references/agent-inbox-architecture.md for the full cross-machine
communication design (three-repo separation, message flow, pitfalls).
Efficiency Principles
Score cheaply — nomic embeddings at ~50ms cost less than 1% of an LLM call. Score on every iteration.
Fail fast — no-progress detection after 3 iterations saves potentially infinite wasted rounds.
Bound early — set max_iterations before starting. A rule of thumb: if you can't achieve the goal in 5 iterations, you won't achieve it in 50.
Fresh context — each iteration starts clean. Don't accumulate stale reasoning.
Measure, don't guess — the scoring function replaces "feels right" with quantitative evidence. Trust the score, not the instinct.
