-
State-schema preflight is mandatory before the first dossier. Create the unified pyramid paths exactly as specified: 00-index.md is a file at the state root; 01-summary/ and 02-analysis/ are directories; 03-dossiers/ is flat. Validate board-metadata.json against the fields consumed by scripts/pyramid_utils.py before regenerating the pyramid, especially pass_rate_history entries with pass_rate, accepted, and rejected. A plausible state tree with mismatched names or keys can produce an incomplete summary without raising an error.
-
A rollout worker's PASS label is not a validation result. Before recording any baseline or candidate score, inspect the raw output against the rubric yourself. Literal output contracts need literal checks: a worker may claim it used the required headings while emitting bold labels instead of Markdown headings, or claim a permitted verdict while using an alias. Score the artifact, not the worker's self-assessment.
-
Verify claimed skill absence against the candidate artifact before scoring it. A worker may say a routed procedure is missing even when the candidate reference contains it. Search/read the candidate yourself. If the content exists and the task should have routed there, classify the trajectory as infrastructure-invalid and rerun only that task; do not reject the edit based on the worker's false absence claim.
-
Acceptance criteria are conjunctive hard gates, not aggregate suggestions. An edit that improves one mapped task still fails when any user-approved acceptance criterion remains unmet. Preserve the partial gain in the rejection buffer, reformulate it in a later epoch, and do not let a higher aggregate score override a missing rollback, security, causality, or correctness requirement.
-
An expected failure can come from the wrong enforcement layer. Matching an error code or rejection is not proof that the behavior under test caused it. Establish prerequisites and a positive control before scoring the negative case: for PostgreSQL RLS, grant the test role schema/table privileges and prove an authorized same-tenant operation succeeds before accepting 42501 as RLS evidence; otherwise ordinary relation-permission denial can create a false PASS. Apply the same causal check to authentication, network, parser, and policy tests: verify the request reached the intended gate, then verify that gate rejected it. If the trajectory cannot distinguish the causal layer, mark it failed rather than crediting a coincidental error.
-
A validation trajectory must contain the full direct task response. A textual review of whether the skill covers the rubric, or a summary containing only pass counts and self-scores, is infrastructure-invalid even when it sounds correct. Prefer having the worker return the direct user-facing answer while the orchestrator owns the dossier envelope and records task ID, delegation metadata, raw output, failure modes, and parent scoring. Use the five-key worker JSON contract only when an automated consumer genuinely requires it and the model has demonstrated reliable adherence. If one complete response is mechanically separable, preserve it unchanged and record the normalization boundary instead of rerunning solely for envelope or fence contamination; never reconstruct missing or truncated content. See references/validation-trajectory-integrity.md.
-
Verify root-pyramid completeness after every regeneration. update_root_pyramid() can return successfully while its phase discovery omits dossiers whose filename has no hyphen after the phase name (for example, epoch-N-reflection.json). Compare every current 03-dossiers/epoch-N-*.json file against the generated epoch overview; if anything is missing, retain the dossier as the source of truth and record the indexing gap before relying on the L1/L2 navigation.
-
Not separating training and validation sets. The most common and most damaging mistake.
-
Validating a candidate with a task that does not trigger its edit. Before dispatching baseline or candidate workers, map every proposed edit to at least one held-out criterion. A task that lacks the condition the edit governs is infrastructure-invalid for that edit: replace it and rerun its baseline before candidate validation. Record the discarded trajectory and why; do not count a convenient pass as evidence.
-
Skipping the validation gate. The gate is the only thing preventing overfitting to the training batch.
-
Unbounded edits. An optimizer that rewrites the entire skill causes skill drift.
-
Running epochs without plateau detection. After 4 epochs without improvement, run slow-meta.
-
Combined code + skill fixes: Check for a tool bug BEFORE proposing skill edits. Fix the tool, re-run validation, then observe whether skill edits change outcomes.
-
Leaf help must expose inherited safety flags. A CLI may accept global options after a subcommand while leaf --help hides them, causing workers to falsely claim that dry-run, JSON output, confirmation, or bounds are unsupported. Probe the exact leaf help used in the task. If behavior exists but help omits it, fix the shared help contract, run the full tests, rebuild both baseline and candidate copies, and rerun paired validation before scoring the skill edit.
-
Stale test assertions exposed by tool fixes: After applying tool-bug fixes, run the full test suite before validation.
-
Omnibus PR scope creep — verify each claimed issue is actually closed. When a PR body claims to close multiple issues (e.g., Closes #25, #26, #27, #28), verify that the code actually addresses each one. A PR that refactors the infrastructure may declare the issues closed but leave the bash-level phase runners writing to old directories. After merge, check each issue individually — the PR body's claim is a plan, not a guarantee. If issues remain open, file a follow-up or extend the PR before merge.
-
Rollout subagents can go off-skill. Frame training tasks as "using ONLY the skill content" to prevent source code drift.
-
Late delegation results can invalidate a provisional fallback. A local deterministic fallback may finish before delegate_task workers return. Keep the epoch provisional until every dispatched worker has returned or been explicitly marked failed; if late trajectories arrive, record them and revise the reflection/validation artifacts before final reporting.
-
Edit ordering matters when edits overlap in location. Check whether a replace-type edit's old_string includes territory an add-type edit modifies.
-
Revalidate proposal applicability against the post-merge baseline before constructing candidates. A proposal can become stale when an earlier epoch already applied the same wording or when the current baseline contains a stronger equivalent. Never apply a substring replacement blindly: first assert that the exact old_text occurs once and that the intended new state is not already present. If the edit is already present, record it as redundant/rejected and validate the cumulative candidate without duplicating the text. This prevents false regressions caused by malformed validation copies.
-
A validated candidate can become stale before Merge. Late corrections may strengthen the live target after the validation copy was built. Before merging, diff the final candidate against the current target and classify every difference as accepted edit, later correction, rejected edit, or path-normalization artifact. Apply only accepted deltas onto the current target; never replace stronger live text with an older candidate wholesale. Rebuild the post-merge snapshot from the exact installed target and run one mapped regression task against that installed artifact before closing Merge.
-
When baseline pass rate is already 100%, validation measures regression only. Accept edits based on non-regression plus manual verification.
-
Hermes kanban --priority prefers numeric values. Use integers for guaranteed compatibility.
-
Command syntax edits require end-to-end testing. Run every command in a proposed edit's examples before merging. Treat a plausible flag as untrusted until the exact installed binary's --help output confirms it; a server/API capability does not imply a CLI capability.
-
Safety-classifier fixes must prove they introduce no false negatives. Replacing a conservative regex or guard with a parser-like scanner can fix harmless false positives while opening a confirmation bypass. Run paired benign/dangerous cases, adversarial delimiter escape parity, one bounded live read, one blocked-before-side-effect positive control, and an exact-head re-review after any finding is fixed. Use references/safety-classifier-validation.md for the compact gate.
-
Pipeline grammar is part of the command contract. A rollout can understand the safety principle yet compose a shell pipeline incorrectly (for example, placing | gzip outside a --stdin-from-command child, which pipes the parent tool's output instead). Add a held-out task at the exact shell/process boundary, and score producer-failure propagation independently: a wrapper pipeline without pipefail is not a safe PASS.
-
Production-backed rollouts need two independent gates. User-authorized read-only access may establish a real baseline, but validation workers must not start a second process, mutate service/config state, or read secret values. For plan-only tasks, score mutation leakage as a failure even when the proposed procedure is technically correct. Make the target's actual deployment shape, including legacy hybrid layouts such as systemd wrapping a home-directory checkout, explicit in the task context before evaluating discoverability.
-
A greenfield template is an executable contract, not just documentation. A rollout can pass while a canonical template contradicts the skill's own decision rules or contains a field that the current implementation rejects. Before accepting template edits, run the rendered template through the target tool's config/schema validation and one bounded, non-destructive smoke path. Treat official documentation and the running implementation as separate evidence; when they disagree, record the compatibility boundary and prefer the tested implementation for portable defaults.- Atom/molecule scope creep is a rollout signal. Capture structural mismatches as failure modes.
-
The first failure pattern in Reflect is usually the most visible, not the most impactful. Survey the full landscape before converging.
-
seed-board.sh JSON quoting fails on mixed-quote task content. Use Python seed for tasks with special characters.
-
Shell-script phase runners are unreliable — use delegate_task for all rollouts. The run-phase.sh --exec mode frequently returns 0 records written because hermes oneshot calls fail silently (output swallowed by 2>/dev/null). The reflect phase similarly returns {"error": "execution failed"} via oneshot. For rollouts, use delegate_task (handles multi-line instructions correctly, runs tasks in parallel, captures full output). For reflection, write the JSON directly from rollout records rather than relying on the shell script's oneshot prompt.
-
Multi-epoch progression: Epoch 1 adds the what, Epoch 2 adds the how. Design training tasks for multi-epoch awareness.
-
Parallel rollouts via delegate_task speed the pipeline. Use subagents for rollout execution.
-
The "buried content" trap in Propose. Before writing any "add" edit, check whether the target content already exists elsewhere in the skill or its reference files. A pitfall, a paragraph in a different section, or a reference document may already contain the exact rule you're about to duplicate. If the content exists but agents aren't following it, the real fix is prominence (cross-references, hard-gates, blockquote callouts), not duplication. This session produced a clear example: all four initially-proposed "add content" edits were already present in the target skill — just in locations agents weren't reaching during the Procedure. The edits that survived validation were the prominence fixes, not the content additions. Always search the full skill before proposing an add-type edit.
-
The "frame accuracy" trap — buried content is about TEXT, not CONCEPT. A procedurally perfect SkillOpt cycle can still be directionally wrong if the skill's core conceptual framing doesn't match its canonical source or prematurely chooses for the user. The artifact-pyramids Epoch 1 (2026-05-31) is the worked example: all edits passed validation and merged cleanly, but the skill's layer naming directly contradicted the published canonical article. A second form appears in infrastructure skills that cover several valid operating models: giving one path most of the procedural prominence, or placing a development-only warning where it can be read as applying to production self-hosting, silently turns documentation structure into a recommendation. Prevent this before Epoch 1: verify canonical terminology and relationships; enumerate each supported operating model independently; scope warnings to the exact model they govern; and add a neutral discovery task where user intent is unknown plus a held-out task where the user explicitly prefers the less-prominent option. Score whether the agent discovers intent and constraints before recommending, not merely whether every option is mentioned somewhere. Frame misalignment is not fixed by polishing the favored path; preserve it in the baseline, measure it, then make the smallest structural correction that restores accurate, user-intent-led routing.
-
Version humility — don't over-version session-young skills. A skill created in the same session is at best v0.0.x, not v2.0. The version number should reflect the skill's maturity and iteration count, not the significance perceived in the moment. A skill that's existed for 3 hours and gone through 3 revisions is v0.0.3. Reserve minor bumps (0.x.0) for features proven across multiple sessions, and major bumps (x.0.0) for stable, cross-session releases. The artifact-pyramids Epoch 1 (2026-05-31) is the worked example: the PR was titled v2.0 for a skill created the same day, when v0.0.3 was the honest version. This inflates expectations and confuses the version narrative for anyone reading the repo history. Mirror-image pitfall: updating the PR/commit title but not the body. When you correct a version (or name, or framing) in a PR title, audit every other occurrence — the body, the commit message, the branch description, any linked issues. The artifact-pyramids Epoch 1 correction (v2.0 → v0.0.3) is the worked example: the gh pr edit --title fix changed the headline but left the body describing the PR in "v2.0" language, which the user caught as incomplete. Fix: after any title-level correction, re-read the full PR body and commit message for stale references before declaring done. A partial fix that leaves contradictions in the artifact is worse than no fix — it creates a state where two different versions claim authority in different places of the same record.
-
Style-catalog optimization must validate routing and complete rendered coverage. For a creative skill that recommends reusable named visual styles, begin with text-only tasks that request one named treatment, a use-case rationale, and a style-only prompt clause. Span distinct registers across training and held-out tasks, and score stable naming, fit, and reproducible visual grammar. Keep scene content separate from style. After the user approves the catalog and criteria, generate and pixel-review at least one accepted example for every named treatment before claiming the catalog is validated or complete. A representative subset validates only those sampled treatments; use subset sampling only when the user explicitly scopes the deliverable that way. Recommendation tasks validate discoverability and routing; rendered examples validate treatment fidelity and full-catalog coverage.
-
Creative-skill acceptance criteria need user validation too. When running SkillOpt on skills with subjective outputs, the acceptance criteria you write for validation tasks may be wrong — not too strict for the agent, but too strict for what the user considers "good." The image-magnus919 Epoch 2 run is the worked example: a "~30 word max" SUBJECT criterion was proposed, and the user corrected it as contradicting the rich, detailed subjects that produce the best covers. Prevent this by including the proposed criteria in your Proposals presentation. Ask: "Here's how I plan to evaluate success — are these criteria right?" before running the validation gate. This is distinct from the "buried content" trap, which is a content-discovery issue — this is a criteria-quality issue.
-
Style probes must not smuggle in unwanted identity transformations. A style-validation scene that also ages, de-ages, slims, enlarges, sexualizes, injures, or otherwise transforms a known subject cannot distinguish style fidelity from portrayal failure. The magnus-image Epoch 2 run is the worked example: an Analog Dread probe requested an independently aging reflection; the image followed the task but produced an unflattering older Magnus, so the criterion was wrong even though the rendering matched it. Keep known subjects at their natural age and proportions unless the transformation itself is the user-approved behavior under test. If the user rejects the portrayal, mark the trajectory criteria-invalid, exclude it from final metrics, replace the task, and rerun only the paired baseline/candidate trajectory and generated artifact.
-
Creative-skill rollouts cannot verify style fidelity. When running SkillOpt on creative/image-generation skills, rollout subagents produce prompts and submit them to the model — but neither the subagent nor any deterministic check can verify whether the output actually LOOKS like the skill's defined style. The subagent can verify that the workflow steps were followed (template used, size correct, format correct) but not whether the image matches the intended aesthetic. Style fidelity is a human-in-the-loop verification. Accept this limitation: creative-skill SkillOpt improves the PROCEDURAL quality (subject derivation, error recovery, path selection) but not the STYLE quality, which was already correct in the locked template. The two dimensions are independent.
-
"Reverse" progressive disclosure — extracting content out, not adding in. SkillOpt is usually additive (adding rules, examples, warnings). But when a skill is already very large (30KB+), the most impactful optimization may be REFACTORING: moving inline content to reference files, replacing it with a summary table and cross-reference, and loading the full detail on demand. The groktopus-branding Epoch 1 run is the worked example: 5 full cover prompt templates (~200 lines) were moved to references/cover-prompt-templates.md and replaced with a 5-row summary table. Rules of thumb: if inline content is more than ~50 lines and is only needed in a specific subset of task types, it's a candidate for extraction. If it's general procedure or hard rules, keep it inline.
-
Multi-skill sessions reveal methodology gaps. Running SkillOpt on multiple skills in one session (hyperframes → image-magnus919 → groktopus-branding) surfaces patterns that a single-skill run wouldn't show: the "buried content" trap recurred across all 3 skills, the progressive disclosure principle was validated in all 3, and the creative-skill limitations (style fidelity, acceptance criteria iteration) were only visible because a creative and a procedural skill were optimized back-to-back. When the user engages on a multi-skill session, capture cross-cutting observations in the meta-reflection document at references/multi-skill-patterns.md after the second skill completes, rather than waiting for a separate slow-meta phase.
-
Validation via delegate_task works better than oneshot for prose-heavy edits. Subagents can read the full skill context.
-
Authoritative-source gates require source-quality reconciliation, not citation-shaped output. A worker may report PASS after citing a plausible tertiary source, a generic reference table, or its own arithmetic. When a task requires an authoritative source of truth, the verifier must inspect the evidence ledger and confirm that the source is claim-specific, directly retrieved, appropriate to the claim type, and recorded with URL, access date, and scope. A generic or tertiary source is a failed authority gate, not partial proof; reconcile the worker's self-score before recording the rollout result. Treat every conjunct in a compound claim independently: a direct source for a product's price does not validate its launch date, availability, performance, or any other joined assertion. The de-spin pilot's aluminum-versus-titanium claim is the worked example: Wikipedia and Engineering Toolbox made the ratio plausible but could not establish Apple's unspecified exact alloys.
-
A review verdict is a hypothesis, not ground truth. Before a FAIL blocks merge, reconcile each API-contract finding against the current official documentation and live public schema. If a reviewer proposes a field the schema does not expose, while official docs demonstrate the existing call with the claimed identifier form, reject that finding with captured evidence rather than patching toward a nonexistent API. Fix only validated findings, rerun the deterministic gates, then re-review the exact new head. Preserve the false finding and rebuttal in the run artifacts so later reviewers do not resurrect it.
-
Retrieval must govern emission, not decorate it. A worker can retrieve official pages and still invent the bridges between them: hardware capability becomes framework support, a failed search becomes feature absence, command help becomes proof of safe execution, or an unknown component becomes a “typical” archetype. Require the pre-response ledger in references/evidence-governed-execution.md, then score every material claim and command precondition in the raw response. During validation, require the compact observable gate result defined there; an internal ledger that never appears in the trajectory cannot be audited. A ledger written after prose is evidence-shaped rationalization, not a gate.
-
A failed discovery path is not an absent canonical source. A guessed URL, redirect, blocked result, or noisy publisher search proves only that route failed. Before scoring an Unsupported/Unknown result on authoritative-source absence, require a publisher-scoped discovery query using the exact entity and factual anchors, retrieve the specific canonical candidate, and record the query or URL. The evaluator must independently test claimed absence when a direct source could reasonably exist; otherwise a worker can turn a search failure into a confident false negative.
-
Do not assign causal credit from a later stochastic pass. If a proposed edit fails its mapped held-out task and is removed, a later final-candidate worker following the same behavior without that edit does not retroactively validate it. Record the edit as rejected; the later pass establishes only that the current candidate can pass, not why. This prevents reverse-engineering a success story from variance in model behavior.
-
A passing cumulative candidate does not override isolated-edit rejection. When the all-edits copy beats baseline but every component edit fails its independent gate, do not merge the bundle. The cumulative result reveals a possible interaction, not attributable accepted edits. Preserve it as an interaction hypothesis for the next epoch, reformulate the coordinated behavior as one bounded proposal, and validate that proposal independently. This keeps the Merge phase limited to edits with causal credit under the declared acceptance rule.
-
Equal hard-pass counts can hide failure substitution. Compare criterion-level failure identities, not only each task's boolean PASS/FAIL or aggregate hard-pass rate. A candidate can remove one defect from an already-failing task while introducing a different hard defect, leaving the task marked FAIL in both baseline and candidate. That is still a regression, not “no hard-outcome change.” Record the baseline and candidate failure sets per task; reject when the candidate adds a new hard failure even if the count is unchanged. The Supabase recovery probe is the worked example: a baseline leaked the pgsodium root key with cat, while candidates replaced that defect with invented time thresholds or unnecessary key regeneration. Neither is acceptable merely because all versions remained FAIL.
-
Late worker results are still part of the gate. Background rollout or validation workers may finish after the orchestrator appears done. Do not close the epoch or discard the fallback record until every dispatched batch has returned. Reconcile late PASS/FAIL results into the phase dossiers; if a held-out task fails, open a corrective epoch/PR rather than treating the earlier merge as final.
-
User-facing completion is gated on worker reconciliation. Do not send a final or completion-style response while delegated rollout or validation batches remain active. A progress update may say that work is underway, but it must not imply the phase or run is complete. After the async result arrives, record it, reconcile the metrics, and only then report the phase status.
-
Workflow continuity is part of user intent. When the user says “next” immediately after a SkillOpt status report, advance the active SkillOpt phase or epoch. Do not reinterpret it as a generic queue, inbox, or unrelated follow-up command. If a merge gate is blocked, “next” means the next permitted optimization action, not an unrelated task.
-
Version inflation — don't call a third revision "v2.0". A skill created this session, patched once, then re-aligned is at most v0.0.3. The version number should reflect the skill's actual maturity, not the scope of your latest change. If the skill was born today, it's 0.x.y — even if the change feels significant. "v2.0" belongs to a shipped, proven product at a real architectural boundary.
-
Cross-profile write guard on profile-local skills. When the target skill lives in a different Hermes profile's skills directory (~/.hermes/profiles/<name>/skills/), skill_manage(action='patch') calls will be blocked by the cross-profile soft guard. The tool returns an error: "Editing another profile's skills/ will affect that profile's future sessions." To proceed after confirming the user intends to edit that profile's skills, add cross_profile=True to the skill_manage call. This applies to Merge phase edits when the SkillOpt target is in a non-default profile. The non-git reference (references/non-git-skills.md) covers the "no git repo" case; the cross-profile guard is a separate concern that co-occurs when the skill lives under profiles/<name>/.
-
Orchestrator context gap — delegate_task subagents don't load the target skill. When a SkillOpt validation task requires the subagent to USE a skill (not just edit it), the subagent invoked via delegate_task does not load profile-scoped skills — it has only the orchestrator's context string. The skill's procedural instructions (folder structure, format rules, quality gates) are invisible. Fix: include the skill's critical procedural rules directly in the delegation context string. A bare "load the skill" instruction is insufficient — the context must contain the actual structural requirements. This is especially important for skills that define folder hierarchies or placement rubrics.
-
Delegation-context integrity is part of the validation gate. Before dispatching a rollout or validation worker, verify every absolute target path, candidate path, task identifier, quoted claim, source URL, and timestamp/line-range anchor in the delegation context. Preflight name collisions before the first rollout: if an installed/profile skill shares the target's slug or command vocabulary, run one candidate-specific help/dry-run identity probe, record a stable output fingerprint, and reuse one stored delegation template with only {skill_root} substituted for baseline versus candidate. Name collisions need an executable allowlist: a worker may silently load or invoke the sibling even when given the candidate's absolute SKILL.md path. Pin the exact candidate executable and permitted subcommands, explicitly forbid sibling scripts, and inspect the raw response for wrong command names or paths. When collisions recur, require one candidate-specific identity probe from its --help or dry-run output before accepting the task response. Treat forbidden source-code function names or line references as the same identity failure when the worker was limited to skill prose and CLI output. Classify any wrong-skill trajectory as infrastructure-invalid and rerun only that task with the tighter allowlist. Trace every referenced rule to its source and include all rules needed to score the task, not merely a section heading or filename. A worker who receives a mutation workflow but not the skill's First Use credential boundary, or an API-reference label but not its plan/authorization caveat, is evaluating a synthetic incomplete skill. Classify that trajectory as infrastructure-invalid, discard it from the metrics, and rerun the same task with corrected context. For candidate validation, state explicitly whether the worker should evaluate one isolated edit or a cumulative candidate so results cannot silently mix edits.
-
Do not inject a proposed edit into the baseline prompt. Shared delegation context may restate only rules already present in the baseline skill. Candidate-specific behavior must arrive through the candidate skill copy, not through an orchestrator summary such as “never infer absence” or “unknown values stay unknown.” Render both contexts from one stored template with only a {skill_root} substitution, normalize that substitution away, and assert the prompts are otherwise byte-identical before dispatch. If the baseline worker receives the behavior being tested, the baseline and candidate no longer differ at the intervention boundary; discard those trajectories and rerun with neutral common context. Rubric criteria may be visible when they describe observable outcomes without revealing the intervention. Keep exact candidate-specific mechanisms evaluator-only: if the proposed edit teaches throws_ok(..., '42501', ...), a baseline prompt that names that function and SQLSTATE has already taught the solution. Give workers the scenario and required outcome; store exact implementation checks in the parent-side scorer. If either baseline or candidate saw a solution-bearing rubric, exclude both trajectories for that task and rerun them with the same neutral prompt.
-
New file path resolution during validation. When a proposed edit creates a new file (reference, template, script), the validation copy of the skill sits at a temporary location (e.g., /tmp/hermes-skillopt-<name>/skill-validated.md), not in the real skill directory. Relative paths from the new file — especially ../ traversal or sibling-directory references — will not resolve against the temp copy's location. The crawl4ai Epoch 2 run (2026-06-03) is the worked example: the new references/example-repository-reference.md file was created in the real skill directory, but the validation copy's link pointed to references/example-repository-reference.md relative to /tmp/..., which didn't exist. Fix: when preparing validation copies for edits that create new files, either (a) mirror the real skill's directory structure in the temp location, or (b) include the relative path resolution in the validation context so subagents know where to look.
-
SKILL.md vs. reference drift — procedural content in two places diverges. When a skill's SKILL.md and its reference files both contain procedural instructions (checklists, format rules, quality gates), they drift apart over time as one is updated but the other is not. Prevention: keep ALL procedural "this is what you MUST do" instructions in the SKILL.md. Reference files should contain only explanatory content, worked examples, and additional context. If a checklist or format rule appears in both, pick one source and delete the duplicate. The Epoch 2 run on the technical-architect profile (2026-05-31) is the worked example: the quality gate was updated in architect-pyramid/SKILL.md but the reference file retained the old checklist, requiring a separate sync edit.
-
Version convention — doc edits are patch, new features are minor. Guidance and workflow additions are fixes regardless of section size.
-
Repository schema outranks SkillOpt metadata conventions. Before adding a version field or any other frontmatter metadata during Merge, run the target repository's real validator and inspect its allowed schema. If the validator rejects the proposed metadata, reject the SkillOpt proposal; do not weaken the validator or force the convention into the target skill. Record the rejected edit in slow-meta.
-
Worker fallback must preserve the evaluation claim. If delegated rollout workers fail to return usable trajectories, do not fabricate rollout results or label deterministic checklist checks as equivalent execution evidence. Use the bounded fallback only when necessary, explicitly record the limitation, and distinguish synthetic/source validation from real task-execution validation in the final report.
-
Cross-skill edits require format-aware old_string matching. An old_string that works for SKILL.md may fail with different indentation in a script.
-
REQUIRED: Proposals need user review before validation — unless the user explicitly waives that gate. Present all proposals conversationally before running validation by default. If the user gives a clear directive such as "run the epochs end to end autonomously" or "don't bother with review and approval," record the proposals as artifacts and continue through Validate, Merge, and Report without pausing. The waiver applies to the approval pause, not to the edit budget, held-out validation, artifact recording, or verification requirements. Edits affecting evaluation methodology remain sensitive and should still be bounded and explicitly recorded.
-
After user sign-off or explicit waiver, drive autonomously. Once the user reviews proposals and gives a directive signal ('go', 'proceed', 'yes'), or explicitly waives proposal review, execute the remaining phases (Validate, Merge, Report) without stopping for permission at each gate. Do not re-present proposals. Do not ask "should I run validation now?" after the user already said yes. The directive is a commit signal. This is distinct from investigation, where switching technical approaches still requires user input unless the user has explicitly authorized autonomous execution for the whole run. Once the plan is approved or waived, execute it.
-
REQUIRED: Before switching technical approaches during investigation, stop and ask. This applies to: changing model providers, swapping infrastructure strategies, modifying config approaches, altering test parameters (timeouts, batch sizes, formats), or switching debug techniques. Do not silently iterate through approaches — present the situation and ask before pivoting. The user explicitly considers this non-negotiable: "this pisses me off when you change directions without asking me first." This is distinct from the Proposals step (which covers review of planned edits) — this covers the investigation phase where you're still figuring out what's wrong. Common triggers: assuming a timeout value without checking ("60s is short timeout"), deciding to trim skills without consultation, switching between Docker/SSH/local approaches without a check-in.
-
REQUIRED: Completing an epoch means completing ALL six phases in order — not just the last two. Rollout → Reflect → Propose → Validate → Merge. Editing scoring scripts and running validation IS NOT completing an epoch. The user must see the rollout results, participate in reflection, and review proposals before validation runs. Pushing edits to main without this cycle means the epoch didn't happen. This is the "epoch completion theater" failure mode: the agent mistakes implementation work for process completion. The Groktobench HARP Epoch 1 attempt (2026-05-30) was declared complete despite none of the first four phases having been done with the user — it was edits to main, then validation, then an Epoch 2 PR that was actually just more implementation. The user's corrective signal: "You didn't even do epoch 1 yet. We are starting from ground zero here." The fix: hard reset main to last tag, delete all unmerged branches, close associated PRs/issues, rebuild the evaluation environment from clean state, and start from Rollout.
-
REQUIRED: Test definitions must use only upstream Hermes skills. Probes and validation tasks must reference only skills that ship with stock Hermes Agent. Custom or environment-specific skills (Jasper, user-created) make evaluations non-reproducible. Test: if someone clones the benchmark and runs it against stock Hermes, would all referenced skills resolve? If not, it's not portable.
-
Git workflow for epoch changes — branch, PR, merge. All code changes from SkillOpt (scoring edits, probe definitions, config changes) must go through a branch-PR-review process. Pushing directly to main bypasses user review. The user owns version numbering, release timing, and methodological direction. Stop after creating the PR — the merge is the user's call.
-
Epoch stacking preference — ask before branching per epoch. Some users prefer one PR per epoch (branch from main, PR, review, merge, repeat). Others prefer to pile all epochs onto a single PR and merge once at the end. When running consecutive epochs, ask: "New branch/PR for this epoch, or pile onto the existing one?" Don't assume either approach. If the user says "pile onto the same PR," commit and push to the existing branch without creating a new one.
-
Merge phase applies changes to working copy only — they are unstaged and uncommitted. Phase 5 (Merge) applies accepted edits to the skill document in-place using patch/write_file on the local filesystem. After Merge, git status shows modified files on whatever branch the working tree happens to be on (frequently main). These changes have no branch context, no commit history, and no PR — they are invisible to git's history until explicitly committed. This creates a gap between "SkillOpt says epoch complete" and "the change is actually in version control." The fix: immediately after the Merge phase of the final epoch, commit the working changes to a feature branch and open a PR:
git checkout -b feat/skillopt-<skill-name>-e<N>
git add path/to/skill/
git commit -m "fix: SkillOpt Epoch N — <skill-name> optimization"
git push -u origin HEAD
gh pr create --base main --head feat/skillopt-<skill-name>-e<N> --title "fix: SkillOpt Epoch N — <skill-name> optimization"
Do not skip this step. The "Done" column on the kanban board is not a substitute for a git commit. If you complete the kanban pipeline without committing, the improvements exist only in your current session's working tree — they vanish on checkout, reset, or another agent editing the same file.
