overnight-worktree-agent-waves

name	overnight-worktree-agent-waves
version	1.0.0
category	coordination
description	Operate unattended overnight or background multi-agent waves with worktree isolation, artifact reconciliation, verification fallbacks, and closure-first batching.
tags	["overnight","worktrees","agents","verification"]

Overnight Worktree Agent Waves

When to Use

Use when launching, monitoring, salvaging, or verifying overnight/background Codex/Codex/Hermes lanes across isolated worktrees.

Class-Level Workflow

1. Partition queues by plan state and dependency risk before launch.
2. Use absolute worktree/log paths and warm required virtualenvs before unattended runs.
3. When the control checkout is dirty or worker contention is unclear, start with read-only Codex -p --permission-mode plan scout lanes using absolute prompt/output paths before creating write-enabled worktrees; see ai/durable-provider-throughput-dispatch reference references/dirty-workspace-plan-mode-scout-wave.md.
4. Treat no-op or stalled runs as recoverable: inspect logs, cwd, prompt files, and artifacts before relaunching.
5. Reconcile artifact placement drift back to canonical issue/review locations.
6. Verify claimed commits against the actual working tree before closing issues.
7. Convert unresolved blockers into explicit follow-up issues rather than leaving stale-open work.
8. When surfacing morning/overnight status, report both live GitHub label counts and artifact-audited readiness counts; do not equate status:plan-review with "ready for approval" unless current plan files and review artifacts show no blocking MAJOR/FAIL/UNAVAILABLE/provider-capacity defects.

Morning Status / Approval-Readiness Summary

Use this pattern when the user asks what happened overnight, how many issues are ready for plan approval, or how many issues were executed:

1. Gather live GitHub state first: query open status:plan-review, open status:plan-approved, and recent closed issues (e.g. last 24h). Preserve clickable issue URLs in the final table.
2. Cross-check local/remote artifacts: inspect prompt-pack results/, generated/, review artifacts, lane logs, tmux sessions, and remote ace-linux-2 result files before counting anything as complete.
3. Separate three counts:
   • labeled status:plan-review count,
   • artifact-audited approval candidates (plans + valid current review artifacts, no blocking verdicts),
   • executed/closed count (issues actually merged/closed or implementation-complete by verified commits/PRs).
4. Call out invalid review evidence explicitly: zero-byte review artifacts, provider capacity failures (e.g. Gemini 429/model capacity), auth failures, sandbox/tool-schema failures, or stale reviews mean "needs review rerun", not "ready".
5. State lane side effects honestly: if the overnight continuation was planning/review/GTM-only, report implementation/close count as zero for that lane even if broader repo issues closed during the same 24h window.
6. End with actionable blockers: list closest approval-prep candidates, issues needing substantive plan repair, and machine/provider constraints affecting the next wave.

Continuous autofeed / next-wave status checks

Use this when a user asks whether long-running batches are still being tracked, whether new runs are being spun up, or for a status update after an autofeed wave.

1. Report tracking before conclusions: list the active monitor/cron jobs, repeat counters, last status, next run time, and delivery target. A drained tmux list is not failure if a monitor is still cycling and result artifacts are appearing.
2. Check four surfaces, not one:
   • tmux sessions/processes for currently running lanes,
   • lane logs for launcher/runtime errors,
   • prompt-pack results/ and generated/ artifacts for completed or autofed work,
   • remote worker logs/results (e.g. ace-linux-2) and rsync state.
3. Treat runner errors as first-class status: if logs contain launcher failures such as Codex: command not found, LOGDIR: unbound variable, or -budget-usd: command not found, classify the original lane as failed/stale even when a monitor or follow-up artifact later produced useful output. Verify which component produced each artifact before crediting the lane.
4. Do not count self-feeding artifacts as approval evidence until verified: auto-generated review summaries, command packs, and follow-up prompts can advance the conveyor, but promotion readiness still requires current live issue state, valid non-stale review artifacts, and legal/provenance checks.
5. For remote non-interactive Codex lanes, ensure the runner exports user-local bins before invoking Codex:

   export PATH="$HOME/.npm-global/bin:$HOME/.local/bin:$PATH"
   

   Prefer an SSH heredoc for remote launches when inline quoting expands local variables too early (symptom: LOGDIR: unbound variable).
6. When auto-feed launches follow-ups, include bounded dedupe rules: no duplicate session/log/artifact names, no status:plan-approved, no outreach, no unapproved implementation, and at most the configured safe number of follow-ups per monitor pass.
7. Harden Codex -p runners with argv arrays when relaunching follow-up waves. If a log shows -budget-usd: command not found, treat it as likely shell line-wrap or stale-runner parsing drift around --max-budget-usd. Replace long single-line invocations with an array runner that uses an absolute/validated Codex binary, closes stdin, and smoke-checks the flag before launch:

    PATH=
   CLAUDE_BIN=
    --version >/dev/null
    -- | grep -q -- 
   cmd=(
      -p
     --permission-mode 
     --no-session-persistence
     --output-format text
     --max-budget-usd 
     
   )
    </dev/null 2>&1 |  -- 
   

Consolidated Session Learnings

The references/ directory contains archived narrow skills absorbed during the 2026-04-29 umbrella consolidation pass. Use the subsections below as the class-level index, then open the named reference when a case-specific recipe is needed.

Absorbed Narrow Skills (2026-04-29)

cron-plus-immediate-hermes-batch-launch

• Session reference: references/cron-plus-immediate-hermes-batch-launch.md.
• Preserved insight: When the user asks for nightly/continuous batches and immediate progress, create scheduled cron jobs plus manual background Hermes sessions, then verify/report both surfaces separately with prompt/log paths and process IDs.

background-Codex-worktree-absolute-path-launch

• Former skill demoted to references/background-Codex-worktree-absolute-path-launch.md.
• Preserved insight: Prevent overnight/background Codex worker launch failures in git worktrees by using absolute prompt/log paths and immediate post-launch polling.

Codex-quota-failover-to-codex-for-overnight-plan-lanes

• Former skill demoted to references/Codex-quota-failover-to-codex-for-overnight-plan-lanes.md.
• Preserved insight: Recover an overnight multi-worktree planning wave when some Codex lanes hit quota by relaunching only the failed lanes with Codex in the same isolated worktrees and prompt files.

closure-first-overnight-batch

• Former skill demoted to references/closure-first-overnight-batch.md.
• Preserved insight: Run a high-leverage overnight batch by clearing stale-open approved issues first, converting shared blockers into tracked issues, and reserving only one lane for true implementation.

large-parallel-planning-wave-environment-failure-handoff

• Former skill demoted to references/large-parallel-planning-wave-environment-failure-handoff.md.
• Preserved insight: Handle large pre-plan-review planning waves that succeed analytically but fail to persist artifacts due to quota exhaustion, sandbox write failures, or cancelled GitHub mutations.

live-state-aware-overnight-implementation-prompts

• Former skill demoted to references/live-state-aware-overnight-implementation-prompts.md.
• Preserved insight: Design overnight implementation prompts that begin with a live repo/CI precheck so workers continue from partial progress instead of replaying stale handoffs.

overnight-plan-artifact-placement-drift-reconciliation

• Former skill demoted to references/overnight-plan-artifact-placement-drift-reconciliation.md.
• Preserved insight: Reconcile overnight planning waves where Codex workers advance GitHub issue state but the expected plan/review artifacts are missing from the designated external worktree because the worker wrote from a sandbox/in-repo worktree and pushed directly to the branch.

overnight-plan-wave-artifact-drift-reconciliation

• Former skill demoted to references/overnight-plan-wave-artifact-drift-reconciliation.md.
• Preserved insight: Reconcile overnight planning waves when Codex workers move GitHub issues to status:plan-review but local artifacts are missing, split across sandbox worktrees, or only present on a pushed remote branch.

overnight-planning-noop-run-salvage

• Former skill demoted to references/overnight-planning-noop-run-salvage.md.
• Preserved insight: Recover when unattended overnight Codex planning runs exit 0 but produce no required artifacts; salvage the wave by auditing existing plan state, generating missing summary artifacts manually, and preserving morning monitoring surfaces.

overnight-pre-plan-review-wave-artifact-drift

• Former skill demoted to references/overnight-pre-plan-review-wave-artifact-drift.md.
• Preserved insight: Run overnight planning-only waves for issues before status:plan-review, and reconcile cases where GitHub state advances but plan/review artifacts land in a sandbox or remote branch instead of the active local worktree.

overnight-verify-close-and-blocker-conversion

• Former skill demoted to references/overnight-verify-close-and-blocker-conversion.md.
• Preserved insight: Use overnight Codex lanes to clear stale-open GitHub issues by verification-first closure, and convert blocked PR-repair attempts into dedicated blocker issues instead of speculative edits.

overnight-verify-close-batch

• Former skill demoted to references/overnight-verify-close-batch.md.
• Preserved insight: Build overnight parallel batches that close stale-open GitHub issues by proving landed work already satisfies the issue, instead of wasting implementation lanes on redoing completed work.

overnight-wave-pack-worktree-isolation

• Former skill demoted to references/overnight-wave-pack-worktree-isolation.md.
• Preserved insight: Safely launch overnight multi-terminal workspace-hub planning packs from isolated worktrees when the main checkout is dirty or prompts share planning/index files.

overnight-worktree-Codex-noop-recovery

• Former skill demoted to references/overnight-worktree-Codex-noop-recovery.md.
• Preserved insight: Recover overnight Codex worktree batches that appear to succeed but produce no artifacts; harden rerun prompts and launch commands.

overnight-worktree-uv-warmup-and-log-path-guardrails

• Former skill demoted to references/overnight-worktree-uv-warmup-and-log-path-guardrails.md.
• Preserved insight: Prevent false stalls and missing-log failures in overnight Codex worktree batches by pre-warming uv environments, using exact log-path directory creation, and interpreting buffered logs correctly.

overnight-worktree-verification-fallback

• Former skill demoted to references/overnight-worktree-verification-fallback.md.
• Preserved insight: Verify overnight multi-worktree Codex batches when auto-sync, duplicate-lane convergence, and sandbox-blocked review create misleading local state or review provenance.

workspace-hub-overnight-plan-monitor

• Former skill demoted to references/workspace-hub-overnight-plan-monitor.md.
• Preserved insight: Monitor and reconcile workspace-hub overnight planning or implementation batches, including process status, result artifacts, issue/commit verification, and controlled failed-lane recovery.

plan-gated-overnight-queue-partition

• Former skill demoted to references/plan-gated-overnight-queue-partition.md.
• Preserved insight: Partition a plan-gated GitHub queue before launching overnight work so ineligible pre-approval issues are routed to planning/review lanes and only approved issues are used for merge-capable execution.

verify-Codex-run-commit-vs-working-tree-before-closing

• Former skill demoted to references/verify-Codex-run-commit-vs-working-tree-before-closing.md.
• Preserved insight: After a Codex implementation run, verify the claimed file set against the actual commit and working tree before treating the issue as fully complete.