workspace-hub-overnight-plan-monitor

name	workspace-hub-overnight-plan-monitor
description	Monitor and reconcile workspace-hub overnight planning or implementation batches, including process status, result artifacts, issue/commit verification, and controlled failed-lane recovery.
triggers	["Monitor Codex/Codex/Gemini overnight planning runs in workspace-hub","Monitor parallel Codex implementation batches launched by Hermes background process sessions","Count completed dossiers under docs/plans/overnight-prompts/*/results or /mnt/local-analysis/overnight-batch-*/results","Check which batch terminal processes are still alive from logs/*.pid or Hermes process session IDs","Summarize final recommendations from dossier markdown files or implementation lane summaries","Reconcile missing central result artifacts when workers wrote `.nightly-results/` inside the repo","Convert failed/no-output/max-turn lanes into blocker notes or narrow recovery prompts"]

Workspace-hub overnight plan monitor

Use this when a cron/job or user asks for a status report on an overnight planning or implementation batch (for example a 10-pack under logs/.../*.pid and docs/plans/overnight-prompts/.../results/*.md, or a Hermes-launched batch under /mnt/local-analysis/overnight-batch-*/{logs,prompts,results}).

Why this skill exists

A few practical quirks showed up during monitoring:

1. read_file() prefixes lines with N|, which can corrupt naive PID parsing if you just regex the first number.
2. search_files(target='files') may return results under files, not matches, so handle both shapes.
3. Cached/deduped file reads can be awkward for tiny PID files; direct Python file reads from terminal/execute_code are often more reliable for read-only monitoring.
4. Dossier markdowns are not uniform; extract recommendations from the Final Recommendation section with tolerant parsing.

Recommended workflow

1. List PID files and result files
   • Use search_files(target='files') on the exact batch directories.
   • For file listings, read from result.get('files', []) first and fall back only if needed.

1a. Verify expected worktree roots before trusting missing artifacts

• For explicit worktree monitoring requests, first test whether each expected root exists and is a git repo/worktree (Path.exists(), .git file/dir, or git worktree list --porcelain).
• If expected worktree roots are missing, say the run is incomplete at the requested locations and list the launched paths for operator inspection.
• Use a bounded exact-path fallback in the main checkout (for example /mnt/local-analysis/workspace-hub/<same-relative-summary-path> and the same prompt directory) to find central salvage artifacts, but classify them separately from the expected worktree outputs.
• Avoid broad rglob/search_files over all of /mnt/local-analysis for summary names unless tightly bounded; it can time out on large workspaces.

2. Check live processes with direct Python/OS calls

   • Prefer a short Python script via terminal or execute_code:
     • Path.glob('*.pid')
     • pf.read_text().strip()
     • os.kill(pid, 0) to test liveness
   • This avoids line-number prefixes from read_file().

3. Count completed dossiers from the results directory

   • Count *.md files in the exact results/ folder.
   • Do not infer completion from prompts outside results/.

4. Map dossier to terminal/issue

   • Terminal name usually comes from the result filename stem.
   • Issue number is best extracted from the markdown metadata table row.
   • Handle both plain and emphasized field labels, for example:
     • | Issue | #2053 |
     • | **Issue** | #2055 |
     • or | Issue | WRK-1113 |
   • Prefer this table over generic regexes, because the body often references dependency issues too.
   • If the value is a markdown link like [ #2063 ](url) or [ #2063](url), normalize it to just #2063 in the report.

5. Extract a one-line recommendation

   • Parse from the ## ... Final Recommendation section.
   • Normalize formatting first:
     • strip leading ###
     • remove surrounding **...**
     • collapse repeated whitespace
   • Use this priority:
     1. Explicit RECOMMENDATION: line
     2. First status heading/line in the section (often bold or ###)
     3. First substantive explanatory sentence or bullet after that heading
   • Skip boilerplate lead-ins when choosing the explanatory text, such as:
     • Rationale: / **Rationale:**
     • Action required: / Action needed:

Reference Python pattern

from pathlib import Path
import os, re

pid_dir = Path('/mnt/local-analysis/workspace-hub/logs/Codex-2026-04-09-10pack')
result_dir = Path('/mnt/local-analysis/workspace-hub/docs/plans/overnight-prompts/2026-04-09-10claude/results')

result_files = sorted(result_dir.glob('*.md'))

alive = []
for pf in sorted(pid_dir.glob('*.pid')):
    m = re.search(r'\d+', pf.read_text().strip())
    pid = int(m.group(0)) if m else None
    is_alive = False
    if pid is not None:
        try:
            os.kill(pid, 0)
            is_alive = True
        except OSError:
            pass
    if is_alive:
        alive.append(pf.stem)

Pitfalls

• Do not parse PID files with a regex against raw read_file() output unless you strip the N| prefix first.
• Do not count prompt files outside results/ as completed dossiers.
• Do not grab the first #205x/WRK-xxxx in the document body; dependency references can mislead issue extraction.
• Keep the task read-only: no file writes, no repo modifications.

Worker-claim verification rule

Do not trust a finished worker's stdout summary by itself.

A worker can claim:

• issue moved to status:plan-review
• plan file written
• review artifacts present

while one of those local artifacts is actually missing from the monitored worktree.

For every completed worker, verify all three surfaces before reporting success:

1. Live GitHub issue state
   • check labels on the issue (status:plan-review, status:plan-approved, etc.)
   • inspect the most recent comments if the worker claimed a plan-summary or blocker comment
2. Canonical local plan artifact
   • confirm the claimed docs/plans/YYYY-MM-DD-issue-NNN-*.md file actually exists in the active worktree
3. Local review artifacts
   • confirm the claimed scripts/review/results/*plan-NNN-* files actually exist in the active worktree

If GitHub state advanced but local artifacts are missing:

• classify the issue as governance/artifact reconciliation required
• do not report it as cleanly verified
• say explicitly: label/comment confirmed, local artifact state not confirmed

This prevents false confidence when a worker comment or completion log overstates what landed on disk.

Implementation-batch reconciliation pattern

When the batch is an implementation/closeout batch rather than a pure plan-review dossier batch, treat stdout as a lead, not proof.

1. Poll Hermes process sessions if no PID files exist

   • Use process(action='poll', session_id=...) for launches created by terminal(background=true).
   • Record status, exit_code, and the output preview.
   • A worker can exit successfully while writing its summary to an alternate sandbox path; a worker can also be superseded by another lane while its original process exits -15.

2. Verify GitHub and commit surfaces for every claimed completion

   • gh issue view <n> --json state,stateReason,labels,comments,url
   • git rev-parse --verify <sha>^{commit}
   • git branch -a --contains <sha> or direct remote containment checks
   • git show --stat --name-only --oneline <sha>
   • Classify separately: issue state, remote landed state, local checkout/worktree state.

3. Reconcile result artifact locations

   • Workers may be sandboxed to /mnt/local-analysis/workspace-hub and unable to write /mnt/local-analysis/overnight-batch-*/results.
   • Search for .nightly-results/<date>-terminal-*-summary.md in the repo checkout and batch worktrees.
   • If the user expects a central batch directory, mirror/copy the missing summaries into overnight-batch-*/results/ and note that this is an orchestrator reconciliation action.

4. Handle failed/no-output/max-turn lanes decisively

   • Do not relaunch blindly.
   • First inspect the lane log, GitHub issue state, and owned-path diff/status only.
   • If the log is just Error: Reached max turns (...) and owned-path status/diff is empty, write a central blocker/current-state note documenting that no owned-path edits landed.
   • Then launch a narrower recovery prompt scoped to the exact owned paths, required validators, and closeout steps.
   • If the lane was intentionally superseded (for example original process exited -15 but another path completed the issue), preserve both a blocker/supersession note and a completion summary so morning review sees why the raw process status looks failed.

5. Handle post-reboot ghost completions

   • After a reboot, Hermes background process state may be gone and the worker log/central summary may be empty or missing even though the implementation commit landed on .

Output checklist

• Result file count
• Completed terminal/issue list
• One-line recommendation per completed dossier or implementation summary
• Alive process count and alive terminal/session list
• For each completed issue: whether GitHub state was confirmed, whether claimed commits are present on the intended remote, whether claimed files appear in the commit, whether local artifacts exist
• Missing or mirrored summary artifacts, including source and destination paths
• Failed-lane classification: max-turn/no-output/superseded/blocked, with blocker path if written
• Recovery session ID/log path if a narrow recovery run was launched
• Explicit monitoring is complete statement when either all expected artifacts exist or no processes remain; otherwise name the exact still-running recovery sessions