review-open-issues

name	review-open-issues
description	Sweep and prioritize this repository's complete open GitHub issue queue using current generation-run, provenance, instrument, and code evidence. Use for full backlog triage or deciding which issues are genuinely blocking the next D4D generation arm; do not use as permission to close issues, launch billed generation, or implement fixes.
metadata	{"category":"workflow","requires_database":false,"requires_internet":true,"version":"1.0.0"}

Review and prioritize open issues

Produce a complete, dependency-aware triage of the data-sheets-schema open issue queue. The queue, the arm analysis notes, and the provenance records are different surfaces: sweep the queue itself, then test every claim against the current repository and the records that attest what actually ran.

This is a read-only review by default. It does not implement fixes, launch API sweeps or agentic runs, close or edit issues, change labels, or maintain a tracker unless the user separately authorizes that exact mutation.

When to use: the user asks to review, triage, or prioritize issues or the backlog; asks "any open issues affecting generation?" before an arm; or a review pass has just filed a batch of issues that need sorting.

When NOT to use: picking the next unit of work to implement, or acting on a single known issue. This skill produces a ranking, not a fix. It is expensive enough that it should not run on every "what's next" question.

Sources of truth

Use these before relying on an issue title or an old note:

• CLAUDE.md for the pipeline, the canary rule, prompt pins, reasoning capture, and the dataset-scope declaration;
• data/preprocessed/source_manifest.yaml — source_priority, scope:, and naming: are declarations; an issue that contradicts them is an issue about the manifest, not the record;
• src/download/prompts/canonical_hashes.yaml and d4d api prompts check --strict — which prompt text a condition is;
• d4d runs check --strict and each run's *_provenance.yaml — what a run actually consumed (bundle md5, prompt request hash, schema digest, phases, api_usage or run_observed, validation, pair_consistency, grounding, form, report_claims);
• src/data_sheets_schema/schema/d4d_generation_record.yaml — the record contract, and check_record for whether a record conforms;
• .claude/commands/d4d-uniform-rules.md and d4d-full-core.md — the rules and the agentic playbook (the single copies; #563);
• notes/generic_v5_analysis_plan.md, notes/arm_*_results.md, notes/arm_comparison.md (regenerated by scripts/arm_comparison.py) — pre-registered predictions, dated instrument amendments, and the numbers;
• .claude/agents/d4d-rubric*-semantic.md and data/evaluation_llm/*/label_aware/ — the evaluation instruments and what they scored;
• current source, tests, and CI for actual behavior.

Treat issue bodies and titles as claims, not current status. Read comments: this repository records corrections, narrowed residuals, and review findings there. A merged PR is evidence only after its code and the issue's acceptance criteria are checked.

Workflow

1. Fetch the entire queue

gh repo view --json nameWithOwner,url,defaultBranchRef
gh issue list --state open --limit 5000 --json number | jq length
gh issue list --state open --limit 5000 \
  --json number,title,body,comments,labels,createdAt,updatedAt,author
gh label list --limit 200

State the exact number reviewed and whether coverage was complete. Never silently accept gh's default 30-item limit. Read every body and its comments; for a long queue, inspect related groups in parallel but keep one disposition per issue.

2. Build the dependency graph before assigning rank

Place each issue at the earliest pipeline stage it affects:

source download / manifest (scope, priority, naming)
  -> preprocessing and bundles (bundle_md5; audit-bundles)
  -> prompt condition and pin; playbook and uniform rules
  -> generation run (API sweep with lock/resume, or agentic subagent)
  -> provenance record and deterministic checks
       (pair consistency, grounding, form, report claims)
  -> canary gate and baseline (worst-of-arm; UNMEASURABLE ≠ OK)
  -> canonical selection (one canonical per project; supersession)
  -> renders and rubric evaluations (label-aware, evaluator recorded)
  -> arm analysis notes, comparison table, figures, outward claims

An upstream identity or evidence-boundary problem invalidates everything downstream of it; a downstream measurement problem does not touch the records. Group issues that share a root cause without hiding the numbers.

For each issue, record when applicable:

• pipeline stage and which arm(s)/labels/projects it touches;
• the instrument it was measured under (British spellings v2.1 (#653), the undeclared-prefix count (ark/URN handling per #671), the gc_label_variants counter, the report_claims parser) and whether a fix is an instrument change (condition boundary, both sides recomputed) or a defect (fix now);
• whether it affects what gets generated, what gets recorded, or what gets measured — three different urgencies;
• prerequisites, blockers, duplicates, superseding issues;
• cheapest decisive evidence and the acceptance test;
• execution class: read-only audit; local deterministic recompute (backfill-checks without --execute, arm_comparison.py, runs check); one canary run (billed CBORG or one Opus subagent); a full 12-run arm; a schema change requiring make gen-project.

3. Check current reality and staleness

• Search exact issue references with a word boundary — #48 must not match #480:

  git log --all --oneline --perl-regexp --grep '#<N>\b'
  gh pr list --state merged --search '<N>' --limit 100
  

  gh pr list --search matches the number anywhere; open each hit and verify it resolves the issue before citing it.

• Use rg to confirm named paths, commands, flags, and constants still exist (d4d subcommands move; AGENTIC_PHASES, _PHASE_KEYS, PLAYBOOK_ONLY and similar are the kind of thing that gets renamed).

• Compare acceptance criteria with the merged change. A partial fix keeps the issue open with a narrowed residual; say which part is done.

• Distinguish an observation from its action issue; prefer closing a fully recorded observation as superseded when a separate open issue owns the remaining work.

• Verify artifacts by content and provenance, not by prose: a metric without its label, instrument version, and record hash is not a canonical number. Recompute rather than trust where the repo makes that cheap (backfill-checks without --execute, arm_comparison.py).

4. Apply stop-the-line checks

Treat these as P0 when live or externally consequential:

• an evidence-boundary breach: a generation phase that read a prior D4D record, or launch text carrying project-specific content (#422);
• a prompt edited without its pin rotated, or a labelled condition whose record hashes no pinned text (uncanonical, #432/#436);
• a record attesting inputs it did not consume — wrong bundle md5, wrong schema digest, a dropped validation verdict (#396) — or no record at all because the recorder was run outside the repo root (refused since #672; before that it crashed, #659);
• two sweeps writing one label (#513), or a resumed run that skipped an artifact that does not validate;
• an instrument changed mid-condition with only one side recomputed, or a baseline copied from a different arm under a header claiming otherwise (#676-class);
• a gate or table that reports an unmeasured value as a held floor (claims_checked: 0 shown as "0 findings", #684; UNMEASURABLE shown as OK);
• canonical supersession that discards evidence in the live corpus (#677);
• an outward claim (note, deck, README) that a review has shown to rest on any of the above.

Spend figures from api_usage (billed input/output) and run_observed (cache-inclusive runner totals, #681/#682) are different quantities and must never be averaged or ranked together; a total summed from the wrong transcript directory has already misreported one arm (#688). An issue that conflates them is a correctness issue, not a style one.

5. Assign priority and execution order

Priority is consequence; readiness/cost is a separate annotation.

• P0 — stop the line. Corrupted or misattested records, evidence-boundary breaches, unpinned condition text, or an outward claim built on a vacuous floor. Must be resolved before the next arm is launched or the claim cited.
• P1 — affects the next arm. Anything that changes what gets generated or what gets recorded under the current condition: recorder gaps (backfill-checks not inline, #687), launcher tooling whose absence caused a real error (#688), a rule with no norm that produces 3→130 variance (#685), a procedure change with evidence behind it (#689, #694). Note which of these are condition-boundary changes: they are scheduled, not rushed.
• P2 — measurement, selection, evaluation, docs, generalization. Instrument caveats (#674, #675, #684, #691), selection modelling (#660, #677, #690), rubric-agent contracts (#158, #667, #679, #697), HTML/layout conventions (#680), the generalization tracker (#621–#637), long-standing design items (#176, #297, #457).
• CLOSE/UPDATE. Fixed, superseded, duplicate, or already implemented in the file the issue names (check the file — #155 and #161 were closed this way). Cite the commit, PR, code location, or line.

Calibrate P0 sparingly: if more than ~10% of the queue lands there, recheck. Then order within and across tiers by:

1. upstream stages before downstream consumers;
2. what the next arm needs before what the next paper needs;
3. recover already-generated evidence (records, transcripts) before regenerating;
4. read-only and deterministic recomputes before a canary; a canary before a fan-out (the canary rule in CLAUDE.md);
5. instrument changes only at a condition boundary, with both sides recomputed and a dated amendment in the plan note;
6. combine issues only when one patch genuinely satisfies each one's acceptance criteria.

Do not prioritize by age, by a P0 string in a stale title, or by how much generation spend an issue represents.

6. Report

Return a compact report with:

1. coverage: repository, timestamp, number reviewed, completeness;
2. the direct answer to "does anything block the next arm" — yes/no, with the issue numbers;
3. a dependency-ordered P0/P1/P2 table: issue, stage, current status, evidence, blockers, execution class, acceptance test, and whether it is a condition-boundary change;
4. CLOSE/UPDATE candidates with specific evidence;
5. unresolved evidence gaps;
6. a short sequence showing which costly work (canary, arm, instrument change) must wait for what.

Separate measured findings, code inspection, inference, and proposed work. Call out old issues explicitly rather than silently dropping them.

Conventions this skill enforces

• Full-queue coverage, not first-page sampling. State the count and whether coverage was complete.
• Evidence over vibes. Every CLOSE/UPDATE/duplicate cites a commit, PR, artifact, or code location.
• Instrument-change discipline. A finding whose fix changes a counter, classifier, rubric agent, or prompt is a condition-boundary change; say so and do not schedule it inside a running condition.
• Findings become issues (repository rule): a triage that surfaces a new defect proposes an issue; it does not fix it in passing.
• Titles are claims and they drift. Re-read titles at report time.
• The queue moves during the sweep. Re-check the open set immediately before reporting and say so if it changed.
• No @-mentions in comments or reports without explicit per-mention authorization (standing rule).

Measurement discipline

The recurring failure in this repository is mismeasuring, not misreading. Before citing any of the following, confirm how it was obtained:

• Exit codes through pipes. cmd | tail -1; echo $? reports tail's status, not cmd's. Use cmd > /tmp/o 2>&1; echo $? or ${PIPESTATUS[0]}.
• Whitespace-splitting file lists. git status --porcelain | awk '{print $2}' splits a path containing spaces. Use --porcelain -z | tr '\0' '\n'.
• Glob patterns tested by shape. git check-ignore --no-index <path> tests what a pattern does; a regex over the pattern tests what it looks like.
• Stored check blocks vs live recompute. form, grounding, and report_claims blocks carry recorded_by and an instrument note; compare arms only under one instrument (arm_comparison.py recomputes form live).
• A zero is not a floor. report_claims.findings: [] with claims_checked: 0 is unmeasured (#684); grounding.distinct all zero can mean the record states no identifiers at all.
• Two config directories. Subagent transcripts live under ~/.claude or ~/.claude-work depending on the account; a run_observed total summed from one of them undercounted a resumed run once (#688).
• A running agent's transcript is not its result. Wait for the completion notification rather than reading a background agent's output file, which is its raw JSONL.
• Backticks in a double-quoted -m. Write commit messages with a quoted heredoc (<<'EOF') and read them back.
• Truncated tool output. Re-read the cited file at the cited line.

Notes and limitations

• gh issue list --json omits comments unless requested; this repository narrows scope in comments, so a body-only fetch overstates what is open.
• An issue may be addressed in code while its acceptance criteria are not; keep it open with the residual named.
• Some residuals ask for evidence the repository records as unrecoverable (capture_postdates_run, killed invocations' api_usage, #656); say so and recommend superseding rather than leaving them open indefinitely.
• Both arms share the claudecode_agent method directory (#690); do not infer runtime from the directory, read model.agent_runtime or the label.

Related

• d4d runs check --strict, d4d api prompts check --strict, d4d download audit-bundles --strict, d4d download scope --check — the deterministic gates; run them before trusting an issue's "still broken".
• d4d provenance backfill-checks without --execute — recompute a record's check blocks and report them without writing.
• scripts/arm_comparison.py — regenerates the cross-arm table and figures.
• d4d provenance reasoning — distinguishes runtime-cannot-capture from a missing log.

Mutation boundary

Do not close, comment on, relabel, retitle, or create issues during the review. If the user later asks to act, present the exact issue numbers and proposed mutation first; apply closures one at a time with cited evidence. Never treat general approval as authorization for a bulk close.

Do not launch d4d api batch, d4d api run, or an agentic generation subagent as part of triage. A recommended canary is a proposal, not permission to spend.

Merging stays the maintainer's call; do not open cross-repository issues or use @ mentions without explicit authorization.