assess-strat

name	assess-strat
description	Assess strategies against quality criteria. Pass a strategy file path, directory of strategies, or a strategy key.
allowed-tools	Read, Write, Edit, Glob, Grep, Bash, Agent, TaskGet

Usage

/assess-strat artifacts/strat-tasks/RHAISTRAT-1469.md
/assess-strat artifacts/strat-tasks/
/assess-strat RHAISTRAT-1469

Instructions

Plugin Root

When this skill is invoked, resolve the absolute path of the plugin root directory. This SKILL.md is at <plugin_root>/skills/assess-strat/SKILL.md — the plugin root is two levels up. Determine this path once at the start and use it for all script and file references. Store it as {PLUGIN_ROOT} for substitution into commands and agent prompts.

Rules

• No shell pipes or compound commands. Run all scripts as simple commands (e.g., python3 {PLUGIN_ROOT}/scripts/setup_run.py /path/to/strategies). Do not use |, &&, ;, 2>/dev/null, or redirects. The Bash tool returns command output as a string — parse it programmatically in your logic, not with sed/awk/wc/grep pipelines.

Architecture

Single-input mode scores one strategy from a file path. Batch mode scores all strategies in a directory. Results are saved as individual files in a timestamped run directory. Strategies are local markdown files — no Jira fetch needed.

Directory Structure

assessments/                             # in the plugin directory (persistent)
  20260415-143000/                       # timestamped run
    RHAISTRAT-1469.result.md
    queue.txt                            # pending keys (managed by next_batch.py)
    strat_dir.txt                        # path to source strategy directory
    scores.csv                           # generated by parse_results.py when complete
  current -> 20260415-143000             # symlink to active/latest run

/tmp/strat-assess/single/                # single-mode temp files
  RHAISTRAT-1469.result.md

Single input (file path or strategy key)

Detect the input type:

• File path (starts with / or ./ or ~, or contains .md): Use the file directly.
• Strategy key (matches [A-Z]+-\d+ or STRAT-\d+): Search for the corresponding .md file in common locations:
  1. artifacts/strat-tasks/{KEY}.md
  2. .context/strat-tasks/{KEY}.md
  3. Search with Glob: **/{KEY}.md

Then assess:

1. Run python3 {PLUGIN_ROOT}/scripts/prep_single.py {KEY} to clean up stale result files and ensure the output directory exists.
2. Spawn one background agent (model: opus, run_in_background: true, subagent_type: assess-strat:strat-scorer) with the launch prompt below, setting {DATA_FILE} to the resolved strategy file path and {RUN_DIR} to /tmp/strat-assess/single.
3. Read the result from /tmp/strat-assess/single/{KEY}.result.md, wrap it with a header, and present it to the user.

Batch (directory of strategies)

Phase 0: Preflight checks.

• Run python3 {PLUGIN_ROOT}/scripts/preflight.py {STRAT_DIR} to check strategy directory and current run state. Parse the output:
  • STRAT_DIR_EXISTS=true/false — if false, ask the user for the correct path
  • STRAT_COUNT=N — number of strategy files found
  • CURRENT_RUN=path/none, CURRENT_ASSESSED=N, CURRENT_COMPLETE=true/false — existing run state
  • If there is an incomplete current run (CURRENT_COMPLETE=false), inform the user it will be resumed.

Phase 1: Set up run directory.

• Run python3 {PLUGIN_ROOT}/scripts/setup_run.py {STRAT_DIR} (add --limit N if the user requested a subset).
• The script handles all resume logic and outputs:
  • RUN_DIR=<path> — the absolute path to use for this run
  • STRAT_DIR=<path> — absolute path to source strategies
  • PENDING=<count> — number of strategies to assess
  • QUEUE_FILE=<path> — path to the queue file
• Parse the output to get {RUN_DIR}, {STRAT_DIR}, and {PENDING} count. Do NOT try to memorize or generate the key list yourself — the queue file is the single source of truth for which keys to process.

Phase 2: Assess with a pipeline of 30 concurrent agents.

• Use next_batch.py to get keys from the queue. Never generate key sequences yourself — always get keys from the script.
• Launch loop: Repeat until BATCH_SIZE=0 (queue empty):
  1. Run python3 {PLUGIN_ROOT}/scripts/next_batch.py {RUN_DIR} --batch-size 30 to pop the next batch of keys. Parse the output:
     • BATCH_SIZE=N — number of keys in this batch (0 = queue exhausted)
     • REMAINING=N — keys still in queue after this batch
     • Keys listed after the --- separator
  2. Launch one agent per key (model: opus, run_in_background: true, subagent_type: assess-strat:strat-scorer) with the launch prompt below.
  3. Wait for agents to complete (poll every 30 seconds), then loop back to step 1 to pop the next batch.
• Active polling: Poll running agents every 30 seconds. Do not passively wait for completion notifications — they can be missed.
• Progress checking: Run python3 {PLUGIN_ROOT}/scripts/check_progress.py {RUN_DIR} to get COMPLETED=N, TOTAL=N, REMAINING=N.

Phase 3: Generate CSV and present results.

• Run python3 {PLUGIN_ROOT}/scripts/parse_results.py {RUN_DIR} to parse all .result.md files and generate {RUN_DIR}/scores.csv. The presence of scores.csv marks the run as complete.
• Run python3 {PLUGIN_ROOT}/scripts/summarize_run.py {RUN_DIR} to produce the full summary analysis. Present the output to the user.

Agent Launch Prompt

The coordinator MUST substitute all placeholders with actual values before passing this prompt to the agent:

You are a strategy quality assessor. Your task:
1. Read `{PROMPT_PATH}` for the full scoring rubric.
2. Follow its instructions exactly, substituting {KEY} for the strategy key and {RUN_DIR} for the run directory. Read the strategy from {DATA_FILE} (not the path in the rubric's step 1).
Strategy key: {KEY}
Data file: {DATA_FILE}
Run directory: {RUN_DIR}

Placeholders:

• {PROMPT_PATH} → absolute path of {PLUGIN_ROOT}/scripts/agent_prompt.md
• {DATA_FILE} → for batch: {STRAT_DIR}/{KEY}.md, for single: the resolved file path
• {KEY} and {RUN_DIR} → actual values

This ensures every agent reads the identical rubric from the single source of truth.

Coordinator Output Format

Single strategy — wrap agent output with a header:

## Strategy Assessment: RHAISTRAT-1469
[agent output]

Batch — after Phase 3, present the summary analysis from the CSV to the user. Include:

• Total assessed, verdict distribution (approve/revise/split/reject)
• Score distribution
• Criteria averages and zero-score counts
• What-if analysis (e.g., "if Testability 0→1, N more would approve")
• Near-miss strategies (close to approval)

Scripts Reference

Script	Purpose
preflight.py	Checks strategy directory and current run status
setup_run.py	Creates timestamped run directory with resume support
agent_prompt.md	Full scoring rubric and instructions for assessment agents — use verbatim
next_batch.py	Pops the next N keys from the queue file; ensures each key is processed exactly once
check_progress.py	Reports completed vs total strategies for a run directory
parse_results.py	Extracts scores from .result.md files into scores.csv with verdicts
prep_single.py	Cleans up stale result files for a key in /tmp/strat-assess/single/
summarize_run.py	Produces summary analysis from scores.csv: verdict rates, criteria averages, what-if analysis, near-misses
export_rubric.py	Exports the scoring rubric to artifacts/strat-rubric.md

Required Permissions

Add to your user or project .claude/settings.json:

{
  "permissions": {
    "allow": [
      "Bash(python3 <PLUGIN_PATH>/scripts/preflight.py:*)",
      "Bash(python3 <PLUGIN_PATH>/scripts/setup_run.py:*)",
      "Bash(python3 <PLUGIN_PATH>/scripts/next_batch.py:*)",
      "Bash(python3 <PLUGIN_PATH>/scripts/check_progress.py:*)",
      "Bash(python3 <PLUGIN_PATH>/scripts/parse_results.py:*)",
      "Bash(python3 <PLUGIN_PATH>/scripts/summarize_run.py:*)",
      "Bash(python3 <PLUGIN_PATH>/scripts/export_rubric.py:*)",
      "Bash(python3 <PLUGIN_PATH>/scripts/prep_single.py:*)",
      "Bash(mkdir:*)",
      "Bash(ls:*)"
    ],
    "additionalDirectories": [
      "/tmp/strat-assess",
      "<PLUGIN_PATH>"
    ]
  }
}

<PLUGIN_PATH> is a placeholder — manually replace it with the absolute path to this plugin (e.g., /Users/you/devel/assess-strat) before adding to your settings.

$ARGUMENTS