test-plan-create

name	test-plan-create
description	Generate a test plan from a strategy (RHAISTRAT or RHOAIENG issue), with optional ADR for extra technical depth. Use when starting test planning for a new RHOAI feature with a defined Jira strategy.
argument-hint	<JIRA_KEY> [ADR_FILE_PATH]
user-invocable	true
model	opus
allowedTools	["Read","Write","Bash","Glob","Skill","AskUserQuestion"]

Test Plan Generator

Generate a complete test plan for a RHOAI feature based on a refined strategy, and optionally an ADR document for additional technical depth.

Usage

/test-plan-create <JIRA_KEY> [ADR_FILE_PATH]

Examples:

• /test-plan-create RHAISTRAT-400
• /test-plan-create RHOAIENG-48676
• /test-plan-create RHAISTRAT-400 /path/to/adr.pdf

Inputs

From arguments

Parse $ARGUMENTS to extract:

1. First argument (required): Jira key - either a RHAISTRAT strategy (e.g., RHAISTRAT-400) or RHOAIENG issue (e.g., RHOAIENG-48676)
2. Second argument (optional): Local path to an ADR document (markdown, text, or PDF)

Auto-detection

If no arguments are provided and a strategy file was just generated by /strat.create or /strat.refine in this session, use it automatically — proceed directly to Step 1.

Interactive fallback

If no arguments are provided and no strategy file is available from the current session, ask the user for:

Required

1. Jira key: Strategy or issue key (e.g., RHAISTRAT-400, RHOAIENG-48676)

Optional

1. ADR file path: Local path to ADR document (markdown, text, or PDF) for technical details
2. ADR link: Google Doc URL (stored as reference in metadata, not fetched)
3. Feature directory name: snake_case name for the feature directory (e.g., mcp_catalog). If not provided, derive from the feature name.

Process

Step 0: Pre-flight Checks

0.1 Python dependencies

Install the test-plan package (makes all scripts importable):

(cd $(git -C ${CLAUDE_SKILL_DIR} rev-parse --show-toplevel) && uv sync --extra dev)

If installation fails, inform the user and do NOT proceed. Once installed, all Python scripts will work from any directory.

0.2 Jira Environment Variables

Verify that Jira API credentials are configured via environment variables:

(cd $(git -C ${CLAUDE_SKILL_DIR} rev-parse --show-toplevel) && uv run python -c "from scripts.jira_utils import require_env; [require_env(v) for v in ('JIRA_URL','JIRA_USER','JIRA_TOKEN')]")

If the check above exits non-zero, STOP immediately. Do NOT proceed with the rest of the skill. Do NOT attempt alternative data sources (MCP tools, cached files, web fetches, or any other workaround). Report the error and tell the user to set the missing variables:

• JIRA_URL: Base URL for the Jira instance (e.g., https://redhat.atlassian.net)
• JIRA_USER: Username or email for authentication
• JIRA_TOKEN: API token for authentication

If environment variables are set, proceed to Step 0.3.

0.3 Determine Output Directory

IMPORTANT: Test plan artifacts should NOT be created in the skill repository to avoid polluting the skill codebase.

1. Check for --output-dir flag in arguments:

   • If present: use that directory and skip validation (contributor override)
   • Set FORCE_OUTPUT_DIR=true

2. If no --output-dir flag, check for saved preference:

   # Try to read saved preference from .claude/settings.json
   saved_dir=$(jq -r '.["test-plan"]?.output_dir // empty' .claude/settings.json 2>/dev/null)
   

3. Ask user where to create artifacts via AskUserQuestion:

   • If saved preference exists:

     Where should test plan artifacts be created?

     Press Enter to use saved location: <saved_dir> Or provide a different directory path:

   • If no saved preference:

     Where should test plan artifacts be created?

     Provide a directory path (e.g., ~/Code/opendatahub-test-plans/plans/<team-name>), or press Enter for: ~/Code/opendatahub-test-plans/plans/

     Note: Replace <team-name> with your team name (e.g., ai-hub, dashboard, etc.)

4. Parse user input:

   • Empty/Enter → use default (~/Code/opendatahub-test-plans/plans/) or saved preference
   • Path provided → use that path
   • Expand ~ to home directory

5. Validate against skill repository (unless FORCE_OUTPUT_DIR=true):

   export CLAUDE_SKILL_DIR
   force_flag=$([ "$FORCE_OUTPUT_DIR" = "true" ] && echo "--force" || echo "")
   (cd $(git -C ${CLAUDE_SKILL_DIR} rev-parse --show-toplevel) && uv run python scripts/repo.py validate-local-path  ) ||  1
   

Step 1: Gather Information

1. Strategy: If a Jira key was provided, fetch it using the fetch_issue.py script. If auto-detected, read the local file from artifacts/strat-tasks/ instead — do NOT fetch.

   Fetching from Jira:

   repo_root=$(git -C ${CLAUDE_SKILL_DIR} rev-parse --show-toplevel)
   tmp_result=$(cd "$repo_root" && uv run python scripts/parse_strat.py new-strat-tmp) || exit 1
   strategy_file=$(echo "$tmp_result" | jq -r '.strategy_file')
   (cd "$repo_root" && \
    uv run python scripts/fetch_issue.py <JIRA_KEY> --output "$strategy_file")
   

   Auto-detected from artifacts/strat-tasks/<JIRA_KEY>.md (shared cache; also a Jira-outage fallback for other skills):

   resolve_result=$(cd $(git -C ${CLAUDE_SKILL_DIR} rev-parse --show-toplevel) && uv run python scripts/parse_strat.py resolve-local "<JIRA_KEY>") || exit 1
   strategy_file=$(echo "$resolve_result" | jq -r '.strategy_file')
   

   • components is extracted deterministically in Step 1.5 (parse_strat.py save-snapshot).

2. ADR (if provided): Read the ADR file for additional technical detail (API endpoints, data models, implementation specifics).

Step 1.5: Parse Strategy Sections and Snapshot the Strategy

Run the STRAT parser on the fetched strategy file, before the snapshot step below moves/copies it:

repo_root=$(git -C ${CLAUDE_SKILL_DIR} rev-parse --show-toplevel)
gate_result=$(cd "$repo_root" && uv run python scripts/parse_strat.py workflow-inputs "$strategy_file")
gate_exit=$?

if [ "$gate_exit" -ne 0 ]; then
  echo "workflow-inputs failed to parse the strategy; cannot proceed" >&2
  echo "$gate_result" >&2
  exit 1
fi

gate_status=$(echo "$gate_result" | jq -r '.status')

if [ "$gate_status" = "ok" ]; then
  ac_json=$(echo "$gate_result" | jq -c '.ac_json')
  nfr_json=$(echo "$gate_result" | jq -c 'if .nfr_json.found then .nfr_json else empty end')
  oos_json=$(echo "$gate_result" | jq -c 'if .oos_json.found then .oos_json else empty end')
  ac_count=$(echo "$gate_result" | jq -r '.ac_count')
  nfr_category_flags=()
  while IFS= read -r cat; do [ -n "$cat" ] && nfr_category_flags+=(--nfr-category "");  < <(  | jq -r )
  strat_gaps=
  [ -z  ] && strat_gaps=
  [ -z  ] && strat_gaps=


feature_name=
(  && uv run python scripts/validate.py feature-name ) ||  1
feature_dir=

Snapshot the strategy at $feature_dir/.source-strategy.md — creates the feature dir, moves temp fetches, copies (never deletes) the shared cache:

snapshot_result=$(cd "$repo_root" && uv run python scripts/parse_strat.py save-snapshot "$strategy_file" "$feature_dir") || exit 1
strategy_file=$(echo "$snapshot_result" | jq -r '.strategy_file')
components=$(echo "$snapshot_result" | jq -r '.components | join(",")')

If $gate_status is no_acceptance_criteria (no ACs found or count is 0), STOP immediately:

1. Write a lowest-score review:

   (cd $(git -C ${CLAUDE_SKILL_DIR} rev-parse --show-toplevel) && uv run python scripts/frontmatter.py set \
       <absolute_path_to_output_dir>/<feature_name>/TestPlanReview.md \
       feature="<feature_name>" source_key=<JIRA_KEY> score=0 pass=false verdict=Rework \
       scores='{"specificity":0,"grounding":0,"scope_fidelity":0,"actionability":0,"consistency":0}' \
       auto_revised=false)
   

2. Write review body: "Strategy has no acceptance criteria. Cannot generate AC-traced test plan."
3. Stamp test-plan-rubric-fail on the Jira issue (non-blocking). Do NOT proceed to Step 2.

Step 2: Analyze (Parallel Sub-Agents)

Scope constraint: This pipeline generates e2e/system and UI test plans only — no unit, integration, or component test levels in Section 2.1. Each objective (Section 1.3) must cite its grounding (see Section 1.3 note below); Step 3.2 validates this deterministically.

Invoke these three forked analyzer skills in parallel using the Skill tool. Each runs in its own isolated context, reading the strategy/ADR from the paths passed to it.

Pass <feature_name>/.source-strategy.md (+ ADR path if provided) as file paths — do not inline the strategy text. Also pass Step 1.5 JSON extractions as ground truth (do not re-derive).

• test-plan.analyze.endpoints: Pass strategy + ADR + ac_json + oos_json + nfr_json. Extracts feature scope (in-scope, out-of-scope, grounded test objectives) and e2e test surface. Produces findings for Sections 1 and 4.
• test-plan.analyze.risks: Pass strategy + ADR + nfr_json. Determines e2e/UI test levels, types, priorities, risks with mitigations, and NFR assessments. Produces findings for Sections 2, 7, and 8.
• test-plan.analyze.infra: Pass strategy + ADR. Identifies test environment, data, users, infrastructure, and tooling requirements. Produces findings for Section 3.

Once all three sub-agents return:

1. Merge their structured findings into the test plan template (Step 3)
2. Collect their ## Gaps sections for Step 3.5

• Do NOT add information that was not present in any sub-agent's output

Step 3: Generate Files

1. Ensure test_cases/ exists: mkdir -p -- "$feature_dir/test_cases"
2. Resolve the strategy browse URL from the configured Jira URL (never invent or copy a host from this skill; falls back to the JIRA_BASE_URL alias like require_env does):

   jira_url="${JIRA_URL:-$JIRA_BASE_URL}"
   strat_url="${jira_url%/}/browse/${JIRA_KEY}"
   

   Use this exact strat_url value for {strat_url} in the template and for the Jira strategy link in README.md. Do not hardcode a Jira host or guess from examples in this document.
3. Read the template from ${CLAUDE_SKILL_DIR}/test-plan-template.md using the Read tool
4. Generate <feature_name>/TestPlan.md by filling in the template with the gathered information. Follow the template structure exactly — do not add, remove, or reorder sections. Do NOT write frontmatter manually — Step 3.1 handles it.
   • Line length: Wrap all prose lines to a maximum of 100 characters. This does not apply to tables, code blocks, or headings — only paragraph text and list items.
   • Markdown headings: Use proper markdown heading syntax (##, ###, ####) for all section and subsection titles. Never substitute bold text (**Title**) for a heading. This applies to all generated files (TestPlan.md, TestPlanGaps.md, README.md).
5. For Section 9.2 (Interface Coverage): fill the Interface column from Section 4. Leave Test Cases and Coverage empty (filled later).
6. Generate <feature_name>/README.md with:
   • Feature name and one-line description
   • Links to Jira strategy (use strat_url from step 2), ADR (if provided)
   • Link to TestPlan.md
   • Brief mention of where automated tests will be implemented

Step 3.1: Set Frontmatter

After generating TestPlan.md, set its frontmatter using the frontmatter.py script via Bash. This validates the metadata against the schema before writing.

First, auto-detect source type from Jira key prefix:

if [[ <JIRA_KEY> == RHAISTRAT-* ]]; then
    SOURCE_TYPE="strat"
elif [[ <JIRA_KEY> == RHOAIENG-* ]]; then
    SOURCE_TYPE="issue"
fi

Then set frontmatter:

IMPORTANT: Run Python scripts from the test-plan repo directory (where pyproject.toml is). Do NOT cd to the output directory before running scripts — use absolute paths for file arguments.

(cd $(git -C ${CLAUDE_SKILL_DIR} rev-parse --show-toplevel) && uv run python scripts/frontmatter.py set <absolute_path_to_output_dir>/<feature_name>/TestPlan.md \
    feature="<feature_name>" \
    source_key=<JIRA_KEY> \
    source_type=$SOURCE_TYPE \
    status=Draft \
    author="<team_name>" \
    components="$components" \
    additional_docs="<comma-separated list of doc links, or []>")
(cd $(git -C ${CLAUDE_SKILL_DIR} rev-parse --show-toplevel) && uv run python scripts/version.py set <absolute_path_to_output_dir>/<feature_name>/TestPlan.md 1.0.0)

• components: from Step 1.5 — empty string coerces to [].
• additional_docs: include ADR link and any other document links provided by the user. Use [] if none.
• last_updated is auto-set to today's date by the script.
• reviewers defaults to [].

If the script exits with an error, fix the field values and retry — do not write frontmatter by hand.

Step 3.2: Validate Generated Test Plan

After setting frontmatter, run the deterministic validation checks ($ac_count/$nfr_category_flags were parsed in Step 1.5 — Step 1.5's STOP gate guarantees $ac_count is set here):

testplan="<absolute_path_to_output_dir>/<feature_name>/TestPlan.md"
feature_dir="<absolute_path_to_output_dir>/<feature_name>"
repo_root=$(git -C ${CLAUDE_SKILL_DIR} rev-parse --show-toplevel)

team_list=$(cd "$repo_root" && uv run python scripts/get_component_test_dir.py --teams-only "$feature_dir") || {
    echo "ERROR: scripts/get_component_test_dir.py --teams-only failed — stopping." >&2
    echo "$team_list" >&2
    exit 1
}

(cd "$repo_root" && \
 scope_result=$(uv run python scripts/validate_test_scope.py "$testplan" \
     --include-teams="$team_list" --checks-dir=scripts/checks) && \
 (echo "$scope_result" | jq -e '.valid' >/dev/null || { echo "$scope_result" >&2; exit 1; }) && \
 uv run python scripts/validate.py ac-citations "$testplan" --ac-count "$ac_count" "${nfr_category_flags[@]}" && \
 uv run python scripts/validate.py ac-coverage "$testplan" --ac-count "$ac_count" && \
 uv run python scripts/validate.py structure "$testplan" && \
 uv run python scripts/validate.py category-prefixes "$testplan" && \
 uv run python scripts/validate.py interface-types "" && \
 uv run python scripts/validate.py infra-scope )

If any check fails, fix the violations in TestPlan.md and re-run once. If it still fails, STOP and report to the user.

Step 3.5: Collect Gaps and Prompt for Additional Documents

Write each sub-agent's full raw analysis (from Step 2) verbatim to <feature_dir>/.analysis-endpoints.md, <feature_dir>/.analysis-risks.md, and <feature_dir>/.analysis-infra.md. Do not hand-slice the ## Gaps section — the script extracts it.

Then run:

(cd $(git -C ${CLAUDE_SKILL_DIR} rev-parse --show-toplevel) && \
 uv run python scripts/consolidate_gaps_and_stamp.py \
   --feature-name "<feature_name>" \
   --source-key <JIRA_KEY> \
   --source endpoints=<feature_dir>/.analysis-endpoints.md \
   --source risks=<feature_dir>/.analysis-risks.md \
   --source infra=<feature_dir>/.analysis-infra.md \
   --last-updated "$(date -u +%F)" \
   --skip-cleanup \
   --out <feature_dir>/TestPlanGaps.md)

On success this writes TestPlanGaps.md (body + frontmatter) and prints {"gap_count": int, "status": str, "next": "proceed"|"prompt_user"}. Never hand-count gaps, hand-edit frontmatter, or run check-interactive yourself. If it exits non-zero, temp files are left for debugging; fix and re-run.

• next is proceed: skip the menu. Go to Step 3.6.
• next is prompt_user: present the menu below.

Interactive gaps menu (only when next is prompt_user):

Present AskUserQuestion. List gaps from TestPlanGaps.md, then offer:

1. Provide documents — paste file paths to resolve gaps
2. Proceed to review — continue as-is
3. Proceed + generate test cases — continue and auto-run /test-plan-create-cases

If option 1: Read the documents, re-run only the relevant Step 2 sub-agents, update the test plan, then re-run consolidate_gaps_and_stamp.py with the same command as above (including --skip-cleanup). Follow next from the new JSON. If option 2: Proceed to Step 3.6. If option 3: Proceed to Step 3.6, and after Step 4 automatically invoke /test-plan-create-cases with the feature directory.

Step 3.6: Stamp Jira label — test plan created

Add the test-plan-auto-created label to the source Jira issue to mark that an AI-generated test plan exists. This enables the org-pulse dashboard to track AI involvement in the test planning pipeline.

Read source_key from <feature_name>/TestPlan.md frontmatter before stamping:

source_key=$(cd $(git -C ${CLAUDE_SKILL_DIR} rev-parse --show-toplevel) && uv run python scripts/frontmatter.py read <absolute_path_to_output_dir>/<feature_name>/TestPlan.md source_key)

Then add the label using the add_jira_labels.py script:

(cd $(git -C ${CLAUDE_SKILL_DIR} rev-parse --show-toplevel) && \
 uv run python scripts/add_jira_labels.py "$source_key" test-plan-auto-created)

Label stamping is non-blocking — if it fails, log a warning and continue. Do not retry or halt the workflow.

Step 4: Review, Score, and Improve

After the gaps flow is complete, invoke the internal test-plan.review skill with the feature directory.

The reviewer runs the quality rubric (Specificity, Grounding, Scope Fidelity, Actionability, Consistency — 0-2 each, 10-point scale; full criteria definitions live in test-plan.review), handles auto-revision internally (up to 2 cycles), and writes <feature_name>/TestPlanReview.md with scores and feedback.

Handle the review output:

1. Read the verdict from <feature_name>/TestPlanReview.md frontmatter

2. Auto-fix (if any): Apply clearly correct improvements suggested by the reviewer with these constraints:

   • Consistency fixes (e.g., missing entries in Section 9.2 that are in Section 4)
   • Generic priority definitions that should be feature-specific (when the specific language is in the strategy)
   • NEVER invent resolution paths for TBDs — if the strategy doesn't specify where to find version requirements or missing details, leave them as plain "TBD". The gaps are already documented in TestPlanGaps.md.
   • Only add content that is directly traceable to the source documents (strategy, ADR, API specs, design docs, or any additional_docs) — do not make assumptions about where documentation exists or what it contains.

   Use the Edit tool for any auto-fixes applied.

3. Present summary: Show the user the final score/verdict, any auto-fixes applied, and any remaining gaps from TestPlanGaps.md

4. If verdict is Rework: Advise the user to provide additional source documents (ADR, API spec, design doc) to resolve quality issues before generating test cases

Step 4.5: Stamp rubric verdict label

After reading the review verdict from TestPlanReview.md, stamp the appropriate label on the source Jira issue. This enables the org-pulse dashboard to track review outcomes.

Determine which labels to add:

• Verdict "Ready" → add label test-plan-rubric-pass
• Verdict "Revise" → add label test-plan-rubric-revise
• Verdict "Rework" → add label test-plan-rubric-fail
• Any other verdict value → log a warning and skip rubric label stamping

Read frontmatter values explicitly before stamping:

verdict=$(cd $(git -C ${CLAUDE_SKILL_DIR} rev-parse --show-toplevel) && uv run python scripts/frontmatter.py read <absolute_path_to_output_dir>/<feature_name>/TestPlanReview.md verdict)
auto_revised=$(cd $(git -C ${CLAUDE_SKILL_DIR} rev-parse --show-toplevel) && uv run python scripts/frontmatter.py read <absolute_path_to_output_dir>/<feature_name>/TestPlanReview.md auto_revised)
source_key=$(cd $(git -C ${CLAUDE_SKILL_DIR} rev-parse --show-toplevel) && uv run python scripts/frontmatter.py read <absolute_path_to_output_dir>/<feature_name>/TestPlan.md source_key)

Build label list and apply:

if [ "$auto_revised" = "true" ]; then
    (cd $(git -C ${CLAUDE_SKILL_DIR} rev-parse --show-toplevel) && \
     uv run python scripts/add_jira_labels.py "$source_key" --verdict "$verdict" test-plan-auto-revised)
else
    (cd $(git -C ${CLAUDE_SKILL_DIR} rev-parse --show-toplevel) && \
     uv run python scripts/add_jira_labels.py "$source_key" --verdict "$verdict")
fi

Label stamping is non-blocking — if it fails, log a warning and continue. Do not retry or halt the workflow.

What this skill does NOT do

• Does NOT generate individual test case files
• Does NOT fetch child stories under the epic
• Does NOT fetch the Google Doc ADR — it reads a local file only
• Does NOT resolve GitHub PR review comments (use /test-plan-resolve-feedback <PR_URL> after publishing)
• Section 5 (Test Cases): placeholder — 5.2 categories must be test types (E2E, UI, NEG, NFR, UPG), not feature areas
• Section 6 (E2E Test Scenarios): left as placeholder — to be filled by /test-plan-create-cases
• Section 2.1 (Test Levels): e2e/system and UI levels only — no unit, integration, or component levels
• Section 1.3 (Test Objectives): Objectives must cite grounding via (AC: #N — text) or (NFR: category — text); every AC number 1..ac_count must be cited by at least one objective (Step 3.2 ac-coverage check)
• Section 7 (Non-Functional Requirements): filled by test-plan.analyze.risks — each category must be addressed or marked Not Applicable
• Section 9: placeholders — 9.1 and 9.2 filled by /test-plan-create-cases and /coverage-assessment

$ARGUMENTS