test-plan-create

name	test-plan-create
description	Generate a test plan from a strategy (RHAISTRAT or RHOAIENG issue), with optional ADR for extra technical depth. Use when starting test planning for a new RHOAI feature with a defined Jira strategy.
argument-hint	<JIRA_KEY> [ADR_FILE_PATH]
user-invocable	true
model	opus
allowedTools	["Read","Write","Bash","Glob","Skill","AskUserQuestion"]

Test Plan Generator

Generate a complete test plan for a RHOAI feature based on a refined strategy, and optionally an ADR document for additional technical depth.

Usage

/test-plan-create <JIRA_KEY> [ADR_FILE_PATH]

Examples:

• /test-plan-create RHAISTRAT-400 (preferred - formal strategy)
• /test-plan-create RHOAIENG-48676 (alternative - bug/epic/task)
• /test-plan-create RHAISTRAT-400 /path/to/adr.pdf

Inputs

From arguments

Parse $ARGUMENTS to extract:

1. First argument (required): Jira key - either a RHAISTRAT strategy (e.g., RHAISTRAT-400) or RHOAIENG issue (e.g., RHOAIENG-48676)
2. Second argument (optional): Local path to an ADR document (markdown, text, or PDF)

Auto-detection

If no arguments are provided and a strategy file was just generated by /strat.create or /strat.refine in this session, use it automatically — proceed directly to Step 1.

Interactive fallback

If no arguments are provided and no strategy file is available from the current session, ask the user for:

Required

1. Jira key: A RHAISTRAT strategy key (e.g., RHAISTRAT-400) or RHOAIENG issue key (e.g., RHOAIENG-48676)

Optional

2. ADR file path: Local path to an exported ADR document (markdown, text, or PDF) for additional technical detail (API specs, data models, etc.)
3. ADR link: Google Doc URL (stored as reference in metadata, not fetched)
4. Feature directory name: snake_case name for the feature directory (e.g., mcp_catalog). If not provided, derive from the feature name.

Process

Step 0: Pre-flight Checks

0.1 Python dependencies

Install the test-plan package (makes all scripts importable):

(cd $(git -C ${CLAUDE_SKILL_DIR} rev-parse --show-toplevel) && uv sync --extra dev)

If installation fails, inform the user and do NOT proceed. Once installed, all Python scripts will work from any directory.

0.2 Jira Environment Variables

Verify that Jira API credentials are configured via environment variables:

# Check for required environment variables
for var in JIRA_URL JIRA_USER JIRA_TOKEN; do
    if [ -z "${!var}" ]; then
        echo "Error: $var environment variable is required" >&2
        echo "See CONTRIBUTING.md for Jira API setup instructions" >&2
        exit 1
    fi
done

If any environment variable is missing:

1. Inform the user that Jira API credentials are required to fetch strategy details
2. Ask the user to set up environment variables:
   • JIRA_URL: Base URL for the Jira instance (e.g., https://issues.redhat.com)
   • JIRA_USER: Username or email for authentication
   • JIRA_TOKEN: API token for authentication
3. Do NOT proceed until credentials are available or the user provides a local strategy file from artifacts/strat-tasks/ as an alternative

If environment variables are set, proceed to Step 0.3.

0.3 Determine Output Directory

IMPORTANT: Test plan artifacts should NOT be created in the skill repository to avoid polluting the skill codebase.

1. Check for --output-dir flag in arguments:

   • If present: use that directory and skip validation (contributor override)
   • Set FORCE_OUTPUT_DIR=true

2. If no --output-dir flag, check for saved preference:

   # Try to read saved preference from .claude/settings.json
   saved_dir=$(jq -r '.["test-plan"]?.output_dir // empty' .claude/settings.json 2>/dev/null)
   

3. Ask user where to create artifacts via AskUserQuestion:

   • If saved preference exists:

     Where should test plan artifacts be created?

     Press Enter to use saved location: <saved_dir> Or provide a different directory path:

   • If no saved preference:

     Where should test plan artifacts be created?

     Provide a directory path (e.g., ~/Code/opendatahub-test-plans/plans/<team-name>), or press Enter for: ~/Code/opendatahub-test-plans/plans/

     Note: Replace <team-name> with your team name (e.g., ai-hub, dashboard, etc.)

4. Parse user input:

   • Empty/Enter → use default (~/Code/opendatahub-test-plans/plans/) or saved preference
   • Path provided → use that path
   • Expand ~ to home directory

5. Validate against skill repository (unless FORCE_OUTPUT_DIR=true):

   # Validate path is not in skill repo
   export CLAUDE_SKILL_DIR
   force_flag=$([ "$FORCE_OUTPUT_DIR" = "true" ] && echo "--force" || echo "")
   (cd $(git -C ${CLAUDE_SKILL_DIR} rev-parse --show-toplevel) && uv run python scripts/repo.py validate-local-path "$target_dir" $force_flag) || exit 1
   

6. Ask to save preference via AskUserQuestion (unless using --output-dir):

   Save this location for future /test-plan-create runs? [yes/no]

   • If yes: Save to .claude/settings.json:

     # Create .claude directory if it doesn't exist
     mkdir -p .claude
     
     # Update or create settings.json with output_dir preference
     if [ -f .claude/settings.json ]; then
         jq '.["test-plan"].output_dir = "'"$target_dir"'"' .claude/settings.json > .claude/settings.json.tmp
         mv .claude/settings.json.tmp .claude/settings.json
     else
         echo '{"test-plan": {"output_dir": "'"$target_dir"'"}}' > .claude/settings.json
     fi
     
     echo "✓ Saved preference to .claude/settings.json"
     

   • If no: Continue without saving (will ask again next time)

7. Create output directory if it doesn't exist:

   mkdir -p "$target_dir"
   cd "$target_dir"
   echo "✓ Creating test plan artifacts in: $target_dir"
   

8. Store for session context:

   • Export environment variable for /test-plan-create-cases auto-detection:

     export TEST_PLAN_OUTPUT_DIR="$target_dir"
     

   • This allows /test-plan-create-cases to auto-use the same location if called in same session

Step 1: Gather Information

1. Strategy: If a Jira key was provided, fetch it using the fetch_issue.py script. The strategy contains both the technical approach (HOW) and the business need (WHAT/WHY). If auto-detected, read the local file from artifacts/strat-tasks/.

   # Fetch strategy and save to temporary file
   strategy_file=$(mktemp)
   (cd $(git -C ${CLAUDE_SKILL_DIR} rev-parse --show-toplevel) && \
    uv run python scripts/fetch_issue.py <JIRA_KEY> --output "$strategy_file")
   
   # Read the saved strategy
   strategy_content=$(cat "$strategy_file")
   
   # Clean up
   rm "$strategy_file"
   

   • Extract components from the Jira response by parsing the markdown output (list of RHOAI product component names, e.g., ["AI Hub", "Model Serving"])
   • If Components field is empty or missing, set components = []
   • Store for use in frontmatter (Step 3.1)
2. ADR (if provided): Read the ADR file for additional technical detail (API endpoints, data models, implementation specifics).

Step 2: Analyze (Parallel Sub-Agents)

Invoke these three forked analyzer skills in parallel using the Skill tool. Each runs in its own isolated context with the strategy and ADR content.

Pass the full strategy content (and ADR content if available) inline in the skill arguments so each sub-agent has the source material.

• test-plan.analyze.endpoints: Extracts feature scope (in-scope, out-of-scope, test objectives) and identifies API endpoints/methods under test. Produces findings for Sections 1 and 4.
• test-plan.analyze.risks: Determines test levels, test types, priority definitions, risks with mitigations, and non-functional requirement assessments. Produces findings for Sections 2, 7, and 8.
• test-plan.analyze.infra: Identifies test environment configuration, test data, test users, infrastructure, and tooling requirements. Produces findings for Sections 3 and 9.

Once all three sub-agents return:

1. Merge their structured findings into the test plan template (Step 3)
2. Collect their ## Gaps sections for Step 3.5

• Do NOT add information that was not present in any sub-agent's output

Step 3: Generate Files

1. Create the feature directory using Bash: mkdir -p <target_dir>/<feature_name>/test_cases
   • target_dir was determined in Step 0.3
   • We're already in target_dir from Step 0.3, so: mkdir -p <feature_name>/test_cases
2. Read the template from ${CLAUDE_SKILL_DIR}/test-plan-template.md using the Read tool
3. Generate <feature_name>/TestPlan.md by filling in the template with the gathered information. Follow the template structure exactly — do not add, remove, or reorder sections. Do NOT write frontmatter manually — Step 3.1 handles it.
   • Line length: Wrap all prose lines to a maximum of 100 characters. This does not apply to tables, code blocks, or headings — only paragraph text and list items.
4. For Section 10.2 ({Endpoint/Method} Coverage): fill in the Endpoint column using the endpoints identified in Section 4. Leave the Test Cases and Coverage columns empty — they will be filled later in the process.
5. Generate <feature_name>/README.md with:
   • Feature name and one-line description
   • Links to Jira strategy, ADR (if provided)
   • Link to TestPlan.md
   • Brief mention of where automated tests will be implemented

Step 3.1: Set Frontmatter

After generating TestPlan.md, set its frontmatter using the frontmatter.py script via Bash. This validates the metadata against the schema before writing.

First, auto-detect source type from Jira key prefix:

# Auto-detect source_type from Jira key
if [[ <JIRA_KEY> == RHAISTRAT-* ]]; then
    SOURCE_TYPE="strat"
elif [[ <JIRA_KEY> == RHOAIENG-* ]]; then
    SOURCE_TYPE="issue"
fi

Then set frontmatter:

IMPORTANT: Run Python scripts from the test-plan repo directory (where pyproject.toml is). Do NOT cd to the output directory before running scripts — use absolute paths for file arguments.

(cd $(git -C ${CLAUDE_SKILL_DIR} rev-parse --show-toplevel) && uv run python scripts/frontmatter.py set <absolute_path_to_output_dir>/<feature_name>/TestPlan.md \
    feature="<feature_name>" \
    source_key=<JIRA_KEY> \
    source_type=$SOURCE_TYPE \
    version=1.0.0 \
    status=Draft \
    author="<team_name>" \
    components="<comma-separated component names from Jira, or []>" \
    additional_docs="<comma-separated list of doc links, or []>")

• components: comma-separated list of component names from Jira Components field (e.g., "AI Hub,Model Serving"). Use [] if none.
• additional_docs: include ADR link and any other document links provided by the user. Use [] if none.
• last_updated is auto-set to today's date by the script.
• reviewers defaults to [].

If the script exits with an error, fix the field values and retry — do not write frontmatter by hand.

Step 3.5: Collect Gaps and Prompt for Additional Documents

After generating the test plan, collect all gaps reported by the three sub-agents from Step 2.

1. Extract gaps from each sub-agent's ## Gaps output section

2. Write <feature_name>/TestPlanGaps.md with all gaps organized by source sub-agent:

   # Gaps — <Feature Name>
   
   ## Scope & Endpoints
   {gaps from test-plan.analyze.endpoints, or "No gaps identified."}
   
   ## Test Strategy & Risks
   {gaps from test-plan.analyze.risks, or "No gaps identified."}
   
   ## Environment & Infrastructure
   {gaps from test-plan.analyze.infra, or "No gaps identified."}
   

3. Set frontmatter on TestPlanGaps.md using the frontmatter.py script (reuse SOURCE_TYPE from Step 3.1):

   (cd $(git -C ${CLAUDE_SKILL_DIR} rev-parse --show-toplevel) && uv run python scripts/frontmatter.py set <absolute_path_to_output_dir>/<feature_name>/TestPlanGaps.md \
       feature="<feature_name>" \
       source_key=<JIRA_KEY> \
       status=Open \
       gap_count=<number_of_gaps>)
   

   • gap_count: total number of individual gaps across all three sections
   • If no gaps were identified, set status=Resolved and gap_count=0
   • last_updated is auto-set by the script

4. Check for non-interactive mode:

   # Auto-proceed in CI or when explicitly requested
   if [ -n "${CI:-}" ] || [ -n "${CLAUDE_NON_INTERACTIVE:-}" ]; then
       NON_INTERACTIVE=true
   else
       NON_INTERACTIVE=false
   fi
   

5. If gaps exist, handle based on mode:

   Non-interactive mode (NON_INTERACTIVE=true):

   # Log gaps and auto-proceed to review
   echo "========================================" >&2
   echo "Gaps identified (auto-proceeding):" >&2
   echo "========================================" >&2
   cat "<feature_name>/TestPlanGaps.md" >&2
   echo "========================================" >&2
   # Proceed directly to Step 4 (review)
   

   Interactive mode (NON_INTERACTIVE=false):

   Present the user with a structured action menu via AskUserQuestion. List the gaps first, then offer numbered options. Example:

   The following gaps were identified in the test plan:

   • Endpoint paths for the catalog API are not specified — an API spec or ADR would resolve this
   • RBAC roles are unclear — a feature refinement doc would help
   • KServe CSI configuration details are missing — a design doc would resolve this

   What would you like to do?

   1. Provide documents — paste file paths (ADR, API spec, design doc) to resolve gaps
   2. Proceed to review — continue with the test plan as-is (gaps will be noted in TestPlanGaps.md)
   3. Proceed to review + generate test cases — continue and automatically run /test-plan-create-cases after review

6. If the user chooses option 1: Read the provided documents, re-run only the relevant sub-agents from Step 2 with the new material, update the test plan, update TestPlanGaps.md (removing resolved gaps, adding any new ones), update the gaps frontmatter (gap_count, status), then present the menu again with remaining gaps (if any).

7. If the user chooses option 2, no gaps exist, or non-interactive mode: Proceed to Step 3.6.

8. If the user chooses option 3: Proceed to Step 3.6, and after the review is complete (Step 4), automatically invoke /test-plan-create-cases with the feature directory.

Step 3.6: Stamp Jira label — test plan created

Add the test-plan-auto-created label to the source Jira issue to mark that an AI-generated test plan exists. This enables the org-pulse dashboard to track AI involvement in the test planning pipeline.

Read source_key from <feature_name>/TestPlan.md frontmatter before stamping:

source_key=$(cd $(git -C ${CLAUDE_SKILL_DIR} rev-parse --show-toplevel) && uv run python scripts/frontmatter.py read <absolute_path_to_output_dir>/<feature_name>/TestPlan.md source_key)

Then add the label using the add_jira_labels.py script:

(cd $(git -C ${CLAUDE_SKILL_DIR} rev-parse --show-toplevel) && \
 uv run python scripts/add_jira_labels.py "$source_key" test-plan-auto-created)

Label stamping is non-blocking — if it fails (e.g., API unavailable, insufficient permissions, network error), log a warning and continue to the next step. Do not retry or halt the workflow.

Step 4: Review, Score, and Improve

After the gaps flow is complete, invoke the internal test-plan.review skill with the feature directory.

The reviewer runs the quality rubric (5 criteria, 0-2 each, 10-point scale):

• Specificity — feature-specific vs boilerplate
• Grounding — traceable to source material vs fabricated
• Scope Fidelity — matches strategy scope
• Actionability — a QE engineer could start testing
• Consistency — sections agree with each other

The reviewer handles auto-revision internally (up to 2 cycles) and writes <feature_name>/TestPlanReview.md with scores and feedback.

Handle the review output:

1. Read the verdict from <feature_name>/TestPlanReview.md frontmatter

2. Auto-fix (if any): Apply clearly correct improvements suggested by the reviewer with these constraints:

   • Consistency fixes (e.g., missing entries in Section 10.2 that are in Section 4)
   • Generic priority definitions that should be feature-specific (when the specific language is in the strategy)
   • NEVER invent resolution paths for TBDs — if the strategy doesn't specify where to find version requirements or missing details, leave them as plain "TBD". The gaps are already documented in TestPlanGaps.md.
   • Only add content that is directly traceable to the source documents (strategy, ADR, API specs, design docs, or any additional_docs) — do not make assumptions about where documentation exists or what it contains.

   Use the Edit tool for any auto-fixes applied.

3. Present summary: Show the user:

   • Final score and verdict from TestPlanReview.md
   • Any auto-fixes applied
   • Any remaining gaps from TestPlanGaps.md
   • Full visibility into the test plan's quality before proceeding to test case generation

4. If verdict is Rework: Advise the user to provide additional source documents (ADR, API spec, design doc) to resolve quality issues before generating test cases

Step 4.5: Stamp rubric verdict label

After reading the review verdict from TestPlanReview.md, stamp the appropriate label on the source Jira issue. This enables the org-pulse dashboard to track review outcomes.

Determine which labels to add:

• Verdict "Ready" → add label test-plan-rubric-pass
• Verdict "Rework" → add label test-plan-rubric-fail
• Verdict "Revise" → do not add a rubric label (intermediate state handled by auto-revision)
• Any other verdict value → log a warning and skip rubric label stamping

Read frontmatter values explicitly before stamping:

verdict=$(cd $(git -C ${CLAUDE_SKILL_DIR} rev-parse --show-toplevel) && uv run python scripts/frontmatter.py read <absolute_path_to_output_dir>/<feature_name>/TestPlanReview.md verdict)
auto_revised=$(cd $(git -C ${CLAUDE_SKILL_DIR} rev-parse --show-toplevel) && uv run python scripts/frontmatter.py read <absolute_path_to_output_dir>/<feature_name>/TestPlanReview.md auto_revised)
source_key=$(cd $(git -C ${CLAUDE_SKILL_DIR} rev-parse --show-toplevel) && uv run python scripts/frontmatter.py read <absolute_path_to_output_dir>/<feature_name>/TestPlan.md source_key)

Build label list and apply:

# Build label list based on verdict
if [ "$verdict" = "Ready" ]; then
    rubric_label="test-plan-rubric-pass"
elif [ "$verdict" = "Rework" ]; then
    rubric_label="test-plan-rubric-fail"
else
    echo "Warning: Unexpected verdict '$verdict', skipping rubric label" >&2
    rubric_label=""
fi

# Add labels (conditionally include auto-revised)
if [ -n "$rubric_label" ]; then
    if [ "$auto_revised" = "true" ]; then
        (cd $(git -C ${CLAUDE_SKILL_DIR} rev-parse --show-toplevel) && \
         uv run python scripts/add_jira_labels.py "$source_key" "$rubric_label" test-plan-auto-revised)
    else
        (cd $(git -C ${CLAUDE_SKILL_DIR} rev-parse --show-toplevel) && \
         uv run python scripts/add_jira_labels.py "$source_key" "$rubric_label")
    fi
fi

Label stamping is non-blocking — if it fails, log a warning and continue. Do not retry or halt the workflow.

What this skill does NOT do

• Does NOT generate individual test case files
• Does NOT fetch child stories under the epic
• Does NOT fetch the Google Doc ADR — it reads a local file only
• Does NOT resolve GitHub PR review comments (use /test-plan-resolve-feedback <PR_URL> after publishing)
• Section 5 (Test Cases): left as placeholder — to be filled later in the process
• Section 6 (E2E Test Scenarios): left as placeholder — to be filled by /test-plan-create-cases
• Section 7 (Non-Functional Requirements): filled by test-plan.analyze.risks — each category must be addressed or marked Not Applicable
• Section 10.1 (Test Case Summary): left as placeholder — to be filled later in the process
• Section 10.2 Test Cases column: left empty — to be filled by /test-plan-create-cases
• Section 10.2 Coverage column: left empty — to be filled by /coverage-assessment

$ARGUMENTS