test-plan-create-cases

name	test-plan-create-cases
description	Generate individual test case files from an existing test plan. Use after test plan approval to generate individual TC specifications with preconditions, steps, and expected results organized by category and priority.
argument-hint	[FEATURE_SOURCE] [--output-dir PATH]
user-invocable	true
model	opus
allowedTools	Read, Write, Edit, Bash, AskUserQuestion

Test Case Generator

Generate individual test case specification files from an existing test plan.

Usage

/test-plan-create-cases [FEATURE_SOURCE] [--output-dir PATH]

Examples:

• /test-plan-create-cases (prompts for the feature directory)
• /test-plan-create-cases mcp_catalog
• /test-plan-create-cases /path/to/feature_dir
• /test-plan-create-cases mcp_catalog --output-dir . (contributor override)

Inputs

If $ARGUMENTS is empty, set FORCE_OUTPUT_DIR=false and go to Interactive fallback.

If $ARGUMENTS is non-empty, parse after Step 0.1. Consume --output-dir before the positional feature source:

OUTPUT_DIR=$(cd $(git -C ${CLAUDE_SKILL_DIR} rev-parse --show-toplevel) && \
  uv run python scripts/parse_skill_args.py --output-dir "$ARGUMENTS")
FORCE_OUTPUT_DIR=false
if [ -n "$OUTPUT_DIR" ]; then
    FORCE_OUTPUT_DIR=true
fi

--output-dir is a contributor override. When FORCE_OUTPUT_DIR=true, run marker validation in Step 0.2.2 and omit skill-repository path validation in Step 0.2.3. FEATURE_SOURCE is the positional argument or the interactive answer; the flag's PATH only sets FORCE_OUTPUT_DIR. If the flag is present with no positional feature source, go to Interactive fallback.

From arguments (optional)

After flags are consumed, if a remaining argument does not start with --, it is the feature source:

• Local directory path: mcp_catalog or /path/to/mcp_catalog
• GitHub branch: https://github.com/org/repo/tree/test-plan/RHAISTRAT-400
• GitHub PR: https://github.com/org/repo/pull/5

Action: Set FEATURE_SOURCE to that positional value and proceed to Step 0.2.

Interactive fallback (no positional feature source)

If $ARGUMENTS is empty, or no positional feature source remains after flags, invoke AskUserQuestion:

Where is the feature directory containing your test plan?

You can provide:

• Local directory path (e.g., /Users/username/Code/ai-hub-test-plans/mcp_catalog)
• GitHub branch URL (e.g., https://github.com/org/repo/tree/test-plan/RHAISTRAT-400)
• GitHub PR URL (e.g., https://github.com/org/repo/pull/5)

Action: Capture the user's selection as FEATURE_SOURCE and proceed to Step 0.2.

Process

Step 0: Pre-flight Check

0.1 Python dependencies

Install the test-plan package (makes all scripts importable):

(cd $(git -C ${CLAUDE_SKILL_DIR} rev-parse --show-toplevel) && uv sync --extra dev)

If installation fails, inform the user and do NOT proceed. Once installed, all Python scripts will work from any directory.

0.2 Locate Feature Directory

1. Use the shared locate-feature-dir utility to resolve FEATURE_SOURCE (local path or GitHub branch/PR) into a local directory:

   result=$(cd $(git -C ${CLAUDE_SKILL_DIR} rev-parse --show-toplevel) && uv run python scripts/repo.py locate-feature-dir "$FEATURE_SOURCE")
   if [ $? -ne 0 ]; then
       echo "$result"
       exit 1
   fi
   
   # Parse JSON output
   feature_dir=$(echo "$result" | jq -r '.feature_dir')
   source_type=$(echo "$result" | jq -r '.source_type')
   

2. For local sources, validate the feature directory is self-contained (was created by /test-plan-create, which always writes <feature_dir>/.test-plan-output-dir.json):

   if [ "$source_type" = "local" ]; then
       marker_result=$(cd $(git -C ${CLAUDE_SKILL_DIR} rev-parse --show-toplevel) && uv run python scripts/discover_feature_dir.py "$feature_dir")
       if [ $? -ne 0 ]; then
           echo "$marker_result"
           exit 1
       fi
   fi
   

3. Validate local paths against skill repository unless FORCE_OUTPUT_DIR=true:

Note: GitHub sources are always external repos, so no marker check or skill repo validation needed.

Step 1: Read the Test Plan

1. Read <feature_dir>/TestPlan.md using the Read tool
2. Extract the source_key from the YAML frontmatter — this will be used in Step 3.1 to set frontmatter on each test case file
3. Extract:
   • Section 4 (Interfaces Under Test) — the interface catalog (Interface, Type, Purpose)
   • Section 2 (Test Strategy) — test levels, types, priorities to guide test case depth
   • Section 3 (Test Environment) — preconditions and test data requirements
   • Section 5.2 (Test Case Naming Convention) — the TC-<CATEGORY>-<NUMBER> prefixes and their meanings
   • Section 1.2 (Scope) — in-scope vs out-of-scope boundaries
   • Section 1.3 (Test Objectives) — numbered objectives, each citing an AC. These are the traceability anchors for every generated TC — every TC frontmatter must reference at least one objective from this section (see Step 3.1)
   • Section 6 (E2E Test Scenarios), if already populated from a prior run — existing flow priorities to preserve during regeneration. On a fresh run this section is empty; priority for new flows is assigned per Section 2.3 criteria as scenarios are generated in Step 3

Step 1.5: Read Gaps (if available)

1. Check if <feature_dir>/TestPlanGaps.md exists (generated by /test-plan-create)
2. If it exists, read it to understand known limitations — do NOT create test cases for areas marked as pending or missing details
3. If it does not exist, proceed normally

Step 2: Read the Test Case Template

1. Read the template from ${CLAUDE_SKILL_DIR}/test-case-template.md using the Read tool
2. Follow this template structure for every generated test case
3. Line length: Wrap all prose lines to a maximum of 100 characters. This does not apply to tables, code blocks, or headings — only paragraph text and list items.
4. Omit optional sections (Preconditions, Test Data, Expected Response, Validation) when they are empty or not applicable — do not include empty sections
5. Always leave Automation Status and Notes as placeholders — they are filled later in the process

Step 2.5: Detect Regeneration Mode

1. Check for existing test cases:

   regen_check=$(cd $(git -C ${CLAUDE_SKILL_DIR} rev-parse --show-toplevel) && uv run python scripts/tc_regeneration.py check <feature_dir>)
   mode=$(echo "$regen_check" | jq -r '.mode')
   existing_count=$(echo "$regen_check" | jq -r '.existing_count')
   

2. If mode = "regenerate" (existing test cases found):

   a. Read all existing TC files using Read tool (satisfies Write tool requirement):

   echo "$regen_check" | jq -r '.files[]' | while read file; do
       # Read each existing TC file
   done
   

   b. Ask for confirmation via AskUserQuestion:

   Regeneration Mode

   Found <existing_count> existing test cases in test_cases/.

   Regenerating will overwrite all existing test cases. You can review changes via git diff before publishing.

   Proceed with regeneration? [yes/no]

   c. If no: Exit without changes

   d. If yes: Continue to Step 3 with REGENERATION_MODE=true

3. If mode = "create" (no existing test cases):

   • Continue to Step 3 with REGENERATION_MODE=false

Step 3: Design and Generate Test Cases

Process one category at a time from Section 5.2. For each category:

1. Design all test cases for that category:

   • Cover every interface from Section 4 relevant to this category
   • Include positive, negative, and boundary scenarios (per Section 2.2)
   • Assign priorities (P0/P1/P2) following the criteria in Section 2.3
   • Stay strictly within the scope defined in Section 1.2 — do NOT create test cases for out-of-scope items
   • Map each TC to the Section 1.3 objective(s) it validates — record as objectives in frontmatter (Step 3.1)
   • Before generating each TC, check all previously generated TCs across ALL categories. If another TC already verifies the same behavior (same preconditions, same verification target), do not create a duplicate — add the missing assertions to the existing TC instead

2. Write or Edit the TC-<CATEGORY>-<NUMBER>.md files for that category immediately before moving to the next:

   • If REGENERATION_MODE=true: Use Edit tool for files that already exist (preserves git history), Write tool for new files
   • If REGENERATION_MODE=false: Use Write tool for all files

   Include YAML frontmatter at the top of each file:

   ---
   test_case_id: TC-<CATEGORY>-<NUMBER>
   source_key: <STRAT_KEY_FROM_TEST_PLAN>
   objectives: [<N>, ...]
   priority: <P0|P1|P2>
   status: Draft
   automation_status: Not Started
   last_updated: "<today_date>"
   # upgrade_phase: pre|post|both   # see Step 3.4 — set for ANY TC whose expected results differ between upgrade states
   ---
   

   • source_key: use the value extracted from the test plan's frontmatter in Step 1
   • objectives: list of Section 1.3 objective numbers this TC validates (e.g., [1, 3]) — required, must be non-empty

This category-by-category approach ensures cross-category awareness (no duplicate coverage) while keeping each batch focused.

Expected Results quality: Each Expected Result must be an observable fact that directly confirms the test objective. Avoid vague conclusions ("works as expected", "renders successfully"). Name the specific page state, URL pattern, response code, element, or resource field.

Before writing each assertion, ask: "Is this testing what the TC is fundamentally about, or just a side effect?" Two patterns follow from this:

• Accessibility / reachability tests (does this URL work? does this link open?): assert the absence of error — "page does not contain '500 Internal Server Error'", "response is HTTP 200", "page does not show 'Application is not available'". Do not assert presence of specific UI components (IDE editor pane, console window, specific layout element) — these vary by configuration, workbench image, and product version and will cause failures unrelated to the feature under test.

• Content / format tests (does this show the right value? did something change?): assert the specific observable fact — "URL contains hostname pattern X", "field value equals Y", "element Z is visible". Use this only when the content itself IS what is being verified.

A test that FAILs for the wrong reason is worse than no test at all. When in doubt, prefer the narrower assertion.

Test case robustness rules:

• Background processes: When a TC uses background loops (&), capture each PID (PID_X=$!), define a cleanup() function that kills each PID with kill "$PID" 2>/dev/null || true (so a dead PID does not fail the trap), and register it with trap cleanup EXIT before starting the loops. Use EXIT only — adding INT or TERM causes cleanup to run on the signal and then again on exit.
• Query scoping: When querying Prometheus or other shared data stores, scope queries to the labels the data store actually exposes (e.g., namespace, job, container, pod). Include only the labels that exist in the target metric or data source — do not mandate labels the store does not carry. Do not rely on cluster-wide queries that unrelated workloads could satisfy.
• Validate all results: When asserting label sets, response structure, or field presence, first confirm the result array is non-empty (e.g., jq '.result | length > 0'), then validate ALL entries (e.g., jq '.result | length > 0 and all(...)'). An empty result silently passes all(...), so the length guard is required.
• Synthetic credentials only: Never specify production or real user credentials in preconditions or test data. Use test-only API keys, throwaway OIDC tokens from a test IdP, or synthetic identities.
• No fallback masking: When a TC's preconditions guarantee data exists (e.g., traffic is flowing), do not include or vector(0) or similar fallbacks in test queries — they mask broken pipelines as passing tests. Fallback behavior should be tested in dedicated edge-case TCs.

Anti-hallucination rules:

• Do NOT invent requirements not present in the test plan
• Do NOT create test cases for interfaces marked as "pending details" in Section 4
• If the test plan is ambiguous about what to test, ask the user via AskUserQuestion

Step 4: Generate Index

After all categories are complete (including upgrade TCs if generated):

1. Create <feature_dir>/test_cases/ directory if it doesn't already exist: mkdir -p <feature_dir>/test_cases
2. Each test case file must be self-contained — a tester should be able to execute it without reading the test plan
3. Use realistic test data, not placeholder values like "example.com" or "test123"
4. Generate <feature_dir>/test_cases/INDEX.md atomically (regenerate entire file):
   • Scan all TC-*.md files in test_cases/ directory
   • Extract test_case_id, priority, and title from each
   • Build complete INDEX.md with:
     • Quick stats (total test cases, P0/P1/P2 counts)
     • Test cases organized by category in tables (Test Case ID linked, Title, Priority)
     • Link to parent TestPlan.md
   • Write complete file in one operation (do NOT append incrementally)

Step 5: Update the Test Plan

Update <feature_dir>/TestPlan.md using the Edit tool:

1. Section 5 — Update the note to reflect test cases have been generated, with a link to test_cases/INDEX.md
2. Section 5.1 — Fill in the Test Case Organization table with category, test case count, and priority distribution
3. Section 6.1 — Fill in the E2E Scenario Summary table with the generated TC-E2E-* scenarios (ID, scenario name, interfaces covered, priority)
4. Section 6.2 — Fill in the E2E Coverage Matrix mapping each interface from Section 4 to its E2E scenario IDs
5. Section 9.1 — Fill in the Test Case Summary table with counts per category and priority breakdown
6. Section 9.2 — Fill in the Test Cases column with TC IDs mapped to each interface. Leave the Coverage column empty — it will be filled later by /coverage-assessment

Step 5.5: Update README

Update <feature_dir>/README.md to add a link to the test cases index:

• Add a "Test Cases" section (or update existing) with a link to test_cases/INDEX.md
• Include the total test case count and priority breakdown

Step 5.6: Coverage Validation

After generating all test case files and updating the test plan, validate coverage:

1. Interface coverage: Run uv run python scripts/validate.py interface-coverage <feature_dir>/TestPlan.md (deterministic table diff — do not eyeball Section 9.2/6.2 yourself). If missing_in_9_2 is non-empty, those interfaces lack test case coverage — flag them as gaps. Interfaces marked "pending details" in Section 4 are listed under pending and are already excluded from missing_in_9_2/missing_in_6_2 by the validator.
2. E2E coverage: From the same interface-coverage result, if section_6_2_populated is true and missing_in_6_2 is non-empty, those interfaces lack E2E scenario coverage — generate the missing TC-E2E-* test case(s), update Sections 6.2/9.2, and re-run the validator before proceeding. missing_in_6_2 already excludes pending interfaces, so this never regenerates cases for interfaces excluded by the anti-hallucination rule.
3. Test objective coverage: Check that every test objective from Section 1.3 is addressed by at least one test case. Flag any uncovered objectives.
4. Priority distribution: Verify that TC priorities align with the flow priorities in Section 6.1 — a P0 flow should not only have P2 test cases.
5. Configurable coverage: Check that every env var, config path, or configurable explicitly named in Section 3.1 has at least one TC that exercises a non-default value. If any is uncovered, flag it as a coverage gap.
6. Objective traceability: Check that every generated TC's objectives frontmatter field references at least one valid Section 1.3 objective number, and that every referenced objective has an AC citation. Flag any TC with a missing, empty, or invalid objectives field.
7. Gap cross-reference: If TestPlanGaps.md was read in Step 1.5, verify that no test cases were created for interfaces or areas flagged as pending/missing. If any were, remove them and flag the inconsistency.
8. Append to TestPlanGaps.md: If <feature_dir>/TestPlanGaps.md exists, append a ## Test Case Coverage Gaps section with any coverage gaps found (uncovered interfaces, missing objectives, priority mismatches, missing E2E scenarios, uncovered configurables). If the file does not exist, create it with just this section.

Step 5.7: Validate Frontmatter, Counts, Scope, and Traceability

After all test case files are written, validate frontmatter, TC counts, category scope, and objective traceability:

(cd $(git -C ${CLAUDE_SKILL_DIR} rev-parse --show-toplevel) && \
 uv run python scripts/validate.py test-cases <feature_dir> && \
 uv run python scripts/validate.py tc-counts <feature_dir> && \
 uv run python scripts/validate.py tc-scope <feature_dir> && \
 uv run python scripts/validate.py tc-traceability <feature_dir> && \
 uv run python scripts/validate.py interface-coverage <feature_dir>/TestPlan.md)

If any check fails, fix the issue and re-run.

What this skill does NOT do

• Does NOT modify the test plan's Sections 1-4, 7-8, or 9.3 — those are owned by /test-plan-create (Sections 9.1 and 9.2 ARE filled by this skill — see Steps 5.5/5.6)
• Does NOT fill Automation Status or Notes in TC files — those are filled later by /coverage-assessment
• Does NOT create test cases for out-of-scope items or pending interfaces

$ARGUMENTS