acm-test-case-reviewer

name	acm-test-case-reviewer
description	Use when the user wants a QUALITY REVIEW of an existing ACM Console UI test case markdown file (conventions, discovered-vs-assumed UI, MCP spot-checks, AC consistency)— without re-running JIRA/PR/UI investigation. If the user just finished acm-test-case-generator, review is already Phase 7 inside that skill—do not invoke this unless reviewing a standalone file or an out-of-band draft. TRIGGER: review this test-case.md, pre-Polarion QA, validate test steps. DO NOT TRIGGER: full test case from JIRA (acm-test-case-generator).
disable-model-invocation	true
compatibility	Requires acm-source MCP (for MCP verification spot-checks). Uses acm-knowledge-base skill (no MCP needed).
metadata	{"author":"acm-qe","version":"1.0.0"}

ACM Test Case Quality Reviewer

Validates generated test cases against conventions, verifies UI elements were discovered (not assumed), and enforces Polarion metadata completeness. This is the mandatory quality gate before a test case is delivered.

Prerequisites

• A test case .md file to review
• acm-source MCP server available for MCP spot-checks
• acm-knowledge-base skill available for conventions and area knowledge

Review Process

Step 1: Read the Test Case

Read the full test case markdown file.

Step 2: Read Conventions

Read from the knowledge database:

• ${SKILLS_DIR}/../knowledge/conventions/test-case-format.md
• ${SKILLS_DIR}/../knowledge/conventions/area-naming-patterns.md
• ${SKILLS_DIR}/../knowledge/conventions/cli-in-steps-rules.md

Step 3: Structural Validation

Check each section against conventions. Flag as BLOCKING if:

• Title doesn't match # RHACM4K-XXXXX - [Tag-Version] Area - Test Name
• Any required Polarion metadata field is missing (Type, Level, Component, Subcomponent, Test Type, Pos/Neg, Importance, Automation, Tags, Release)
• ## Type: value is not Test Case
• Description missing Entry Point or Dev JIRA Coverage
• ## Test Steps header missing
• Steps missing ### Step N: Title format, numbered actions, or bullet expected results
• Steps not separated by ---

Flag as BLOCKING if (step quality):

• A step combines passive observation (read/check text) with active interaction (click/navigate) — must be split into separate steps
• CLI backend validation is embedded within a UI-focused step instead of a dedicated step titled "Verify [what] via CLI (Backend Validation)"

Flag as WARNING if:

• Setup commands missing # Expected: comments
• Teardown missing --ignore-not-found on delete commands
• Tags don't match the area conventions
• Implementation details (sort algorithm, default values, parsing logic) not translated into observable verifications in expected results

Step 4: MCP Verification (MANDATORY -- minimum 3 checks)

Use acm-source MCP tools directly for spot-checks.

Prerequisites (not counted toward the 3 minimum):

• set_acm_version -- MUST call before any search/get
• set_cnv_version -- also required for Fleet Virt, CCLM, MTV area reviews

Counted verifications (minimum 3 required):

1. search_translations for 1-2 key UI labels -- verify they match the test case
2. get_routes -- verify the entry point route exists and matches
3. get_component_source for the primary component -- verify at least ONE factual claim (field order, filtering behavior, empty state, conditional rendering)

Entry point label verification: After verifying the route via get_routes, check the entry point label:

• Extract the last segment of the entry point path (e.g., "Managed clusters" from "Infrastructure > Clusters > Managed clusters")
• Call search_translations with exact=true for that label
• If NOT found, search with partial match to find what labels actually exist
• Known mismatches: managedClusters → "Cluster list" (NOT "Managed clusters"). Route keys are camelCase code identifiers, not UI labels.

If live validation output exists in the run directory, cross-check the entry point against what the browser actually showed.

If fewer than 3 verifications are performed, the verdict MUST be NEEDS_FIXES.

Step 5: AC vs Implementation Check

1. Extract the JIRA story's Acceptance Criteria from the investigation context
2. For each AC, check if the test case's expected results are consistent
3. If an AC says behavior X but the test expects behavior Y:
   • Check if a Note explains the discrepancy with source code citation
   • If a Note cites source code: verify the cited behavior via get_component_source
   • If no Note exists: flag as BLOCKING
4. Check scope: test case should validate the target JIRA story's ACs, not the broader PR scope

Step 6: Knowledge File Cross-Reference

Read ${SKILLS_DIR}/../knowledge/ui/<area>.md. Verify:

• Field order claims match the knowledge file
• Filtering behavior claims match the knowledge file
• Component names and CRDs are consistent
• Flag contradictions as BLOCKING

Step 6.5: Test Design Efficiency Check

Review the test case for design inefficiencies (flag as WARNING, not BLOCKING):

1. Redundant resources: Does the setup create multiple instances of the same resource type where one could serve multiple steps via state transitions? Flag: "Setup creates N [resource type] instances -- consider using state transitions on a single instance."
2. Missed state transitions: Does the test verify state A on entity X, then set up state B on a DIFFERENT entity Y and verify there? Flag: "Steps [N] and [M] test the same behavior on different entities -- consider testing before/after on a single entity."
3. Duplicate verifications: Do two steps verify the same element/behavior in the same context with no intervening state change? Flag: "Steps [N] and [M] verify the same behavior -- consider merging."
4. Setup/step ratio: If the setup creates more resources than the test steps consume, flag: "Setup creates [N] resources but only [M] are referenced in test steps -- remove unused resources."

Step 6.6: Coverage Gap Verification

If the synthesized context (from the orchestrator) includes a "Coverage Gap Triage" section:

1. Read the triage decisions.
2. For each gap triaged as "ADD TO TEST PLAN," verify the test case has a step or expected result covering it. If not, flag as WARNING: "Coverage gap [GAP-N] was triaged as ADD TO TEST PLAN but no test step covers it."
3. For each gap triaged as "NOTE ONLY," verify the Notes section mentions it. If not, this is acceptable.
4. Count: "Coverage gaps: [N] total, [X] covered in test steps, [Y] noted, [Z] skipped."

If no Coverage Gap Triage section exists, skip this step.

Step 6.7: Functional Outcome Coverage Check

Determine whether the test case verifies the feature's stated user outcome, or only the UI mechanics and data-path integration.

1. Read the JIRA story description from the investigation context. Extract the feature's value statement — what user-visible outcome does this feature deliver?
2. Check if the code change is UI-only (no backend controller/CRD/webhook files in the PR diff).
3. If the JIRA description contains outcome language (prevents, preserves, enables, blocks, protects, enforces, deploys, migrates, provisions, removes, syncs, propagates) AND the code change is UI-only:
   • Check if any test step verifies the functional outcome (look for steps titled "Verify Functional Outcome" or steps that test beyond "field X is set in YAML")
   • If no outcome step exists: flag severity depends on context
4. If the JIRA links to a bug or incident (linked issues, "customer-escalation" label, or description references a production issue): flag as BLOCKING — incident-driven features require outcome verification
5. Otherwise: flag as WARNING — "Test case validates UI mechanics and data path but does not verify the feature's stated outcome: '[outcome from JIRA]'. Consider adding an E2E verification step."

If the code change includes backend files, or the JIRA has no outcome language, skip this check.

Output Format

TEST CASE REVIEW
================
File: [path]
Area: [area]
Version: [version]

MCP VERIFICATIONS
1. search_translations -- query: "[query]", result: [what was found], matches: [yes/no]
2. get_routes -- query: [area routes], result: [route found], matches: [yes/no]
3. get_component_source -- path: "[file]", claim verified: "[claim]", result: [what source shows], matches: [yes/no]

BLOCKING (must fix):
1. [issue] -- Fix: [instruction]
[or "None"]

WARNING (should fix):
1. [issue] -- Fix: [instruction]
[or "None"]

Assumed vs Discovered:
- [element]: DISCOVERED via [tool + evidence]
- [element]: POTENTIALLY ASSUMED (could not verify)

Verdict: PASS | NEEDS_FIXES

Format requirements for review_enforcement.py parsing:

• The MCP VERIFICATIONS section header MUST exist (case-insensitive)
• Each entry MUST be a numbered line starting with the bare tool name: search_translations, get_routes, get_component_source, etc.
• The text MUST contain get_component_source (source verification)
• The text MUST contain search_translations (translation verification)

For standalone reviews, write the review output to the path provided by the user.

Programmatic Enforcement

After the review, two separate scripts validate the output:

• review_enforcement.py (Phase 7) -- verifies this review output contains at least 3 MCP verification entries, source verification, and translation verification. Overrides verdict to NEEDS_FIXES if checks fail.
• report.py (Phase 8) -- generates Polarion HTML and runs structural convention validation. Separate from the review enforcement.

MCP Availability

MCP	Used In	Fallback
acm-source	Step 4 (MCP verification spot-checks)	Cannot perform MCP verifications. Verdict MUST be NEEDS_FIXES with note: "acm-source MCP unavailable — MCP verification skipped. Re-run review when MCP is available."

acm-source is mandatory for this skill — the 3-verification minimum cannot be met without it. Do not report PASS without MCP verification.

Critical Rules

• ALWAYS perform at least 3 MCP verifications -- no exceptions
• ALWAYS read the primary component source to verify at least ONE factual claim
• ALWAYS cross-reference area knowledge file
• Flag as BLOCKING if any test step states a numeric threshold without evidence
• Flag as BLOCKING if filter prefixes/conditions don't match source code
• The verdict MUST be either PASS or NEEDS_FIXES -- no ambiguity