stp-reviewer

name	stp-reviewer
description	Semantic QE review of STP documents against domain rules, Jira source data, and quality standards
model	claude-opus-4-6
version	1.1.0

STP Reviewer Skill

Phase: Post-Generation Review User-Invocable: false

Purpose

Perform a comprehensive semantic QE review of a generated STP document. This is NOT structural validation (output-validator handles that). This skill evaluates the quality, accuracy, and completeness of the STP content — the same judgment a human QE reviewer would apply.

When to Use

Invoked by the review-stp command after an STP has been generated by /stp-builder.

Two-Layer Review Architecture

This skill uses a two-layer review architecture:

• Layer 1 (General): Shared pipeline rules that apply to any project. These are the same quality rules the generator follows, embedded in this skill file and always active.
• Layer 2 (Project-specific): Domain-specific examples, component lists, and mappings loaded from {project_context.config_dir}/review_rules.yaml at runtime.

Config Loading

If the review-stp command has loaded a review_rules.yaml file and passed it as context, use its contents to enhance rule checks with project-specific knowledge:

• stp_rules.abstraction.internal_components → additional internal component names for Rule A
• stp_rules.abstraction.internal_to_user_mappings → internal-to-user term translations for Rule A
• stp_rules.abstraction.acceptable_locations → where internal mechanisms are acceptable
• stp_rules.dependencies.infrastructure_not_dependency → infrastructure items for Rule D
• stp_rules.dependencies.dependency_examples → valid dependency examples for Rule D
• stp_rules.upgrade.persistent_state_indicators → persistent state markers for Rule E
• stp_rules.testing_tools.standard_tools → standard tools list for Rule G
• stp_rules.testing_tools.standard_frameworks → standard frameworks list for Rule G
• stp_rules.strategy.always_y → strategy checkbox items that must always be checked for Dimension 6
• stp_rules.strategy.requires_justification_for_y → strategy checkbox items needing justification when checked
• stp_rules.metadata.version_source → which Jira field to use for version derivation (Rule F)
• stp_rules.scope.layered_product → layered product ownership boundaries for Dimension 2

Graceful Degradation

If no review_rules.yaml exists for the project, all rules still apply using built-in defaults. Project-specific config adds precision (custom component lists, domain-specific examples) but is not required. The general rules provide comprehensive coverage on their own.

Distinction from output-validator

Concern	output-validator	stp-reviewer
Section count	YES	NO
Table row counts	YES	NO
Markdown formatting	YES	NO
Rule A-P compliance	NO	YES
Requirement coverage gaps	NO	YES
Scenario quality judgment	NO	YES
Risk accuracy	NO	YES
Cross-section contradictions	NO	YES
Source data comparison	NO	YES

Input

stp_file_path: "outputs/stp/{JIRA_ID}/{JIRA_ID}_test_plan.md"
jira_data:
  main_issue: <fetched Jira issue data>
  linked_issues: <linked issues array>
  subtasks: <subtasks array>
project_context: <from project-resolver, includes repo_rules>
review_rules: <from review_rules.yaml, if available>

repo_rules Integration

When project_context.repo_rules is available, the reviewer uses:

• repo_rules.stp_template — For Rule B (template comparison). Compare the STP structure against the fetched template, not the local copy. This is the team's current official template.
• repo_rules.stp_guide — Additional context for reviewing testing goals (SMART criteria, priority levels P0/P1/P2, entry/exit criteria), risk categories, and STP lifecycle expectations.
• repo_rules.testing_tiers — For validating tier classifications in Section III. The fetched tiers guide defines what Tier 1 vs Tier 2 means for this project. Use it to validate Rule J (One Tier Per Row) and Dimension 3 (Scenario Quality) tier distribution checks.

Output

A structured review report written to:

outputs/reviews/{JIRA_ID}/{JIRA_ID}_stp_review.md

---

Review Dimensions

The review evaluates 7 dimensions. Each dimension produces findings classified as:

Severity	Meaning	Impact on Verdict
CRITICAL	Factual error, missing coverage, or rule violation that makes the STP unusable	Blocks approval
MAJOR	Significant quality issue that should be addressed	Does not block but flags for attention
MINOR	Improvement suggestion or stylistic issue	Informational only

---

Dimension 1: Domain Judgment Rule Compliance

Check each rule (A through P) against the STP content. For each rule, sample the relevant sections and evaluate compliance.

Rule A — Abstraction Level

What to check:

• Read every item in Scope of Testing (II.1): does it describe what the user does/observes, or an internal mechanism?
• Read every Testing Goal: same check.
• Read every Test Scenario in Section III: same check.
• Read every Requirement Summary in Section III: is it in user-story format ("As a [role], I want...")?

QE technical terms allowlist (always acceptable in QE-facing STPs):

• API, CRD, CustomResource, RBAC, webhook, operator (when referring to deployed software)
• Feature gate (when describing availability control, not internal implementation)
• Pod, namespace, node, cluster (standard Kubernetes user-facing concepts)
• CLI tool names listed in project.yaml scope_boundaries.cli_tools

If project config is loaded, stp_rules.abstraction.qe_terms_allowed extends this list with project-specific allowed terms.

Red flags (CRITICAL if found in Scope/Goals/Scenarios):

• Internal component references (controller, reconciler, evaluator, sync, annotation, label, trigger, condition) that are NOT in the allowlist above. If project config is loaded, also check stp_rules.abstraction.internal_components for additional project-specific names.
• Implementation-level verbs (reconcile, sync, trigger, propagate)
• Low-level API verbs used as test descriptions (e.g., "deletecollection", "PATCH the CR")
• References to how to run tests or test commands (e.g., "run pytest with...", "execute via CI")
• Internal code paths or file references (e.g., "pkg/storage/handler.go")
• Test framework constructs (e.g., pytest decorators, Ginkgo markers, fixture names) — these are implementation details that belong in the STD, not the STP
• Cross-document reference leaks — mentioning the STD within the STP or vice versa (e.g., "the STD will describe...", "see STD for details"). Each document should be self-contained at its abstraction level. Describe what is needed in abstract terms.
• Internal infrastructure concepts when user-level concepts exist. If project config is loaded, use stp_rules.abstraction.internal_to_user_mappings for project-specific translations. General examples: using internal process names where user-facing product names exist.

Borderline term resolution: When a term could be either internal or user-facing depending on context, apply the litmus test below. Do NOT flag terms from the allowlist.

Red flags (MAJOR if found in Requirement Summary):

• Missing "As a [role]" format
• Technical subject instead of user role (e.g., "The controller..." instead of "As an admin...")

Acceptable locations for internal mechanisms: If project config is loaded, use stp_rules.abstraction.acceptable_locations. Otherwise, the general defaults are:

• Technology Challenges (I.3 sub-items)
• Risks (II.5)
• Checkbox sub-items
• Known Limitations (I.2)

Evaluation method: For each Scope item, Goal, Requirement Summary, and Test Scenario, apply the litmus test:

"Would this sentence appear in customer-facing release notes?"

• YES → PASS
• NO → Flag with severity and suggest rewrite

Rule A.2 — Language Precision

What to check:

• Read every Scope item, Testing Goal, and Test Scenario for colloquial, imprecise, or unprofessional language.

Red flags (MAJOR):

• Anthropomorphizing infrastructure ("without the service noticing", "the cluster decides")
• Colloquial phrasing ("works fine", "stuff like that", "basically")
• Vague qualifiers ("properly", "correctly", "as expected") without measurable criteria
• Hedging language ("should probably", "might need to")

Acceptable:

• Technical precision: "with minimal service disruption" instead of "without the service noticing"
• Measurable outcomes: "within 30 seconds" instead of "quickly"
• Standard QE vocabulary: "verify", "validate", "confirm", "assert"

Severity: MAJOR for anthropomorphization or colloquial phrasing in formal STP sections. MINOR for vague qualifiers that could be made more specific.

Rule B — Section I Meta-Checklist

What to check:

• Read Section I.1 (Requirements Review): it uses a checkbox format (- [ ] items) with 5 items: Review Requirements, Understand Value and Customer Use Cases, Testability, Acceptance Criteria, NFRs. Each checkbox has indented sub-bullets for detail. Feature-specific observations go in the sub-item text.
• Read Section I.2 (Known Limitations): new location for limitations (moved from II.6).
• Read Section I.3 (Technology Review): it uses a checkbox format with 5 items: Developer Handoff, Technology Challenges, API Extensions, Test Environment Needs, Topology. Each checkbox has indented sub-bullets for detail.

How to verify: Read the STP template from project_context.repo_rules.stp_template (preferred) or fall back to {project_context.config_dir}/templates/stp/stp-template.md. Compare each section against the template's prescribed structure and format (tables vs checkboxes, required columns, section ordering).

Red flags (MAJOR):

• Checkbox sub-items contain acceptance criteria lists that belong in Section III
• Checkbox sub-items contain feature-specific technical descriptions that belong elsewhere
• Checkbox sub-items contain value propositions instead of review observations
• Checkbox items are checked but sub-items are empty or missing

Red flags (CRITICAL) — Empty required fields:

• Section III (Requirements-to-Tests Mapping) has no items or is empty — "this is the core of the STP and must be filled"
• A checkbox is checked but its sub-items are empty — "if this was not done/not needed, please update the details"
• Any required section has unfilled placeholder text still present

Red flags (MAJOR) — Stale template:

• STP is not based on the current template version (compare against repo_rules.stp_template or {project_context.config_dir}/templates/stp/stp-template.md)
• Template sections that were updated are not reflected in the STP
• STP uses checkboxes for Section I when the official template uses tables (or vice versa)

Rule C — Prerequisites vs Test Scenarios

What to check:

• Read every row in Section III. Does any Test Scenario describe a configuration requirement rather than a testable behavior?

Pattern matching for prerequisites:

• "X must be set/enabled/configured"
• "Cluster/environment needs X"
• "Requires X to be deployed/installed"
• "Feature gate must be enabled" (prerequisite, not scenario — UNLESS the scenario is "verify feature gate controls availability")

Where prerequisites should appear instead:

• Test Environment (II.3) — bullet list format
• Entry Criteria (II.4)
• Special Configurations (II.3 last item)

Severity: CRITICAL if a prerequisite appears as a test scenario in Section III without a corresponding behavioral verification.

Rule D — Dependencies = Team Delivery

What to check:

• Read the Dependencies checkbox item in Test Strategy (II.2).
• If checked, check that the sub-items describe another team's delivery, not pre-existing infrastructure.

Red flags (MAJOR): Items that are infrastructure, not dependencies. If project config is loaded, use stp_rules.dependencies.infrastructure_not_dependency for project-specific examples. General pattern: listing pre-existing platform infrastructure as a "dependency" when it is actually a test environment requirement.

Acceptable dependency examples: If project config is loaded, use stp_rules.dependencies.dependency_examples. General pattern: another team must deliver a change, merge a PR, or release a component before this feature can be tested.

Rule E — Upgrade Testing Applicability

What to check:

• Read the Upgrade Testing checkbox item in Test Strategy (II.2).
• If checked, verify the feature creates persistent state that must survive upgrades.
• If unchecked, verify the feature does NOT create persistent state.

Decision criteria: Ask: "If a cluster upgrades from version N to N+1, does existing data/state created by this feature need to be preserved or converted?"

• YES → Upgrade Testing should be Y
• NO → Upgrade Testing should be N/A

If project config is loaded, stp_rules.upgrade.persistent_state_indicators provides project-specific indicators of persistent state.

Red flags (MAJOR):

• Checked for a one-time operation (e.g., patching a spec field)
• Unchecked for a feature that stores configuration in persistent resources

Rule F — Version Derivation

What to check:

• Read the version-related rows in Test Environment (II.3).
• Compare against the Jira ticket's version field and project_context.versioning.

If project config is loaded, stp_rules.metadata.version_source specifies which Jira field to use (e.g., fix_version).

Red flags (MAJOR):

• Hardcoded older versions that don't match the Jira version field
• Missing version when the version field is available in Jira

Acceptable:

• "TBD" if the version field is not set in Jira

Rule G — Testing Tools Section

What to check:

• Read Section II.3.1 (Testing Tools & Frameworks) — bullet list format.
• If items list standard tools, flag as unnecessary.

If project config is loaded, use stp_rules.testing_tools.standard_tools and stp_rules.testing_tools.standard_frameworks for the project-specific standard tools list. Otherwise, the general principle applies: tools and frameworks that are standard for the project's test infrastructure should not be listed.

Red flags (MINOR):

• Standard test framework listed
• Standard CLI tools listed
• CI/CD tools listed unless feature-specific

Acceptable:

• Empty list (feature uses only standard tools)
• Non-standard tools (custom profiler, specialized network tool)

Rule G.2 — Environment Specificity

What to check:

• Read each item in the Test Environment list (II.3).
• Are the entries feature-specific or generic boilerplate?

Red flags (MINOR):

• Generic cluster/environment descriptions without specifying why they are needed for this feature
• Entries that would be identical for any unrelated feature

Acceptable:

• Entries that explain the feature-specific reason for each requirement

Rule H — Risk Deduplication

What to check:

• For each risk in Section II.5 (checkbox format), check if the same information already appears in Test Environment (II.3).

Red flags (MAJOR):

• Risk entries that duplicate information already in the Test Environment list

Risks should describe:

• Timeline uncertainties
• Coverage gaps
• Untestable aspects
• Resource constraints (team, not infrastructure)
• External dependencies
• Unknown failure modes

Rule I — QE Kickoff Timing

What to check:

• Read the Developer Handoff checkbox item in Section I.3 and its sub-items.
• Does it describe kickoff during design phase (before implementation)?

Red flags (MAJOR):

• "Should be scheduled after PR is merged"
• "Kickoff after implementation is complete"
• "Post-merge review meeting"

Acceptable:

• "QE kickoff should be scheduled during feature design phase"
• "Kickoff scheduled/completed" (with date if available)

Rule J — One Tier Per Row

What to check:

• Read every item in Section III (bullet-based format). Does each item specify exactly ONE tier?

Red flags (CRITICAL):

• "Tier 1 (Functional), Tier 2 (End-to-End)" in same bullet
• "Tier 1 / Tier 2" in same bullet
• Multiple tiers separated by comma, slash, or "and"

Rule K — Cross-Section Consistency

What to check: Compare content across sections for contradictions.

Specific contradiction patterns:

Check	Section A	Section B	What must be true
1	Scope (II.1)	Out of Scope (II.1)	No item appears in both
2	Testing Goals (II.1)	Known Limitations (I.2)	Goals do not promise what limitations exclude
3	Section I sub-items	Known Limitations (I.2)	Sub-items do not claim capabilities that limitations exclude
4	Test Strategy (II.2)	Section III	If strategy checkbox is unchecked, no scenarios of that type exist
5	Entry Criteria (II.4)	Test Environment (II.3)	Entry criteria reference environment requirements consistently
6	Scope (II.1)	Section III	All scope items have at least one test scenario
7	Out of Scope (II.1)	Section III	No scenarios test out-of-scope items

Severity: CRITICAL for contradictions between Scope and Out of Scope, or Goals and Limitations. MAJOR for other inconsistencies.

Rule L — Section Content Validation (Misplaced Content)

What to check: Verify that content appears in the correct STP section. Content in the wrong section indicates the author misunderstands the STP structure.

Section content rules:

Section	Should Contain	Should NOT Contain
Scope of Testing (II.1)	Testable capabilities and feature areas (checkbox format)	Implementation details, how to run tests
Out of Scope (II.1)	Features/areas explicitly excluded with rationale and PM ack (checkbox format)	Limitations (those go in I.2)
Testing Goals (II.1)	User-observable outcomes to validate	Internal mechanism validations
Test Strategy (II.2)	Checkbox items with feature-specific sub-items	Detailed test scenarios (those go in III)
Dependencies (II.2)	Other team deliveries with Jira references (checkbox item)	Infrastructure requirements (those go in II.3)
Test Environment (II.3)	Infrastructure and platform requirements (bullet list)	Risks (those go in II.5)
Testing Tools (II.3.1)	Only non-standard tools and frameworks (bullet list)	Standard tools (use project config list if available)
Risks (II.5)	Genuine uncertainties with mitigations (checkbox format)	Known environment requirements (those go in II.3)
Known Limitations (I.2)	Feature boundaries and restrictions	Out-of-scope items (those go in II.1 Out of Scope)
Section III	Testable behaviors with requirement mapping (bullet-based)	Configuration prerequisites, setup instructions

Red flags (MAJOR):

• Scenarios appearing in Scope instead of Section III
• Infrastructure requirements listed as Dependencies
• Environment requirements listed as Risks
• Test execution instructions in Scope or Goals
• User stories appearing in Technology Challenges (I.3) instead of Section I.1
• Testability section describing specific test cases that belong in Section III — Testability should describe whether something can be tested, not what to test
• General description containing step-level testing details that belong in Section III

Limitation vs Out-of-Scope Semantic Distinction:

A common misclassification is confusing Limitations with Out-of-Scope items. These have precise, different meanings:

Category	Meaning	Example
Known Limitations (I.2)	A constraint prevents testing — you cannot test this even if you wanted to	"Feature X is not supported by the product"
Out of Scope (II.1)	A deliberate decision to exclude — you could test this but chose not to	"Feature X will be tested in a follow-up cycle"

Red flags (MAJOR) — Limitation vs Out-of-Scope confusion:

• Item in Limitations that has no actual constraint preventing testing — if it could be tested but is simply not planned, move to Out of Scope
• Item in Out-of-Scope that is actually a product limitation — if the product does not support it, move to Known Limitations

Red flags (MINOR) — Risk mitigation linkage:

• Out-of-scope item with risk implications but no corresponding entry in the Risks section (II.5). When excluding a test area, the risk of not testing it should be acknowledged with a mitigation strategy.

Severity: MAJOR for content in wrong section. MINOR for borderline cases.

Rule M — Deletion Test (ISTQB)

What to check: For each section of the STP, apply the ISTQB deletion test: "If this section were removed entirely, would the Go/No-Go decision for the test effort be hindered?"

Process:

1. Read each section's content
2. Ask: Does this content contribute information that a QE lead or PM needs to decide whether testing is ready to proceed?
3. If the content does not contribute to the decision, it is excessive detail

Red flags (MAJOR):

• A section contains multiple paragraphs of background information that repeats what is already in the Jira ticket or VEP — the STP should reference these, not duplicate them
• Overly detailed implementation descriptions in Scope or Goals that go beyond what is needed to understand what to test
• Testing details in Section I that belong in Section III
• Environment details so granular that they describe CI job configuration rather than what infrastructure is needed

Red flags (MINOR):

• Sections that are correct but could be more concise without losing decision-relevant information

Severity: MAJOR for content that adds bulk without aiding the test decision. MINOR for content that is correct but unnecessarily verbose.

Rule N — Link/Reference Validation

What to check: Verify all links and references in the STP are valid, relevant, and not stale.

Process:

1. Extract all URLs from the STP (VEP links, Jira links, PR URLs, enhancement links, documentation references)
2. For each link, verify:
   • Syntactically valid URL format
   • Points to the correct domain for the content type (e.g., enhancement links should point to the project's design proposal repository, not an unrelated project)
   • Not a personal fork URL (e.g., github.com/username/project instead of the official organization URL) — personal forks may become stale or deleted

Red flags (CRITICAL):

• Link that points to a completely wrong resource (e.g., Jira link to wrong project, VEP from a different SIG)

Red flags (MAJOR):

• Enhancement/VEP link points to the wrong domain for the feature type (e.g., storage VEP linked for a network feature)
• Stale references — bug references with resolved/verified status that should be updated, references to deprecated/removed technologies, or outdated identifiers
• Jira reference that does not match the input Jira ID
• PR URLs referencing unrelated repositories

Red flags (MINOR):

• Enhancement links to personal forks that may become stale — prefer upstream/official repository links
• Links that are syntactically valid but cannot be verified as current without network access

Note: This rule consolidates link validation that was previously scattered across Dimensions 5 and 7. All link/reference findings should be reported under this rule.

Rule O — Untestable Aspects Documentation

What to check: When a testing goal, scope item, or scenario is marked as "not testable at this stage," "cannot be verified," or "deferred," verify the documentation is complete.

Required documentation for untestable items:

1. Reason: Why it cannot be tested (missing infrastructure, product limitation, dependency not yet delivered)
2. Timeline/Condition: When it becomes testable ("after version X.Y", "when dependency is delivered", "in follow-up STP")
3. Risk entry: Acknowledgment in the Risks section (II.5) that there is a testing gap

Red flags (CRITICAL) — Priority-Testability Contradiction:

• A P0 (highest priority / GA-blocking) goal or scenario that is explicitly documented as "not testable at this stage" or "cannot be verified" is a logical contradiction. A P0 item cannot simultaneously be the highest priority AND untestable. Resolution options:
  1. Downgrade the priority to P2 (deferred)
  2. Add a timeline for when it becomes testable
  3. Move to a follow-up STP
  4. Resolve the testability blocker

Red flags (MAJOR):

• Untestable item with no reason documented
• Untestable item with no timeline or condition for when it becomes testable
• Untestable item with no corresponding risk entry in Section II.5

Red flags (MINOR):

• Untestable item has all three required elements but the timeline is vague ("sometime in the future" instead of a specific version or condition)

Rule P — Testing Pyramid Efficiency (Fix-Scope Awareness)

Activation guard: This rule ONLY applies when BOTH conditions are met:

1. Jira issue type is Bug, Customer Case, or Defect
2. PR/github data is available (fix_scope is not null)

If either condition is unmet, skip this rule entirely — mark as PASS with "N/A — not a bug ticket or no PR data available."

Process:

Step 1 — Classify fix scope (deterministic algorithm):

From fix_scope.key_changes and fix_scope.files_changed:

1. Extract distinct packages: count unique top-level package paths (e.g., pkg/controller/ and pkg/api/ = 2 packages). Test files (*_test.go, test_*.py) do not count.
2. Count modified functions (from fix_scope.key_changes list length, excluding test functions).
3. Check cluster interaction: does ANY modified file's package path contain cluster-facing indicators? Indicators: client-go, controller-runtime, k8s.io/api, or project-specific cluster packages from components.yaml.

Classification table:

Distinct packages	Modified functions	Cluster interaction	Classification
1	1	No	single-function-isolated
1	1	Yes	single-function-cluster
1	2+	Any	single-package
2+	Any	Any	multi-package
Any	Any	Any + PR description mentions "user workflow" or "end-to-end"	multi-step-workflow

If fix_scope.files_changed is "large" (>5k lines), classify as multi-package.

Step 2 — Evaluate tier efficiency:

For each scenario in Section III, evaluate whether the proposed tier is the minimum that provides equivalent coverage.

Fix Scope → Expected Minimum Tier:

Fix Scope Classification	Minimum Viable Tier	Reasoning
single-function-isolated	Unit Tests	Mocked test validates the fix
single-function-cluster	Tier 1	Single operation in cluster
single-package	Tier 1	Single feature verification
multi-package	Tier 2	Cross-component interaction required
multi-step-workflow	Tier 2	Full workflow verification needed

Red flags (MAJOR):

• All scenarios for a single-function fix are Tier 2 (missing lower-tier test)
• A customer case with a 1-file fix has no Unit or Tier 1 scenario

Red flags (MINOR):

• Tier 2 exists for a small fix — acceptable for regression confidence but note that a lower-tier test should also exist

NOT a finding:

• Having both Tier 1 (verify fix) AND Tier 2 (verify workflow) is ideal
• Feature tickets or multi-package bugs at Tier 2 are fine

Recommendation format: Suggest minimum viable tier AND recommend keeping the higher-tier test. Example: "Fix modifies validateCPUModel() in pkg/cpu/. Consider adding a Tier 1 test that directly verifies the validation logic. The existing Tier 2 migration scenario provides good regression coverage but a Tier 1 test verifies the fix more efficiently."

---

Dimension 2: Requirement Coverage

What to check: Compare the STP's Section III against the source Jira data to find coverage gaps.

Process:

1. Read the Jira main issue's acceptance criteria
2. Read each linked issue's summary and description
3. For each acceptance criterion / user story in Jira, check if Section III (bullet-based format) has a corresponding scenario

Red flags:

• CRITICAL: Acceptance criterion in Jira with no corresponding test scenario bullet
• CRITICAL: Coverage rate below threshold — if acceptance_criteria_covered / total < 0.70 (less than 70% of acceptance criteria have corresponding scenarios), this is a systemic coverage gap that blocks approval regardless of individual findings
• CRITICAL: Any P0 or GA-blocking acceptance criterion is uncovered — a single missing P0 criterion is sufficient for CRITICAL even if overall coverage rate is above 70%
• MAJOR: Linked issue describes a use case not reflected in any scenario
• MAJOR: Jira description mentions error handling / edge cases with no negative scenarios
• MINOR: Feature overview mentions a capability not tested in Section III bullets

Coverage metrics to report:

acceptance_criteria_covered: X/Y
acceptance_criteria_coverage_rate: X/Y (percent%)
p0_criteria_covered: X/Y
linked_issues_reflected: X/Y
negative_scenarios_present: true/false
edge_cases_identified: X (from Jira) / Y (in STP)

Coverage threshold evaluation: After scoring all acceptance criteria, calculate the coverage rate. If below 70%, produce a CRITICAL finding: "Acceptance criteria coverage is {X}% ({covered}/{total}). Minimum threshold is 70%. {list uncovered criteria}." This ensures incomplete STPs do not proceed to STD generation.

Value Proposition & Use Case Quality

The Value Proposition and Use Cases sections describe why the feature matters. These must be customer-facing, factually accurate, and actionable.

Red flags (MAJOR) — Internal audience framing:

• Value proposition lists QE, Dev, SRE, or other internal teams as "users" or beneficiaries. The value proposition must describe customer/end-user benefit.
  • BAD: "For QE — provides measurements for internal benchmarking"
  • GOOD: "Provides users with published network performance thresholds per release"
• Use cases describe test activities instead of user scenarios.
  • BAD: "QE will run measurements across configurations"
  • GOOD: "User deploys application and expects sub-second failover"

Red flags (MAJOR) — Value proposition inaccuracy:

• Stated value does not match the actual feature capability (per Jira/VEP). Cross-reference the value against the Jira description to verify it describes the correct incremental benefit.
• Value is duplicated from a parent/related feature without describing what this specific feature adds beyond the parent.

Red flags (MINOR):

• Use case section is empty or contains only generic text ("Users will benefit from this feature") without concrete scenarios.

Proactive Scope Completeness Probing

Beyond checking Jira acceptance criteria against scenarios, the reviewer must proactively challenge scope completeness — asking "what about X?" for test areas that are commonly overlooked. This is the most common type of human reviewer feedback.

UI Coverage Challenge:

• If the feature has a user-facing component (linked UI Jira, "UI" mentioned in description, or Usability Testing checked in strategy), check whether UI testing is addressed in either Scope or Out of Scope.
• If UI testing is neither in scope nor explicitly excluded in Out of Scope, flag:
  • MAJOR: "Feature has a user-facing component but UI testing is not addressed. Add UI testing to scope, or add to Out of Scope with rationale and PM acknowledgment."

Negative / Edge Case Challenge:

• Count the negative scenarios in Section III. If fewer than 2 negative scenarios exist for a feature with 10+ total scenarios, flag:
  • MAJOR: "Only {N} negative scenario(s) found among {total} scenarios. Consider adding edge cases: What happens with invalid input? What about boundary conditions? What if dependent resources are missing?"
• Proactively suggest missing negative scenarios based on feature type:
  • Network features: "What about connectivity loss during operation? Conflicting configurations?"
  • Storage features: "What about insufficient storage? Concurrent operations on same volume?"
  • RBAC features: "What about unauthorized users? What about escalation attempts?"

OS / Platform Coverage Challenge:

• If test scenarios only reference one OS type, flag:
  • MINOR: "All scenarios reference {OS} only. Is it acceptable that other OS types are not tested? If so, add to Out of Scope with PM sign-off."

Monitoring / Alerting Alignment:

• If Monitoring Testing is checked in Test Strategy (II.2), verify that at least one scenario in Section III covers monitoring or alerting behavior.
• If no monitoring scenario exists despite the item being checked:
  • MAJOR: "Monitoring Testing is checked in strategy but no monitoring/alerting scenarios exist in Section III. Add scenarios or uncheck the strategy item."

Regression Scope Challenge:

• If Regression Testing is checked in Test Strategy (II.2), verify the STP addresses what regression tests are needed — either specific scenarios or a reference to existing test suites that cover regression.
• If regression is checked but scope is vague ("regression testing will be done"):
  • MINOR: "Regression Testing is checked but scope is not specific. What existing tests need to pass? What existing behaviors must not change?"

Cross-Team / Cross-SIG Impact:

• If the feature interacts with multiple SIGs (Participating SIGs has 2+), verify that cross-team testing is addressed in scope or out of scope.
• If a participating SIG's area is not covered by any scenario:
  • MAJOR: "Participating SIG {sig_name} is listed but no scenarios test {sig_name}-related behavior. Add cross-team scenarios or explain why testing is not needed."

Explicit Out-of-Scope Acknowledgment:

• Out of Scope items should have rationale and ideally PM/lead acknowledgment.
• If Out of Scope items lack rationale:
  • MINOR: "Out of Scope item '{item}' has no rationale. Add justification and consider getting PM sign-off for scope exclusions."

Invalid Scope Exclusion (Subtractive Review):

• For each test scenario in Section III, evaluate whether it tests behavior the feature actually supports. If a scenario targets behavior the feature explicitly does not support (per VEP/Jira user stories/non-goals), flag for removal:
  • MAJOR: "Scenario '{scenario}' tests unsupported/unexpected user behavior. Per the VEP/user story, this behavior is not a feature goal. Remove or reclassify."
• Cross-reference scope items against VEP non-goals section when available.
• If a scenario describes "invalid user behavior" that produces undefined results, it should not be tested — undefined behavior cannot be validated.

Team Ownership Boundary Check (Layered Product): If project config is loaded and stp_rules.scope.layered_product is defined, apply layered product ownership rules:

• The product assumes platform features work correctly. Test scenarios should focus on product-specific behavior.
• If a test scenario exercises only platform-level functionality without product-specific involvement, flag:
  • MAJOR: "Scenario '{scenario}' exercises platform functionality without product-specific involvement. This may belong to the upstream/platform team. Verify with the owning team or justify why this product must test this."
• If the feature has cross-team implications (Participating SIGs has 2+), verify that per-scenario ownership is clear — who tests what.

If no layered product config is defined, skip this check.

Epic-Anchored Completeness:

• When Jira data is available, compare the STP scope against the Jira epic description.
• For each major requirement stated in the epic, verify it is either:
  1. Covered by a test scenario in Section III, OR
  2. Explicitly excluded in Out of Scope with justification
• If a requirement from the epic is neither covered nor excluded:
  • MAJOR: "Epic requirement '{requirement}' is not addressed in the STP. Add a corresponding test scenario or add to Out of Scope with rationale."

---

Dimension 3: Scenario Quality

What to check: Evaluate the quality of each test scenario in Section III (bullet-based format).

Quality criteria:

Criterion	Good	Bad
Specificity	"Verify service is reachable after config change"	"Verify feature works"
User perspective	"Verify user sees updated resource after edit"	"Verify internal sync protocol succeeds"
Brevity	5-10 words, single phrase	Multi-sentence description
Actionability	Clear what to test	Vague or ambiguous
Uniqueness	Each scenario tests a distinct behavior	Duplicate or overlapping scenarios

Red flags:

• CRITICAL: Generic scenario ("Verify feature works correctly")
• MAJOR: Technical/internal scenario (uses internal mechanism language)
• MAJOR: Duplicate scenario (two bullets testing the same thing)
• MINOR: Overly verbose scenario (>15 words)

Distribution checks:

• Are there both positive AND negative scenarios?
• Is the P0/P1/P2 distribution reasonable? (P0 for core, not everything)
• Is the Tier 1/Tier 2 split appropriate? (complex workflows as Tier 2)
• Are there at least 2 scenarios per distinct requirement?

Priority Validation Heuristic

Apply these heuristics to verify P0/P1/P2 assignments:

Heuristic	Expected Priority
First positive scenario for the feature's primary capability	P0
Core happy-path functionality	P0
Error handling / negative scenarios	P1 or P2
Regression / backward compatibility	P1
Edge cases / rare conditions	P2
Integration with other features	P1 or P2

Red flags (MAJOR):

• The primary positive scenario for a feature is NOT P0
• All scenarios are P0 (priority inflation — nothing is highest if everything is)
• Error handling scenarios ranked higher than core functionality

Red flags (MINOR):

• More than 50% of scenarios are P0 (suggests priority inflation)
• No P2 scenarios at all (suggests under-differentiation)

---

Dimension 4: Risk & Limitation Accuracy

What to check: Evaluate risks and limitations against the source Jira data and PR information.

For each risk:

• Is it a real uncertainty, or a known environment requirement? (Rule H overlap)
• Is the mitigation strategy actionable?
• Is the status tracked?

For known limitations:

• Do limitations match what the Jira ticket / PR data says about the feature's boundaries?
• Are there limitations mentioned in Jira that are missing from the STP?
• Are there limitations in the STP that contradict Jira data?

Red flags:

• MAJOR: Jira mentions a known limitation not reflected in Section I.2
• MAJOR: Risk mitigation is vague ("Monitor the situation")
• MINOR: Risk status all unchecked (expected for draft, but note it)

---

Dimension 5: Scope Boundary Assessment

What to check: Evaluate whether the scope is appropriate for the feature described in Jira.

Process:

1. Read the Jira issue description and extract what the feature does
2. Compare against Scope of Testing (II.1)
3. Compare against Out of Scope (II.1)

Red flags:

• CRITICAL: Scope includes capabilities the feature does not provide (per Jira)
• CRITICAL: Scope includes testing of unsupported/invalid user behavior — if the feature VEP or user stories explicitly list non-goals, scenarios testing those non-goals should not be in scope. Invalid user behavior produces undefined results that cannot be validated.
• MAJOR: Important feature capability missing from scope
• MAJOR: Out-of-scope item that should actually be tested
• MAJOR: Scope does not align with Jira epic requirements — scope items should be traceable to the epic's stated goals or linked user stories
• MINOR: Scope is too broad relative to the Jira ticket

---

Dimension 6: Test Strategy Appropriateness

What to check: For each checkbox item in the Test Strategy list (II.2):

1. Is the checked/unchecked state correct based on the feature?
2. Are the sub-items substantive and feature-specific?

Classification validation: If project config is loaded, use stp_rules.strategy.always_y for items that must always be checked, and stp_rules.strategy.requires_justification_for_y for items that need justification when checked. Otherwise, apply general rules:

• Functional Testing: Should always be checked for any feature
• Automation Testing: Should always be checked (QualityFlow generates automated tests)
• Performance Testing: Check only if feature has latency/throughput requirements
• Security Testing: Check only if feature changes RBAC, auth, or security boundaries
• Usability Testing: Check only if feature has UI component
• Upgrade Testing: Apply Rule E logic
• Dependencies: Apply Rule D logic

Red flags:

• MAJOR: Functional Testing unchecked
• MAJOR: Strategy checkbox item checked with generic sub-items ("Will be tested")
• MINOR: Strategy checkbox item unchecked without justification in sub-items

N/A vs Y Classification Challenge

Proactively challenge checked/unchecked states. Some items are checked with boilerplate sub-items when they should actually be unchecked (or vice versa).

Red flags (MAJOR) — Should be unchecked but is checked:

• Performance Testing checked but sub-items only say "validate operation completes within acceptable time" without specific latency/throughput requirements — this is functional testing, not performance testing. If no SLA or benchmark target exists, uncheck.
• Security Testing checked but sub-items only mention standard RBAC that any feature has — if the feature does not change security boundaries, uncheck.
• Usability Testing checked but no UI component exists and no UX workflow is impacted — if the feature is API-only, uncheck.

Red flags (MAJOR) — Should be checked but is unchecked:

• Upgrade Testing unchecked but the feature creates persistent state — should be checked per Rule E. If project config is loaded, check against stp_rules.upgrade.persistent_state_indicators.
• Monitoring Testing unchecked but the feature introduces new metrics or alerts — should be checked.
• Regression Testing unchecked for any GA feature — regression is almost always applicable.

Red flags (MINOR) — Technology Challenges misuse:

• A Technology Challenges checkbox item (in I.3) that describes a test requirement rather than a genuine technical challenge should be flagged: "This is not a real challenge but rather what is needed to test the feature. Consider removing or rewording."
• Example: "Need to configure storage backend" is a test requirement, not a challenge. A real challenge: "Storage backend behavior differs across vendors, making consistent testing difficult."

Red flags (MAJOR) — Unchecked cross-referencing:

• If a strategy item is unchecked, verify the rationale is consistent with the Out-of-Scope section. If the item is excluded, the same exclusion should appear in Out-of-Scope or have a sub-item explaining why it does not apply.
• Unchecked reasoning that cites only infrastructure type without explaining why that infrastructure type makes testing inapplicable is insufficient.
• If unchecked because another team covers the testing, name the team explicitly:
  • BAD: "Not applicable — covered elsewhere"
  • GOOD: "Not applicable — {team_name} covers {area} testing in their test plan"

Red flags (MINOR) — Bare unchecked entries:

• When a checkbox item is unchecked without any explanation, require a brief rationale in the sub-items. Every unchecked classification should be justified.

---

Dimension 7: Metadata Accuracy

What to check: Compare metadata fields against Jira source data.

Field	Validation
Enhancement(s)	Links resolve to actual enhancement proposals
Feature Tracking	Correct parent/feature issue from Jira Feature Link field
Epic Tracking	Matches the input Jira ID (Epic key + Parent key)
QE Owner(s)	TBD is acceptable for draft
Owning SIG	Matches Jira labels/components
Participating SIGs	Reasonable based on feature scope

If project config is loaded, use stp_rules.metadata.sig_field to identify which metadata field maps to the SIG ownership.

Red flags:

• MAJOR: Owning SIG doesn't match Jira component/label
• MAJOR: Feature Tracking points to wrong issue
• MAJOR: Cross-artifact naming inconsistency — the feature name in the STP title does not match the Jira summary or the expected STD file name. A feature should have one consistent name across all artifacts (STP, STD, code, Jira).
• MAJOR: Approver/role assignments that do not match actual authority (e.g., listing someone without the SIG approver role as the designated approver)

Note: Enhancement link validation and stale reference checks are now consolidated under Rule N (Link/Reference Validation). Report link-related findings there.

---

Review Report Format

Generate the review report as markdown:

# STP Review Report: {JIRA_ID}

**Reviewed:** {STP_FILE_PATH}
**Date:** {YYYY-MM-DD}
**Reviewer:** QualityFlow Automated Review (v1.1.0)
**Review Rules Schema:** {review_rules._extraction_metadata.schema_version or "N/A"}

---

## Verdict: {APPROVED | APPROVED_WITH_FINDINGS | NEEDS_REVISION}

## Summary

| Metric | Value |
|:-------|:------|
| Dimensions reviewed | 7/7 |
| Critical findings | {count} |
| Major findings | {count} |
| Minor findings | {count} |
| Confidence | {HIGH / MEDIUM / LOW} |

---

## Findings by Dimension

### Dimension 1: Rule Compliance (Rules A-P)

| Rule | Status | Finding |
|:-----|:-------|:--------|
| A — Abstraction Level | {PASS/WARN/FAIL} | {description} |
| A.2 — Language Precision | {PASS/WARN/FAIL} | {description} |
| B — Section I Meta-Checklist | {PASS/WARN/FAIL} | {description} |
| C — Prerequisites vs Scenarios | {PASS/WARN/FAIL} | {description} |
| D — Dependencies | {PASS/WARN/FAIL} | {description} |
| E — Upgrade Testing | {PASS/WARN/FAIL} | {description} |
| F — Version Derivation | {PASS/WARN/FAIL} | {description} |
| G — Testing Tools | {PASS/WARN/FAIL} | {description} |
| G.2 — Environment Specificity | {PASS/WARN/FAIL} | {description} |
| H — Risk Deduplication | {PASS/WARN/FAIL} | {description} |
| I — QE Kickoff Timing | {PASS/WARN/FAIL} | {description} |
| J — One Tier Per Row | {PASS/WARN/FAIL} | {description} |
| K — Cross-Section Consistency | {PASS/WARN/FAIL} | {description} |
| L — Section Content Validation | {PASS/WARN/FAIL} | {description} |
| M — Deletion Test | {PASS/WARN/FAIL} | {description} |
| N — Link/Reference Validation | {PASS/WARN/FAIL} | {description} |
| O — Untestable Aspects | {PASS/WARN/FAIL} | {description} |
| P — Testing Pyramid Efficiency | {PASS/WARN/FAIL} | {description} |

### Dimension 2: Requirement Coverage

| Metric | Value |
|:-------|:------|
| Acceptance criteria covered | {X}/{Y} |
| Linked issues reflected | {X}/{Y} |
| Negative scenarios present | {YES/NO} |
| Coverage gaps found | {count} |

**Gaps identified:**
{list of uncovered acceptance criteria or use cases, if any}

### Dimension 3: Scenario Quality

| Metric | Value |
|:-------|:------|
| Total scenarios | {count} |
| Tier 1 | {count} |
| Tier 2 | {count} |
| P0 | {count} |
| P1 | {count} |
| P2 | {count} |
| Positive scenarios | {count} |
| Negative scenarios | {count} |

**Scenario-level findings:**
{list of specific findings per scenario row, if any}

### Dimension 4: Risk & Limitation Accuracy

{findings}

### Dimension 5: Scope Boundary Assessment

{findings}

### Dimension 6: Test Strategy Appropriateness

{findings}

### Dimension 7: Metadata Accuracy

{findings}

---

## Recommendations

{numbered list of actionable recommendations, ordered by severity}

1. **[CRITICAL]** {recommendation}
2. **[MAJOR]** {recommendation}
3. **[MINOR]** {recommendation}

---

## Confidence Notes

| Factor | Status |
|:-------|:-------|
| Jira source data available | {YES/NO} |
| Linked issues fetched | {YES/NO} |
| PR data referenced in STP | {YES/NO} |
| All STP sections present | {YES/NO} |
| Template comparison possible | {YES/NO} |
| Project review rules loaded | {YES/NO} |

**Confidence rationale:** {explanation of why confidence is HIGH/MEDIUM/LOW}

---

Verdict Criteria

Verdict	Criteria
APPROVED	0 critical findings, 0 major findings
APPROVED_WITH_FINDINGS	0 critical findings, 1+ major or minor findings
NEEDS_REVISION	1+ critical findings

---

Confidence Scoring

Level	Criteria
HIGH	Jira data available, all sections present, template comparison done, review rules default_ratio <= 0.30
MEDIUM	Jira data available but incomplete, OR template not available, OR review rules default_ratio <= 0.60
LOW	Jira data unavailable (content-only review), OR review rules default_ratio > 0.60

When review_rules._extraction_metadata.default_ratio > 0.50, include a note in Confidence Notes: "Review precision reduced: {X}% of rules using generic defaults. Consider adding project-specific review_rules.yaml or enabling repo_files_fetch."

---

Error Handling

• If STP file not found: Return error, do not produce review report
• If Jira data unavailable: Proceed with content-only review (Dimensions 1, 3, 5, 6 only). Set confidence to LOW. Skip Dimensions 2, 4 (require source comparison). Note skipped dimensions in report.
• If template not found: Skip Rule B check. Note in report.
• If project_context unavailable: Skip Rule F, G checks. Note in report.
• If review_rules.yaml not found: Continue with general rules only. All rules still apply; project-specific precision is reduced. Note in Confidence Notes.

---

End of STP Reviewer Skill