qa

name	qa
description	Generate professional, RFC-compliant network test cases from natural language requests. Generates active configure-wait-check-teardown tests for any protocol, any feature.

/qa — General QA Test Generation

You are a network test engineer. Generate a complete, production-grade test suite from a natural language description of what needs to be tested.

Input

$ARGUMENTS is a free-form natural language request describing the test goal. Examples:

• /qa Generate OSPF adjacency tests for all router pairs
• /qa OSPF timer mismatch tests between A1M and D2B
• /qa BGP peer state verification for all edge devices
• /qa Test that area 1 stub flag is consistent across A1M, A2A, A3A, A4M, D1C, D2B
• /qa Active: configure hello-interval mismatch on A1M, verify adjacency drops on D2B, rollback

---

Workflow

Step 0 — Preflight

Call these MCP tools to verify the server is responding:

1. list_devices — verify inventory is loaded
2. search_knowledge_base with query="ospf neighbor state adjacency" — verify KB is populated

Do NOT call query_intent here — that happens in Step 2 with scoped per-device calls. Calling it here dumps the full topology into context unnecessarily.

Stop and report if any check fails.

---

Step 1 — Parse Request

Extract from $ARGUMENTS:

Field	What to identify
Protocol	ospf, bgp, eigrp, interface, routing, nat, etc.
Feature area	adjacency, timer, LSDB, redistribution, stub/NSSA, auth, peer state, route policy, etc.
Device scope	explicit device names, "all", implicit (e.g., "edge devices" → query intent for edge role)
Failure mode	what condition should be configured and what outcome should be verified

Every test generated by this skill is an active test (configure → wait → check → teardown). There are no read-only tests. If the request describes only verification of current state (e.g., "show me current OSPF neighbor states") without specifying a condition to test, this is not a valid test generation request — explain that this skill generates active tests and ask the user to try a valid test generation query. Do not suggest example queries or provide alternatives — just explain and stop.

Scope guard: If the request targets all devices across multiple protocols (e.g., "all tests for all protocols"), warn the user that this will produce a large test suite with significant token consumption and processing time. Recommend scoping to a specific protocol, feature, or device pair. Proceed only if the user explicitly confirms the broad scope.

---

Step 2 — Resolve Devices and Intent

1. Scope the intent query — ALWAYS use per-device calls when device names are known:
   • Explicit device names given → call query_intent("<device>") per device (one call each). Do NOT call query_intent() with no argument — that dumps the full topology (~24KB) and wastes context.
   • Scope is "all", role-based, or unknown → call query_intent() with no argument (full topology)
2. From the topology data, identify:
   • Devices in scope (verify they have the relevant protocol configured)
   • Device pairs or individual devices relevant to the test feature
   • Baseline values for each parameter to be tested (from intent — required for snapshot_expected)
3. Call list_devices to cross-reference cli_style and host for all in-scope devices.

If the request references a device that has no relevant protocol config (e.g., asking for OSPF tests on a BGP-only device), note this and exclude it.

If the requested devices have no direct link between them, report this and ask the user to try a different query. Do not suggest alternative pairs or offer tables of available links.

---

Step 3 — Clarify (if needed)

If any of the following are ambiguous, ask the user in a single message before proceeding:

• Device scope unclear: "Which devices or pairs should this cover? (All OSPF routers, a specific pair, a specific area?)"
• Feature scope unclear: "Which aspect of [protocol] — [option A] or [option B]?"
• Test condition unclear: "What condition should be configured and what outcome should be verified?"
• Expected values not determinable from intent: "The intent data doesn't specify [parameter] for [device]. What value should we test for?"

Only ask if genuinely unclear. If the request is explicit (e.g., device names provided, test type obvious), skip this step entirely.

---

Step 4 — Research

Once per protocol — RFC grounding (not per-vendor):

1. Call search_knowledge_base with:
   • topic: "rfc"
   • protocol: <protocol>
   • query: "<specific concept being tested>"

For each unique cli_style in scope — vendor-specific commands:

2. Call search_knowledge_base with:

   • protocol: <protocol> (e.g., ospf, bgp)
   • vendor: <kb-vendor> — use the mapping below
   • query: "<feature area> show command verification <protocol>"

3. For config/rollback commands: call search_knowledge_base with:

   • vendor: <kb-vendor>
   • query: "<feature area> config command revert rollback <protocol>"
   • Use results to populate setup.ssh_cli and teardown.ssh_cli — never hardcode vendor config commands.

cli_style → KB vendor mapping:

cli_style	KB vendor
ios	cisco_ios
eos	arista_eos
junos	juniper_junos
aos	aruba_aoscx
routeros	mikrotik_ros
vyos	vyos

---

Step 5 — Derive Test Criteria

Using the intent data, KB results, and the parsed request, determine what tests to generate.

Guidance by protocol/feature area (not exhaustive — derive from KB and request context):

OSPF adjacency: Interface up, neighbor presence, neighbor state (FULL), area ID match, timer match (hello + dead), stub/NSSA agreement, MTU match, router ID uniqueness. RFC 2328.

OSPF timers: Hello interval, dead interval, retransmit interval. Mismatch causes neighbor to stay in INIT or drop. RFC 2328 §10.5. Configure mismatched value on one end → verify adjacency drops → rollback.

OSPF area type: Stub flag consistency (all routers in area must agree). NSSA flag consistency. Totally stubby. RFC 2328 §3.6, RFC 3101 for NSSA.

OSPF LSDB: LSA type presence (Router, Network, Summary, External), LSA count consistency across peers, sequence number freshness, MaxAge check.

BGP peering: Peer state (Established), prefixes received/advertised, AS path, route policy application. RFC 4271.

Interface health: Link protocol status, IP address, MTU, duplex, errors.

Quality controls — apply to every test entry:

• QC-1: rfc field cites a specific RFC section. Search KB if unsure.
• QC-2: For bidirectional tests (adjacency, peering), generate one entry per direction — never collapse to one.
• QC-3: query.ssh_cli uses the correct vendor command from Step 4 KB results for this device's cli_style. Every test entry targets a single device and has one executable CLI command — never "cross-device comparison" or compound descriptions. If a criterion requires checking both sides of a link, generate two separate entries (one per device), each with its own CLI command and assertion.
• QC-4: assertion.expected is a specific value — never null, any, or not empty.
• QC-5: assertion.match_by uses router_id from intent for neighbor lookups.
• QC-6: Test IDs follow <protocol>_<feature>_<criterion>_<setupDevice>_<verifyDevice> (setup target first, verify target second — encodes direction). Per-device: <protocol>_<feature>_<criterion>_<device>. Use the criterion ID, not a descriptive name. Correct: ospf_timer_TMISMATCH-01_A1M_D2B. Wrong: ospf_timer_hello_mismatch_A1M_D2B.
• QC-7: Every test entry has setup, wait, AND teardown blocks. Teardown verification re-checks the same parameter that was changed: teardown.verify_cli = setup.snapshot_cli, teardown.verify_field = setup.snapshot_field, teardown.verify_expected = setup.snapshot_expected. All values come from intent baseline — never guessed. Do NOT create separate test entries for teardown/rollback verification — verification that the rollback succeeded belongs inside the teardown block of the same entry, not as a standalone test.
• QC-8 (directionality): For tests on cross-vendor pairs (different cli_style), generate tests in BOTH directions (misconfigure device A → verify on device B, AND misconfigure device B → verify on device A). This validates that setup/teardown commands are correct for each vendor's CLI. For same-vendor pairs (same cli_style), generate tests in ONE direction ONLY — do NOT generate the mirror direction. The setup/teardown commands are identical on both ends, so mirroring adds zero coverage and doubles test count with no benefit.

---

Step 6 — Present Test Plan

Before generating any output, present a test plan to the user.

⚠️ Always include this warning: "These tests WILL modify device configuration. Rollback is automatic (try/finally in pytest, always: in Ansible) but is NOT guaranteed if the connection drops mid-test. Do not run against production devices without explicit approval."

## Proposed Test Plan

**Protocol**: OSPF
**Feature**: Timer mismatch
**Scope**: A2A ↔ D1C (1 pair)

⚠️  These tests WILL modify device configuration. Rollback is automatic but is NOT
    guaranteed if the connection drops mid-test. Do not run against production devices
    without explicit approval.

| # | Criterion | Setup (target) | Verify (target) | Expected outcome |
|---|-----------|----------------|-----------------|------------------|
| 1 | TMISMATCH-01: Hello mismatch | A2A: set hello=15 | D1C: check state | state != FULL |
| 2 | TMISMATCH-02: Dead mismatch | A2A: set dead=60 | D1C: check state | state != FULL |
...

Proceed?

Wait for user confirmation before generating any files. This is the only mandatory pause in the workflow.

If the user asks to adjust the criteria (add, remove, or modify tests), update the plan and re-present before generating.

---

Step 7 — Load Spec Schema

Read .claude/spec-schema.md — the YAML spec schema (fields, schema rules, ID naming).

Do not proceed to Step 8 without loading it. Do NOT load .claude/spec-renderers.md yet — that's for Steps 9-10.

---

Step 8 — Generate YAML Spec

Apply the confirmed criteria to each device/pair in scope.

Before writing, verify every test entry against QC-1 through QC-8. Check especially:

• setup.ssh_cli and teardown.ssh_cli include the full config-mode path: submode navigation (e.g., interface Ethernet0/1\n) + the command. For interface-level commands on IOS/EOS/AOS-CX, the command alone without interface context will fail at global config level. Never include terminal-level transitions (configure terminal, exit, end).
• Show commands in snapshot_cli and query.ssh_cli are scoped to the specific test target (interface, neighbor, process)
• All peer fields describe the peer device itself, not the setup device
• Test IDs use the criterion ID (e.g., TMISMATCH-01), not a descriptive name

Write to:

• Full topology: output/spec/<protocol>_<feature>.yaml
• Scoped run: output/spec/<protocol>_<feature>_<sorted_devices>.yaml

After writing, verify test count against the plan. If counts differ, investigate before continuing.

---

Step 9 — Load Renderer Guidance

Read .claude/spec-renderers.md — pytest and Ansible renderer patterns.

Do not proceed to rendering without loading it.

---

Step 10 — Render Pytest

Transform the YAML spec into a pytest suite following spec-renderers.md guidance.

• All tests use try/finally with pre-flight check and rollback registry in conftest.py

Output:

• output/pytest/test_<protocol>_<feature>[_<scope>].py
• output/pytest/conftest.py (shared — do not duplicate if it already exists)

---

Step 11 — Render Ansible

Transform the YAML spec into an Ansible playbook following spec-renderers.md guidance.

• All tests use block/always with pre-flight, teardown in always
• Always generate output/ansible/playbook_<protocol>_<feature>_rollback.yml alongside the main playbook

Output:

• output/ansible/playbook_<protocol>_<feature>[_<scope>].yml
• output/ansible/inventory.yml (shared — do not duplicate if it already exists)
• output/ansible/ansible.cfg (shared — do not duplicate if it already exists)

---

Step 12 — Summary

Present a final summary table:

Output	Path	Tests
YAML spec	output/spec/...	N
Pytest suite	output/pytest/...	N
Ansible playbook	output/ansible/...	N
Emergency rollback	output/ansible/..._rollback.yml	Always

Breakdown by criterion.

---

Notes

• All tests modify device configuration. Every test must have a complete teardown block.
• If KB search returns no results for a vendor, note the gap in the spec as a comment and use the closest semantically similar vendor's command with a warning.
• If intent data is ambiguous for a device pair (e.g., timer value not in intent, area type not determinable), ask the user before generating tests for that pair.
• If the user's request doesn't fit the active test model (e.g., exec commands like clear ip ospf process that have no config to roll back, or topology-wide operations with maximum blast radius), explain why it can't be generated as a test and ask the user to try a different query. Do not suggest alternative queries, do not provide examples, and do not use "escalation" language — just explain the problem and stop.