marsai-dev-chaos-testing

name	marsai:dev-chaos-testing
description	Gate 5 of development cycle - ensures chaos tests exist using Toxiproxy to verify graceful degradation under connection loss, latency, and partitions.
trigger	- After integration testing complete (Gate 4) - MANDATORY for all development tasks with external dependencies - Verifies system behavior under failure conditions
skip_when	- Not inside a development cycle (marsai:dev-cycle) - Service has no external dependencies (no database, cache, queue, or external API) - Task is documentation-only, configuration-only, or non-code - Frontend-only project with no backend service dependencies
NOT_skip_when	- "Infrastructure is reliable" - All infrastructure fails eventually. Be prepared. - "Integration tests cover failures" - Integration tests verify happy path. Chaos verifies failures. - "Toxiproxy is complex" - One container, 20 minutes setup. Prevents production incidents.
sequence	{"after":["marsai:dev-integration-testing"],"before":["marsai:requesting-code-review"]}
related	{"complementary":["marsai:dev-cycle","marsai:dev-integration-testing","marsai:qa-analyst"]}
input_schema	{"required":[{"name":"unit_id","type":"string","description":"Task or subtask identifier"},{"name":"external_dependencies","type":"array","items":"string","description":"External services (postgres, redis, rabbitmq, etc.)"},{"name":"language","type":"string","enum":["typescript"],"description":"Programming language"}],"optional":[{"name":"gate6_handoff","type":"object","description":"Full handoff from Gate 4 (integration testing)"}]}
output_schema	{"format":"markdown","required_sections":[{"name":"Chaos Testing Summary","pattern":"^## Chaos Testing Summary","required":true},{"name":"Failure Scenarios","pattern":"^## Failure Scenarios","required":true},{"name":"Handoff to Next Gate","pattern":"^## Handoff to Next Gate","required":true}],"metrics":[{"name":"result","type":"enum","values":["PASS","FAIL"]},{"name":"dependencies_tested","type":"integer"},{"name":"scenarios_tested","type":"integer"},{"name":"recovery_verified","type":"boolean"},{"name":"iterations","type":"integer"}]}
verification	{"automated":[{"command":"find . -name '*.chaos.spec.ts' -o -name '*.chaos.test.ts' | head -5","description":"Chaos test files exist","success_pattern":".chaos."},{"command":"grep -rn 'toxiproxy\\|Toxiproxy' --include='*.ts' .","description":"Toxiproxy setup present","success_pattern":"toxiproxy"}],"manual":["Chaos tests use *.chaos.spec.ts naming","All external dependencies have failure scenarios","Recovery verified after each failure injection"]}

Dev Chaos Testing (Gate 5)

Overview

Ensure code handles failure conditions gracefully by injecting faults using Toxiproxy. Verify connection loss, latency, and network partitions don't cause crashes.

Core principle: All infrastructure fails. Chaos testing ensures your code handles it gracefully.

<block_condition>

• No chaos tests = FAIL
• Any dependency without failure test = FAIL
• Recovery not verified = FAIL
• System crashes on failure = FAIL </block_condition>

CRITICAL: Role Clarification

This skill ORCHESTRATES. QA Analyst Agent (chaos mode) EXECUTES.

Who	Responsibility
This Skill	Gather requirements, dispatch agent, track iterations
QA Analyst Agent	Write chaos tests, setup Toxiproxy, verify recovery

---

Standards Reference

MANDATORY: Load testing-chaos.md standards via WebFetch.

<fetch_required> Load chaos testing standards for the project language. </fetch_required>

---

Step 0: Detect External Dependencies (Auto-Detection)

MANDATORY: When external_dependencies is empty or not provided, scan the codebase to detect them automatically before validation.

if external_dependencies is empty or not provided:

  detected_dependencies = []

  1. Scan docker-compose.yml / docker-compose.yaml for service images:
     - Grep tool: pattern "postgres" in docker-compose* files → add "postgres"
     - Grep tool: pattern "mongo" in docker-compose* files → add "mongodb"
     - Grep tool: pattern "valkey" in docker-compose* files → add "valkey"
     - Grep tool: pattern "redis" in docker-compose* files → add "redis"
     - Grep tool: pattern "rabbitmq" in docker-compose* files → add "rabbitmq"

  2. Scan dependency manifests:
     if language == "typescript":
       - Grep tool: pattern "\"pg\"" in package.json → add "postgres"
       - Grep tool: pattern "@prisma/client" in package.json → add "postgres"
       - Grep tool: pattern "\"mongodb\"" in package.json → add "mongodb"
       - Grep tool: pattern "\"mongoose\"" in package.json → add "mongodb"
       - Grep tool: pattern "\"redis\"" in package.json → add "redis"
       - Grep tool: pattern "\"ioredis\"" in package.json → add "redis"
       - Grep tool: pattern "@valkey" in package.json → add "valkey"
       - Grep tool: pattern "\"amqplib\"" in package.json → add "rabbitmq"
       - Grep tool: pattern "amqp-connection-manager" in package.json → add "rabbitmq"

  3. Deduplicate detected_dependencies
  4. Set external_dependencies = detected_dependencies

  Log: "Auto-detected external dependencies: [detected_dependencies]"

<auto_detect_reason> PM team task files often omit external_dependencies. If the codebase uses postgres, mongodb, valkey, or rabbitmq, these are external dependencies that MUST have chaos tests. Auto-detection prevents silent skips. </auto_detect_reason>

---

Step 0.5: Testing Infrastructure Assessment (MANDATORY)

MANDATORY: Before creating chaos tests from scratch, assess whether the project has existing chaos testing infrastructure. Most projects will have unit test setup — that does NOT count. This step checks specifically for chaos test patterns (Toxiproxy, failure injection, chaos-specific test files).

infrastructure_scan = {
  has_chaos_tests: false,
  has_toxiproxy_setup: false,
  infrastructure_exists: false,
}

1. Scan for existing chaos test files:
   - Glob tool: pattern "**/*.chaos.spec.ts", "**/*.chaos.test.ts"
   - Grep tool: pattern "toxiproxy|Toxiproxy" in **/*.spec.ts, **/*.test.ts
   - If any matches → has_chaos_tests = true
   NOTE: Unit test files or integration tests without fault injection do NOT count.

2. Scan for Toxiproxy or fault-injection infrastructure:
   - Grep tool: pattern "toxiproxy|Toxiproxy" in **/*.ts
   - Grep tool: pattern "enableProxy|addToxic|removeToxic" in **/*.ts
   - Grep tool: pattern "CHAOS" in docker-compose* files
   - If any matches → has_toxiproxy_setup = true

3. Determine infrastructure_exists:
   infrastructure_exists = has_chaos_tests OR has_toxiproxy_setup
   NOTE: Existing unit tests or integration tests = infrastructure does NOT exist for chaos purposes.

Step 0.5b: No Infrastructure → Assess Task Criticality

if infrastructure_exists == true:
  → Proceed to Step 1 (normal flow)

if infrastructure_exists == false:
  → Assess task criticality:

  critical_domains = ["login", "logout", "sign-in", "sign-up", "signup", "signin",
                      "authenticate", "authentication", "oauth", "sso", "mfa", "2fa",
                      "payment", "billing", "transaction", "money",
                      "balance", "transfer", "ledger", "invoice", "charge"]

  NOTE: Only tasks that DIRECTLY implement authentication flows (login, signup,
  OAuth callbacks) or handle money (payments, transfers, billing) qualify.
  Tasks that merely touch permissions, roles, tokens, encryption, or credentials
  as part of broader features are NOT critical for this purpose.

  task_is_critical = any critical_domain keyword appears in:
    - unit_id
    - external_dependencies context
    - acceptance_criteria (if available from orchestrator)
    - implementation file paths (if available from Gate 0)

  ┌─────────────────────────────────────────────────────────────┐
  │ NON-CRITICAL TASK + NO INFRASTRUCTURE → SKIP               │
  │                                                             │
  │ Return:                                                     │
  │   status: SKIP                                              │
  │   skip_reason: "Project lacks chaos testing infrastructure  │
  │     (no existing chaos tests, no Toxiproxy setup). Creating │
  │     chaos tests from scratch without established patterns   │
  │     produces low-quality code."                             │
  │   recommendation: "Set up Toxiproxy base configuration and  │
  │     a reference chaos test before enabling this gate."      │
  │   ready_for_next_gate: YES                                  │
  └─────────────────────────────────────────────────────────────┘

  ┌─────────────────────────────────────────────────────────────┐
  │ CRITICAL TASK + NO INFRASTRUCTURE → EPHEMERAL MODE         │
  │                                                             │
  │ Log: "CRITICAL TASK detected without chaos testing          │
  │   infrastructure. Running ephemeral verification."          │
  │ Set: ephemeral_mode = true                                  │
  │ Proceed to Step 1 (normal flow)                             │
  │                                                             │
  │ After Step 4 (output):                                      │
  │   1. Record test results in gate output                     │
  │   2. DELETE all chaos test files created during this gate   │
  │   3. Log files deleted and why                              │
  │   4. Return results with:                                   │
  │     ephemeral_note: "Chaos tests verified [PASS/FAIL] but  │
  │       files were removed. Project needs chaos testing       │
  │       infrastructure setup before persisting tests."        │
  └─────────────────────────────────────────────────────────────┘

<infrastructure_rationale> Creating chaos tests in a project without established patterns produces tests that:

• Don't follow project conventions (wrong directory structure, naming, imports)
• Lack shared test utilities (Toxiproxy helpers, container setup/teardown)
• Add dependencies the project isn't prepared to maintain (toxiproxy, testcontainers)
• Create maintenance burden with no existing patterns to guide future developers

For critical tasks (auth flows, payments), ephemeral verification validates graceful degradation without committing unmaintainable infrastructure. </infrastructure_rationale>

---

Step 1: Validate Input

REQUIRED INPUT:
- unit_id: [task/subtask being tested]
- external_dependencies: [postgres, mongodb, valkey, redis, rabbitmq, etc.] (from input OR auto-detected in Step 0)
- language: [typescript]

OPTIONAL INPUT:
- gate4_handoff: [full Gate 4 output]

if any REQUIRED input is missing:
  → STOP and report: "Missing required input: [field]"

if external_dependencies is empty (AFTER auto-detection in Step 0):
  → STOP and report: "No external dependencies found after codebase scan - chaos testing requires dependencies"

Step 2: Dispatch QA Analyst Agent (Chaos Mode)

Task tool:
  subagent_type: "marsai:qa-analyst"
  prompt: |
    **MODE:** CHAOS TESTING (Gate 5)

    **Standards:** Load testing-chaos.md

    **Input:**
    - Unit ID: {unit_id}
    - External Dependencies: {external_dependencies}
    - Language: {language}

    **Requirements:**
    1. Setup Toxiproxy infrastructure in test utilities
    2. Create chaos tests (*.chaos.spec.ts naming)
    3. Use environment variable gating (CHAOS=1)
    4. Test failure scenarios: Connection Loss, High Latency, Network Partition
    5. Verify 5-phase structure: Normal → Inject → Verify → Restore → Recovery

    **Output Sections Required:**
    - ## Chaos Testing Summary
    - ## Failure Scenarios
    - ## Handoff to Next Gate

Step 3: Evaluate Results

Parse agent output:

if "Status: PASS" in output:
  → Gate 5 PASSED
  → Return success with metrics

if "Status: FAIL" in output:
  → Dispatch fix to implementation agent
  → Re-run chaos tests (max 3 iterations)
  → If still failing: ESCALATE to user

Step 4: Generate Output

## Chaos Testing Summary
**Status:** {PASS|FAIL}
**Dependencies Tested:** {count}
**Scenarios Tested:** {count}
**Recovery Verified:** {Yes|No}

## Failure Scenarios
| Component | Scenario | Status | Recovery |
|-----------|----------|--------|----------|
| {component} | {scenario} | {PASS|FAIL} | {Yes|No} |

## Handoff to Next Gate
- Ready for Gate 8 (Code Review): {YES|NO}
- Iterations: {count}

---

Failure Scenarios by Dependency

Dependency	Required Scenarios
PostgreSQL	Connection Loss, High Latency, Network Partition
MongoDB	Connection Loss, High Latency, Network Partition
Valkey	Connection Loss, High Latency, Timeout
Redis	Connection Loss, High Latency, Timeout
RabbitMQ	Connection Loss, Network Partition, Slow Consumer
HTTP APIs	Timeout, 5xx Errors, Connection Refused

---

Severity Calibration

Severity	Criteria	Examples
CRITICAL	System crashes on failure, data loss	Panic on connection loss, corrupted state on partition
HIGH	No recovery, missing dependency tests	System doesn't recover after failure, untested dependency
MEDIUM	Partial recovery, missing scenarios	Recovery takes too long, missing latency test
LOW	Cleanup issues, documentation	Test artifacts not cleaned, missing chaos docs

Report all severities. CRITICAL = immediate fix (production risk). HIGH = fix before gate pass. MEDIUM = fix in iteration. LOW = document.

---

Anti-Rationalization Table

Rationalization	Why It's WRONG	Required Action
"Infrastructure is reliable"	AWS, GCP, Azure all have outages. Your code must handle them.	Write chaos tests
"Integration tests cover failures"	Integration tests verify happy path. Chaos tests verify failure handling.	Write chaos tests
"Toxiproxy is complex"	One container. 20 minutes setup. Prevents production incidents.	Write chaos tests
"We have monitoring"	Monitoring detects problems. Chaos testing prevents them.	Write chaos tests
"Circuit breakers handle it"	Circuit breakers need testing too. Chaos tests verify they work.	Write chaos tests

---