test-strategy

name	test-strategy
description	Design a comprehensive testing pyramid strategy covering unit, integration, contract, E2E, and performance tests. Outputs test coverage targets, tooling selection, CI configuration, and quality gates.
argument-hint	["application type","team size","tech stack","deployment frequency","current test coverage"]
allowed-tools	Read, Write, Bash

Test Strategy

A test strategy answers four questions: what to test, how much to test, which tools to use, and what must pass before code ships. Without a strategy, teams either test too little (production bugs) or too much at the wrong layer (slow, brittle test suites).

Process

1. Inventory the system — services, integrations, user journeys, risk areas.
2. Define the testing pyramid — proportions of unit/integration/E2E tests.
3. Set coverage targets — by layer, not a single aggregate number.
4. Choose tooling — one test framework per layer per language.
5. Define quality gates — what must pass at each stage of CI.
6. Plan test data — factories, fixtures, test containers.
7. Establish flakiness policy — quarantine, fix within SLA, or delete.
8. Review test metrics — coverage trends, test duration, failure rates.

Output Format

Testing Pyramid

                    ┌────────────────────┐
                    │    E2E / UI Tests   │  5-10%
                    │   (Playwright/      │  Slow, brittle, expensive
                    │    Cypress)         │  Test critical user journeys only
                    ├────────────────────┤
                  ┌─┤ Contract Tests     │  5%
                  │ │ (Pact / Dredd)     │  Verify API contracts
                  └─┼────────────────────┤
               ┌────┤ Integration Tests  │  20-30%
               │    │ (pytest, Jest)     │  DB, cache, external services
               │    │ Real dependencies  │  
               └────┼────────────────────┤
          ┌─────────┤  Unit Tests        │  60-70%
          │         │  (pytest, Jest,    │  Fast, deterministic
          │         │   JUnit)           │  No I/O, pure logic
          └─────────┴────────────────────┘

Coverage Targets by Layer

Layer	Coverage Target	Measurement	Rationale
Unit	80% line coverage	Per-file, tracked in CI	High leverage, fast to run
Integration	Critical paths	Endpoint + DB combos	Validates real behavior
Contract	100% of APIs consumed	Per consumer-provider pair	Prevents breaking changes
E2E	Top 5 user journeys	Explicit scenario list	Too slow/expensive to go broader
Performance	Key endpoints	Latency + RPS at SLO	Catch regressions before prod

Tooling Selection

# test-strategy.yaml — tooling decisions with rationale
tech_stack: python-fastapi-postgres

unit_testing:
  framework: pytest
  runner: pytest-xdist       # Parallel execution
  coverage: coverage.py + pytest-cov
  mocking: unittest.mock + pytest-mock
  factories: factory_boy     # Test data generation
  assertions: built-in + pytest-approx

integration_testing:
  framework: pytest
  database: pytest-postgresql + testcontainers
  http_client: httpx + respx  # Real HTTP calls + mock external APIs
  fixtures: conftest.py with session-scoped DB
  test_data: Alembic migrations + seed scripts

contract_testing:
  framework: pact-python
  broker: PactFlow (managed) or   
   
  

   
   []
    
   


   
     
      
  

         
    
     
   

Project Structure

tests/
├── unit/                    # Pure logic, no I/O
│   ├── test_pricing.py
│   ├── test_validators.py
│   └── test_order_logic.py
├── integration/             # Real DB, real cache, mock external
│   ├── conftest.py          # Fixtures: DB, app client
│   ├── test_orders_api.py
│   ├── test_user_repo.py
│   └── test_payment_flow.py
├── contract/                # Pact consumer/provider tests
│   ├── consumer/
│   └── provider/
├── e2e/                     # Full user journeys in staging
│   ├── test_checkout_flow.py
│   └── test_search_flow.py
├── performance/             # k6 scripts
│   └── load_test.js
├── conftest.py              # Root fixtures
└── pytest.ini

pytest Configuration

# pytest.ini
[pytest]
testpaths = tests
python_files = test_*.py
python_classes = Test*
python_functions = test_*

addopts =
    --strict-markers
    --strict-config
    -v
    --tb=short
    --cov=src
    --cov-report=term-missing
    --cov-report=html:coverage-report
    --cov-fail-under=80

markers =
    unit: Pure unit tests (no I/O)
    integration: Tests requiring database or external services
    e2e: End-to-end tests requiring full environment
    slow: Tests taking >1 second
    contract: Pact contract tests

Test Fixtures (conftest.py)

# tests/conftest.py
import pytest
import pytest_asyncio
from httpx import AsyncClient
from sqlalchemy.ext.asyncio import create_async_engine, AsyncSession
from testcontainers.postgres import PostgresContainer
from factory import Factory, Faker, SubFactory

from src.app import create_app
from src.database import Base, get_db
from src.models import User, Order

# ── Database ───────────────────────────────────────────────

@pytest.fixture(scope="session")
def postgres_container():
    """Single PostgreSQL container for the entire test session."""
    with PostgresContainer("postgres:15") as pg:
        yield pg

@pytest.fixture(scope="session")
def db_engine(postgres_container):
    engine = create_async_engine(postgres_container.get_connection_url())
    yield engine
    engine.dispose()

@pytest_asyncio.fixture(scope="function")
async def db_session(db_engine):
    """Each test gets a clean transaction that's rolled back after."""
    async with db_engine.connect() as conn:
         conn.run_sync(Base.metadata.create_all)
          conn.begin_nested()  savepoint:
            session = AsyncSession(bind=conn)
             session
             session.close()
         savepoint.rollback()  




  ():
    
    app = create_app()
    app.dependency_overrides[get_db] = : db_session
    
      AsyncClient(app=app, base_url=)  client:
         client



 ():
     :
        model = User
    
     = Faker()
    email = Faker()
    name = Faker()
    is_active = 
    created_at = Faker()

 ():
     :
        model = Order
    
     = Faker()
    user = SubFactory(UserFactory)
    status = 
    total_cents = Faker(, =, =)




 ():
    
    user = UserFactory.create()
    token = generate_test_token(user.)
     {: }

Example Test Patterns

# tests/unit/test_order_logic.py — pure unit tests
import pytest
from decimal import Decimal
from src.domain.order import calculate_order_total, apply_discount

class TestCalculateOrderTotal:
    def test_sums_items(self):
        items = [
            {"price_cents": 1000, "quantity": 2},
            {"price_cents": 500, "quantity": 1},
        ]
        assert calculate_order_total(items) == 2500
    
    def test_empty_order_is_zero(self):
        assert calculate_order_total([]) == 0
    
    def test_rejects_negative_quantity(self):
        with pytest.raises(ValueError, match="quantity must be positive"):
            calculate_order_total([{"price_cents": 100, "quantity": -1}])
    
    @pytest.mark.parametrize("discount_pct,expected", [
        (10, 900),
        (50, 500),
        (100, 0),
        (0, ),
    ])
     ():
         apply_discount(, discount_pct) == expected



 pytest
 httpx  AsyncClient



 :
      ():
        response =  client.post(
            ,
            json={
                : [{: , : }]
            },
            headers=auth_headers
        )
         response.status_code == 
        data = response.json()
           data
         data[] == 
    
      ():
        response =  client.post(
            ,
            json={: [{: , : }]},
            headers=auth_headers
        )
        order_id = response.json()[]
        
        
        order =  db_session.get(Order, order_id)
         order   
         order.status == 
    
      ():
        response =  client.post(, json={})
         response.status_code == 
    
      ():
        response =  client.post(
            ,
            json={: []},  
            headers=auth_headers
        )
         response.status_code == 
           response.json()[][][]

CI Quality Gates

# .github/workflows/test.yml
name: Test Suite

on: [push, pull_request]

jobs:
  unit:
    name: Unit Tests
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run unit tests
        run: pytest tests/unit/ -m "not integration" --cov-fail-under=80
      - name: Upload coverage
        uses: codecov/codecov-action@v4

  integration:
    name: Integration Tests
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:15
        env:
          POSTGRES_PASSWORD: test
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
    steps:
      - uses: actions/checkout@v4
          
            
        
           

  
      
     
     [, ]
    
          
               
             
         

Flakiness Policy

# Flaky Test Policy

## Definition
A test is **flaky** if it fails intermittently without code changes.

## Detection
- Any test that fails and passes on re-run without code changes is flagged
- CI tracks failure-to-pass ratio: >2% = flagged as flaky

## SLAs
| Category | Action Required |
|----------|----------------|
| P0: Blocking CI | Fix within 24h or quarantine |
| P1: Fails >10% | Fix within 1 week |
| P2: Fails 2-10% | Fix within 2 weeks |
| Investigate: Fails <2% | Log and monitor |

## Quarantine Process
1. Add `@pytest.mark.flaky(reruns=3)` with a GitHub issue link
2. Add to weekly flaky test review meeting
3. Fix or delete within the SLA

## Never Acceptable
- Deleting a failing test without understanding why it fails
- Increasing retry count without addressing root cause
- Using `time.sleep()` to fix timing issues (use explicit waits)

Rules

• Test pyramid, not test trophy — most tests should be unit tests; E2E is expensive, keep it minimal.
• One test framework per layer — don't mix pytest and unittest; pick one and standardize.
• Tests must run in any order — no implicit state between tests; use fixtures with teardown.
• Fast tests run first — order CI jobs: unit → integration → E2E. Don't run E2E on every PR.
• Coverage is a proxy, not a goal — 80% coverage with weak assertions is worse than 60% with strong ones.
• Delete tests that don't fail for 6 months — dead test code is maintenance burden.
• Test the behavior, not the implementation — test what, not how. Refactoring shouldn't break tests.
• Every bug gets a regression test — before fixing a bug, write a test that reproduces it.
• No production data in tests — use factories and synthetic data; never clone prod to test.
• Quarantine, don't disable — a disabled test is a lie about test coverage.