testing-methodology

name	testing-methodology
description	Testing methodology — test harness design, mock strategies, flakiness prevention, and structured test planning

Testing Methodology

Phase 1: Test Infrastructure Discovery

Before writing any tests, understand what exists:

Test Framework Detection

• What test frameworks are installed? (jest, pytest, go test, etc.)
• How are tests run? (npm test, make test, go test ./...)
• Where do tests live? (__tests__/, *_test.go, tests/)
• What's the naming convention? (*.test.ts, *_test.py, Test*.java)

Existing Patterns

• What test utilities/helpers exist?
• Are there existing fixtures or factories?
• What mocking libraries are used?
• How is test data managed?
• Are there existing harnesses to build on?

Test Execution Environment

• How is CI configured for tests?
• What's the coverage tool and current coverage?
• Are there test databases or containers?
• What environment variables do tests need?

Document findings - this information helps you and future agents write consistent tests.

Phase 2: Test Harness Design

The key to testing complex systems is good harnesses. A harness handles setup, teardown, and provides utilities so individual tests stay focused.

What to Mock vs What to Keep Real

Keep REAL when:

• It's the thing you're actually testing
• It's fast and deterministic (pure functions, in-memory DBs)
• Mocking it would miss real bugs (ORM behavior, query logic)
• The real thing is simpler than a mock

MOCK when:

• External services (APIs, third-party services)
• Non-deterministic sources (time, random, network)
• Slow resources that aren't under test
• Components with complex setup that aren't relevant to the test

The boundary rule: Mock at system boundaries, keep internals real.

Harness Patterns

Integration Server Harness

For API/backend testing:

TestHarness
├── Server (real, started once per suite)
├── Database (isolated per test, migrations run)
├── External Service Mocks (controllable)
└── Test Client (authenticated, with helpers)

Database Harness

For data layer testing:

TestHarness
├── Database Connection (transaction per test, rolled back)
├── Migration Runner (schema always current)
├── Fixture Loader (predictable test data)
└── Query Assertion Helpers

Async/Queue Harness

For message processing:

TestHarness
├── Queue (in-memory or real, isolated)
├── Worker Pool (controllable concurrency)
├── Event Collector (capture for assertions)
└── Completion Waiter (with timeout)

External Service Harness

For integration points:

TestHarness
├── Mock Server (httptest, wiremock, etc.)
├── Request Recorder (verify calls made)
├── Response Programmer (control responses)
└── Failure Injector (test error handling)

Phase 3: Writing Good Tests

Test Structure

Every test should have:

1. Arrange: Set up preconditions (use harness)
2. Act: Execute the thing being tested
3. Assert: Verify the outcome
4. Cleanup: Handled by harness (teardown)

Test Naming

Names should describe the scenario and expected outcome:

• test_create_user_with_valid_email_succeeds
• TestPaymentProcessor_RefundExceedingBalance_ReturnsError
• it('should reject duplicate usernames', ...)

Avoiding Flakiness

Timing issues:

• Never use fixed sleeps (time.Sleep(1s))
• Poll with timeout for async operations
• Use event-driven waiting when possible

// Bad
time.Sleep(2 * time.Second)
assert(result.IsComplete())

// Good
waitFor(func() bool { return result.IsComplete() }, timeout: 5s, poll: 100ms)

State isolation:

• Each test gets fresh state (new DB transaction, clean mocks)
• Don't rely on test execution order
• Clean up resources in teardown, not in test body

Live / heavy tests are opt-in:

• A test that hits a live external dependency (real sandbox, deployed environment, real credential) must be gated behind an explicit opt-in env var (e.g. RUN_LIVE_TESTS=1) and skip-with-a-reason when it's unset.
• The default suite stays hermetic and green in CI; the live test runs only when a human or a dedicated job asks for it. A live test that runs by default is a flake and a credential-leak risk.

Non-determinism:

• Mock time.Now() for time-dependent logic
• Seed random generators deterministically
• Control UUIDs/IDs in tests

Test Categories

Unit Tests

• Test single functions/methods in isolation
• Mock all dependencies
• Fast (<10ms each)
• High coverage of edge cases

Integration Tests

• Test components working together
• Real database, real internal services
• Mock external boundaries
• Medium speed (<1s each)

E2E Tests

• Test complete user flows
• Minimal mocking (only external services)
• Slower but high confidence
• Focus on critical paths

Contract Tests

• Verify API contracts between services
• Both provider and consumer tests
• Catch breaking changes early

Concurrency Tests

• Test race conditions explicitly
• Use race detectors (go test -race)
• Stress test with goroutines/threads
• Test deadlock scenarios

Phase 4: Test Tooling

Create tools that make testing easier. Write to .reliant/tools/ or the project's scripts directory.

Useful Test Scripts

Run specific test suites:

#!/bin/bash
# .reliant/tools/test-unit.sh
go test -short ./...

Run with coverage:

#!/bin/bash
# .reliant/tools/test-coverage.sh
go test -coverprofile=coverage.out ./...
go tool cover -html=coverage.out -o coverage.html
echo "Coverage report: coverage.html"

Seed test database:

#!/bin/bash
# .reliant/tools/seed-test-db.sh
DB_PATH="${TEST_DB:-./test.db}"
sqlite3 "$DB_PATH" < ./testdata/seed.sql
echo "Test database seeded at $DB_PATH"

Run flaky test detector:

#!/bin/bash
# .reliant/tools/flaky-detector.sh
TEST_NAME="$1"
RUNS="${2:-10}"
for i in $(seq 1 $RUNS); do
  if ! go test -run "$TEST_NAME" ./...; then
    echo "FLAKY: Failed on run $i"
    exit 1
  fi
done
echo "STABLE: Passed $RUNS consecutive runs"

Phase 5: Test Plan Output

When reviewing code or planning tests, provide a structured test plan:

Coverage Assessment

• What's currently tested
• What's missing coverage
• Risk areas without tests

Unit Tests to Add

Function/Method	Test Cases	Priority
validateEmail	valid format, invalid format, empty, unicode	High
calculateTotal	normal, with discount, zero items, overflow	High

Integration Tests to Add

Scenario	Components	Mocking Strategy
User registration flow	API + DB + Email	Mock email service
Payment processing	API + DB + Stripe	Mock Stripe API

E2E Tests to Add

User Flow	Steps	Assertions
Complete checkout	Login -> Add item -> Pay -> Confirm	Order created, email sent

Concurrency Tests

• Race conditions to test
• Deadlock scenarios to verify
• Load/stress test considerations

What to Mock vs Keep Real

Component	Decision	Rationale
Database	Real (postgres)	Need to test actual queries
Redis	Mock	Not testing caching logic
Stripe API	Mock	External, non-deterministic

Guidelines

DO:

• Follow existing test patterns in the codebase
• Create reusable harnesses and helpers
• Test edge cases and error paths
• Make test failures self-explanatory
• Run tests before and after changes

DON'T:

• Write tests that pass when code is broken
• Create flaky tests (fix or delete them)
• Over-mock (test realistic scenarios)
• Ignore existing test utilities
• Write tests without understanding the code

Test Quality Checklist

• Tests actually test the right thing (not just coverage theater)
• Tests fail when code is broken
• Tests are deterministic (no flakiness)
• Tests are fast enough for the feedback loop
• Tests have clear failure messages
• Test code is maintainable (DRY with helpers/fixtures)