systematic-testing

name	systematic-testing
description	Apply bottom-up systematic testing to a multi-layer system. Use when asked to test, debug, or verify a pipeline that spans multiple layers (e.g., Python worker → bridge → processor → API → UI).

Systematic Testing Method

When testing or debugging a multi-layer system, follow this method strictly.

Step 1: Map the Layers

Identify every distinct layer in the pipeline from data source to UI. Name each one. Draw the dependency graph showing which layers call which. Determine the bottom (fewest internal dependencies) and top (user-facing).

Step 2: Test Bottom-Up

Start at the lowest layer. Move up one layer at a time. Never test a higher layer until the layer below it is verified.

Step 3: At Each Layer, Write Three Kinds of Tests

Unit tests — Does this layer's logic produce correct output for known input? Mock everything external. Test edge cases: empty input, zero values, single item, max capacity, error states.

Contract tests — Does the output shape of this layer exactly match what the next layer expects as input? Assert every field name, type, and nesting level. This catches schema drift between producer and consumer.

Error tests — Does this layer handle failures from the layer below? Simulate timeouts, malformed responses, missing fields, empty results.

Step 4: At Each Boundary, Verify the Handoff

When Layer A calls Layer B, assert:

• A sends the exact request shape B expects
• B returns the exact response shape A parses
• Side effects (DB writes, progress events, callbacks) fire in the expected order with the expected data

Step 5: Fix Before Ascending

When a test reveals a mismatch, fix it immediately. Do not move to the next layer with a known broken contract below. Bugs compound across layers.

Step 6: Integration Last

Only after all layers pass individually, run cross-layer smoke tests with realistic fixture data end-to-end. These confirm the full chain works. If an integration test fails, go back to the specific layer's unit/contract tests to debug — don't debug in the integration test.

Common Boundary Failures to Watch For

• Schema drift: field renamed in one layer but not the other
• Shape mismatch: object vs array, nested vs flat, optional vs required
• Serialization bugs: double-encoded JSON, dates as strings vs Date objects
• Progress/event shape: emitter sends a shape the consumer doesn't parse
• Config passthrough: config transformed or lost as it crosses layers
• Ordering: operations happen in wrong sequence (e.g., close positions before vs after generating signals)
• Timezone/date parsing: midnight vs noon, UTC vs local, string format differences

Execution Notes

• Create a plan document listing all layers and phases before writing any tests
• Layers with no dependencies on each other can be tested in parallel
• Each phase should have explicit exit criteria before moving to the next
• Prefer fixture/canned data over live API calls in tests