testing-strategy

# Testing Strategy Skill ## Core Principle **Confidence through layered testing.** Tests are not just about catching bugs - they enable confident refactoring, document expected behavior, and provide fast feedback during development. A good testing strategy balances thoroughness with maintainability, using the right test type for each validation need. **Effective testing:** - Catches bugs before production - Enables safe refactoring - Documents system behavior - Provides fast feedback loops - Scales with codebase growth --- ## Testing Pyramid **The testing pyramid guides test distribution across layers:** ``` /\ / \ E2E (10%) /----\ - Full system tests / \ - Slow, brittle, expensive /--------\ - Critical user journeys only / \ /------------\ Integration (20%) / \ - Component interactions /----------------\- Database, API, services \----------------/- Medium speed, moderate cost \--------------/ \------------/ Unit (70%) \----------/ - Individual functions \--------/ - Fast, reliable, cheap \------/ - Maximum coverage here \----/ \__/ ``` ### Unit Tests (70% of tests) **Purpose:** Validate individual functions/methods in isolation **Characteristics:** - Fast (milliseconds per test) - No external dependencies (databases, APIs, file system) - Isolated (each test independent) - Deterministic (same input = same output) **What to test:** - Pure functions (input → output) - Business logic - Edge cases and boundary conditions - Error handling - Data transformations **Example (Python):** ```python def calculate_discount(price: float, discount_percent: float) -> float: """Calculate discounted price.""" if price < 0: raise ValueError("Price cannot be negative") if not 0 <= discount_percent <= 100: raise ValueError("Discount must be between 0 and 100") return round(price * (1 - discount_percent / 100), 2) # Unit tests def test_calculate_discount_normal_case(): assert calculate_discount(100.0, 10.0) == 90.0 def test_calculate_discount_zero_discount(): assert calculate_discount(100.0, 0.0) == 100.0 def test_calculate_discount_full_discount(): assert calculate_discount(100.0, 100.0) == 0.0 def test_calculate_discount_rounding(): assert calculate_discount(99.99, 10.0) == 89.99 def test_calculate_discount_negative_price(): with pytest.raises(ValueError, match="Price cannot be negative"): calculate_discount(-10.0, 10.0) def test_calculate_discount_invalid_percent(): with pytest.raises(ValueError, match="Discount must be between"): calculate_discount(100.0, 150.0) ``` ### Integration Tests (20% of tests) **Purpose:** Validate interactions between components **Characteristics:** - Medium speed (seconds per test) - Real dependencies (test database, external services) - Test realistic scenarios - More complex setup/teardown **What to test:** - Database queries and transactions - API endpoint contracts - Service-to-service communication - Cache interactions - File system operations **Example (JavaScript):** ```javascript describe('User API', () => { let db; beforeEach(async () => { // Setup test database db = await createTestDatabase(); await db.migrate(); }); afterEach(async () => { await db.cleanup(); }); it('creates user and stores in database', async () => { const userData = { name: 'Jane Doe', email: 'jane@example.com', role: 'user' }; const response = await request(app) .post('/api/users') .send(userData) .expect(201); // Verify API response expect(response.body).toMatchObject({ id: expect.any(String), name: 'Jane Doe', email: 'jane@example.com', role: 'user' }); // Verify database state const user = await db.users.findById(response.body.id); expect(user.email).toBe('jane@example.com'); }); it('rejects duplicate email addresses', async () => { await db.users.create({ name: 'John Doe', email: 'jane@example.com' }); const response = await request(app) .post('/api/users') .send({ name: 'Jane Doe', email: 'jane@example.com' }) .expect(409); expect(response.body.error).toMatch(/email already exists/i); }); }); ``` ### End-to-End Tests (10% of tests) **Purpose:** Validate complete user workflows through the system **Characteristics:** - Slow (minutes per test) - Full system (UI, backend, database, external services) - Brittle (breaks with UI changes) - Expensive to maintain **What to test:** - Critical user journeys (signup, checkout, core workflows) - Cross-cutting features (authentication, authorization) - Integration with third-party services - Browser compatibility (if web app) **Example (Playwright):** ```javascript test('user can complete checkout flow', async ({ page }) => { // Login await page.goto('/login'); await page.fill('input[name="email"]', 'test@example.com'); await page.fill('input[name="password"]', 'password123'); await page.click('button[type="submit"]'); // Add item to cart await page.goto('/products'); await page.click('text=Product Name'); await page.click('button:has-text("Add to Cart")'); // Proceed to checkout await page.click('text=Cart'); await page.click('text=Checkout'); // Fill shipping info await page.fill('input[name="address"]', '123 Main St'); await page.fill('input[name="city"]', 'San Francisco'); await page.fill('input[name="zip"]', '94102'); // Fill payment info (test mode) await page.fill('input[name="cardNumber"]', '4242424242424242'); await page.fill('input[name="expiry"]', '12/25'); await page.fill('input[name="cvc"]', '123'); // Submit order await page.click('button:has-text("Place Order")'); // Verify confirmation await expect(page.locator('text=Order Confirmed')).toBeVisible(); await expect(page.locator('text=Order #')).toBeVisible(); }); ``` --- ## Test-Driven Development (TDD) **Red-Green-Refactor workflow:** ### 1. Red (Write Failing Test) Write the test first, before implementation. Test should fail because functionality doesn't exist yet. ```python # test_calculator.py def test_add_two_numbers(): calculator = Calculator() result = calculator.add(2, 3) assert result == 5 # FAILS - Calculator doesn't exist yet ``` ### 2. Green (Make Test Pass) Write minimal code to make the test pass. Don't worry about perfection yet. ```python # calculator.py class Calculator: def add(self, a, b): return a + b # PASSES - Simplest implementation ``` ### 3. Refactor (Improve Code) Clean up code while keeping tests passing. Tests give you confidence to refactor safely. ```python # calculator.py (refactored) class Calculator: """Simple calculator for arithmetic operations.""" def add(self, a: float, b: float) -> float: """Add two numbers and return the result.""" if not isinstance(a, (int, float)) or not isinstance(b, (int, float)): raise TypeError("Arguments must be numbers") return a + b ``` ### TDD Benefits - **Design first:** Writing tests forces you to think about API design - **Documentation:** Tests document expected behavior - **Confidence:** Tests enable safe refactoring - **Coverage:** TDD naturally produces high test coverage - **Bug prevention:** Catch bugs before they're written ### When to Use TDD **Good fit:** - Complex business logic - Bug fixes (write failing test, then fix) - API design (tests define the interface) - Critical functionality (payment processing, security) **Poor fit:** - Spike/exploratory work (unknown requirements) - Trivial code (getters/setters) - UI experimentation (rapid iteration) - Throwaway prototypes --- ## Test Structure (AAA Pattern) **Arrange-Act-Assert pattern for clear, maintainable tests:** ### Arrange (Setup) Prepare test data and system state ### Act (Execute) Run the code being tested ### Assert (Verify) Check the results match expectations **Example:** ```python def test_user_checkout_with_discount(): # Arrange user = User(id=1, membership="premium") cart = ShoppingCart() cart.add_item(Product(id=101, price=100.0)) cart.add_item(Product(id=102, price=50.0)) checkout = CheckoutService(discount_calculator=PremiumDiscount()) # Act total = checkout.calculate_total(user, cart) # Assert assert total == 135.0 # 150 - 10% premium discount ``` ### Test Naming Convention **Format:** `test_[unit]_[scenario]_[expected_result]` **Good names:** ```python test_calculate_discount_with_zero_percent_returns_original_price() test_user_login_with_invalid_password_raises_authentication_error() test_get_user_by_id_when_not_found_returns_none() ``` **Bad names:** ```python test_discount() # What about discount? test_case_1() # What is case 1? test_user() # What about user? ``` --- ## Coverage Targets **Coverage is a metric, not a goal. 100% coverage doesn't mean bug-free code.** ### Recommended Coverage by Component Type | Component Type | Target Coverage | Rationale | |----------------|----------------|-----------| | **Business Logic** | 90-100% | Critical functionality, high bug cost | | **API Endpoints** | 80-90% | Public contracts, integration points | | **Data Models** | 70-80% | Validation and constraints | | **Utilities** | 80-90% | Reused across codebase | | **UI Components** | 60-70% | High churn, manual testing viable | | **Configuration** | 50-60% | Simple, low complexity | ### Coverage Metrics **Line coverage:** Percentage of code lines executed by tests - Easy to measure, but can be gamed - 80% line coverage is reasonable target **Branch coverage:** Percentage of code paths executed - Better than line coverage (catches untested conditions) - 70% branch coverage is good target **Mutation coverage:** Tests that fail when code is mutated - Most rigorous, catches weak assertions - Advanced metric, use for critical code ### What NOT to Test **Avoid testing:** - Third-party library internals (trust the library) - Framework code (already tested) - Trivial getters/setters (no logic to test) - Private implementation details (test public interface) - Generated code (already validated by generator) --- ## Mocking and Fixtures ### When to Mock **Mock external dependencies:** - Database connections - API calls to external services - File system operations - Current time (for time-dependent logic) - Random number generation **Don't mock:** - Your own domain logic (test it directly) - Simple data structures - Pure functions (no side effects to mock)