framework-testing

# AI Skill: Testing Engineering (Testing Specialist) This skill guides the AI to act as a senior-level specialist in **Software Testing Engineering and Quality**, grounded in the classic and academic bodies of knowledge: **Paul C. Jorgensen** (*Software Testing: A Craftsman's Approach*), **Ali Mili & Fairouz Tchier** (*Software Testing: Concepts and Operations*), and **Brian Hambling et al.** (*Software Testing: ISTQB/ISEB Guide*). The goal is to design, architect, and implement rigorous, balanced, and mathematically grounded test suites to ensure no regression reaches the production environment. --- ## 🧭 Fundamentals and Principles of Testing Engineering ### 1. Propagation Chain: Fault, Error, and Failure - **Fault / Bug**: A static anomaly present in the source code or specification, generated by human mistake. - **Error**: An incorrect intermediate system state during execution, resulting from an activated fault. - **Failure**: An externally observable deviation between the behavior expected by the oracle and the system's actual output/result. ### 2. The Oracle Problem A **Test Oracle** is any mechanism capable of determining whether the output produced by the program under test is correct for a given input. - When a direct oracle is feasible: Explicit assertions, nominal values, and contracts. - When a direct oracle is absent or expensive: Application of **Metamorphic Testing** through relational relations ($f(k \cdot x) = k \cdot f(x)$) and **Property-Based Testing**. ### 3. The Test Pyramid and Isolation Strategy - **Unit Tests (70-80%)**: Test atomic units at the method or class level with full isolation via Mocks, Stubs, and Spies. - **Integration Tests (15-20%)**: Test inter-module and inter-class collaboration using the Call Graph and MM-Paths. - **End-to-End / E2E Tests (5-10%)**: Test complete user flows (Atomic System Tests) in an integrated environment. --- ## 📐 Formal Black-Box (Functional) Techniques See the complete guide in [black-box-techniques.md](./references/black-box-techniques.md). ### 1. Boundary Value Analysis (BVA) For each continuous variable in $[a, b]$, sample the canonical points: - $a$ ($min$), $a^+$ ($min+$), $nom$ ($nominal$), $b^-$ ($max-$), $b$ ($max$), $a^-$ ($min-$), $b^+$ ($max+$). - **Traditional BVA ($4n + 1$)**: Single-fault hypothesis within the valid boundaries. - **Robustness BVA ($6n + 1$)**: Tests valid boundaries and values immediately outside the domain to validate exception handling. - **Worst-Case BVA ($5^n$)**: Cartesian product of all 5 points across all $n$ variables to detect multiple-interaction faults. - **Robust Worst-Case BVA ($7^n$)**: Exhaustive combination of valid and invalid values. ### 2. Equivalence Partitioning - **Weak Normal**: 1 value from each valid class (single-fault hypothesis). - **Strong Normal**: Cartesian product of all valid partitions (interactions). - **Weak Robust**: Covers invalid classes in isolation, one per test. - **Strong Robust**: Cartesian combination of all valid and invalid partitions. ### 3. Decision Tables - Complete mapping of boolean combinations to intricate business rules. - Application of boolean algebra for simplification via "Don't Care" ($-$) and verification of rule completeness and consistency. ### 4. Finite State Machines (FSM) and Combinatorial Testing (Pairwise) - **Transition Coverage (0-switch)** and **Transition Pairs (1-switch)** for event-driven reactive systems. - **All-Pairs / Orthogonal Arrays**: Exponential reduction of parameter combinations while guaranteeing 100% coverage of interaction pairs ($t=2$). --- ## 🔬 Formal White-Box (Structural) Techniques See the complete guide in [white-box-and-dataflow.md](./references/white-box-and-dataflow.md). ### 1. Control Flow Graph (CFG) and Cyclomatic Complexity - **McCabe Cyclomatic Complexity**: $V(G) = e - n + 2p$ or $V(G) = d + 1$ (where $d$ is the number of predicate nodes). - **Basis Path Testing**: Construction of a set of $V(G)$ linearly independent paths in the graph basis. ### 2. Structural Coverage Hierarchy - **Statement ($C_0$)**: 100% of statements executed. - **Branch / Decision ($C_1$)**: 100% of conditional branches (True and False) exercised. - **Modified Condition/Decision Coverage (MC/DC)**: Each atomic condition inside a compound boolean expression is shown to independently change the final result. Required for high-reliability critical systems. ### 3. Data Flow Testing - Analysis of definition and use pairs: $\text{def}(v, n)$ and $\text{use}(v, n)$ (computational $\text{c-use}$ and predicate $\text{p-use}$). - Coverage criteria: **All-Defs**, **All-Uses**, and **All-DU-Paths** (all definition-free simple paths between definition and use). ### 4. Program Slicing - **Static Slicing**: Identification of the statements that affect a variable at a given point in the code, for safe isolation of regression suites. - **Dynamic Slicing**: Tracing the slice activated during failing executions for automatic root-cause localization of the fault. --- ## 🧬 Mutation Testing and Reliability Models See the complete guide in [mutation-and-fault-based-testing.md](./references/mutation-and-fault-based-testing.md). - **Mutation Score ($MS$)**: $$MS(T, P) = \frac{\text{Dead Mutants}}{\text{Total Mutants} - \text{Equivalent Mutants}} \times 100\%$$ - **Mutation Operators**: AOR (Arithmetic), ROR (Relational), COR (Conditional), SDL (Statement Deletion). - **Mills Fault Seeding Model (Capture-Recapture)**: $$\hat{N} = \frac{n \cdot S}{s} \implies N_{\text{residual}} = n \left(\frac{S}{s} - 1\right)$$ --- ## 🏗️ Integration and Object-Oriented Strategies See the complete guide in [integration-and-system-testing.md](./references/integration-and-system-testing.md). - **Call-Graph-Based Integration**: Pairwise Integration and Neighborhood Integration instead of naive static decomposition. - **MM-Paths (Method-to-Method Paths)**: Inter-class method execution chains triggered by messages. - **OO Testing**: Mitigation for Inheritance pitfalls (test flattening), Polymorphism (dynamic binding matrices), and Object State. --- ## 📋 In-Depth Reference Documents 1. [Black-Box Techniques (BVA, Partitions, Decision Tables, FSM, Pairwise)](./references/black-box-techniques.md) 2. [Structural Testing, McCabe Complexity, MC/DC, and Data Flow](./references/white-box-and-dataflow.md) 3. [Mutation Testing, Fault Seeding, and Metamorphic Testing](./references/mutation-and-fault-based-testing.md) 4. [Integration Strategies, MM-Paths, and OO Testing](./references/integration-and-system-testing.md) 5. [Test Management, Static Reviews, and ISTQB](./references/istqb-test-management-and-reviews.md) --- ## 🛠️ Practical Example: BVA + Decision Table in TypeScript ```typescript import { describe, it, expect } from 'vitest'; // Discount calculation and eligibility function export interface DiscountInput { customerAge: number; // Valid limits: [18, 100] cartValue: number; // Valid limits: [1, 10000] isLoyalMember: boolean; } export function calculateDiscount(input: DiscountInput): number { if (input.customerAge < 18 || input.customerAge > 100) { throw new Error('Age outside the allowed range [18, 100].'); } if (input.cartValue < 1 || input.cartValue > 10000) { throw new Error('Cart value outside the range [1, 10000].'); } if (input.isLoyalMember && input.cartValue >= 1000) { return 0.20; // 20% discount } if (input.isLoyalMember || input.customerAge >= 60) { return 0.10; // 10% discount } return 0.0; } describe('calculateDiscount - BVA & Decision Table Tests', () => { // 1. Robustness Tests at the Age Limits [18, 100] it.each([ { age: 17, cart: 500, loyal: false, error: true }, // min- (Invalid Robust) { age: 18, cart: 500, loyal: false, expected: 0.0 }, // min { age: 19, cart: 500, loyal: false, expected: 0.0 }, // min+ { age: 99, cart: 500, loyal: false, expected: 0.10 },// max- (Elderly) { age: 100, cart: 500, loyal: false, expected: 0.10 },// max { age: 101, cart: 500, loyal: false, error: true }, // max+ (Invalid Robust) ])('validates age limits (BVA): age=$age', ({ age, cart, loyal, expected, error }) => { if (error) { expect(() => calculateDiscount({ customerAge: age, cartValue: cart, isLoyalMember: loyal })).toThrow(); } else { const discount = calculateDiscount({ customerAge: age, cartValue: cart, isLoyalMember: loyal }); expect(discount).toBe(expected); } }); // 2. Decision Table Rule Tests it('applies a 20% discount for a loyal member with cart >= 1000', () => { const discount = calculateDiscount({ customerAge: 30, cartValue: 1000, isLoyalMember: true }); expect(discount).toBe(0.20); }); }); ``` --- ## 🤖 LLM-Assisted Testing (Winteringham) - **Area-of-effect model:** the tester's judgment sits at the center; LLM abilities (generation, transformation, translation) extend reach. Every tool output loops back to a human — over-reliance shrinks effective coverage, and automation bias (trusting output because a tool produced it) is the named hazard. - **Risk triad for LLM use:** hallucinations, data provenance (can you trust output origins), data privacy (what you send to third-party APIs). Standing posture: healthy skepticism. - **Prompt principles for testers:** separate instructions from data with delimiters; request structured output and switch formats on demand; give the model a bail-out phrase to suppress guessed answers; use few-shot examples; instruct explicit step-by-step reasoning. Maintain a versioned library of reusable prompts. - **Where LLMs fit:** test data generation under explicit rules; risk and test-idea suggestions (never the sole arbiter); code snippets for automation parts; format transformation (text→SQL, language-to-language with verification); summarizing exploratory notes into reports; natural-language breakdowns of unfamiliar code for risk analysis. - **LLM-assisted TDD:** start with a prompt that generates clarifying questions about the story (what/where/why/when/who/how filtered through chosen quality characteristics); triage answers, then run small red-green-refactor loops with an IDE copilot per increment. Generated ambiguity questions force design decisions before code exists. - **Planning and charters:** pair planning prompts with testing heuristics and quality-characteristic lists; convert risks into charters ("explore X with Y to discover Z"); use LLMs during sessions for code understanding, session data and bug investigation, then summarize notes afterwards. - **Context customization decision:** RAG (near-zero learning curve, cheap to start, token costs grow, weak control) versus fine-tuning (steep curve, real costs, slow, strong control); combine only when the compounded debugging burden is acceptable. - **AI test agents:** goal-driven, perceptive, autonomous, adaptive — implemented via function calling where the model selects and sequences tool calls; build incrementally from a dummy agent and watch reliability and scope control. ## 🔗 Integration with Other Skills - [qa-engineer](../../roles/qa-engineer/SKILL.md): Quality planning and orchestration and defect reports. - [framework-pytest](../framework-testing-python/SKILL.md): Test automation in Python with fixtures and formal parameterized tests. - [framework-unittest](../framework-testing-python/SKILL.md): Unit tests with structured TestCase classes. - [framework-jest](../framework-testing-javascript/SKILL.md) / [framework-mocha](../framework-testing-javascript/SKILL.md): Automation in JS/TS ecosystems. - [framework-criterion](../framework-criterion/SKILL.md): Low-level testing for compiled C/C++ languages.