ai-testing-strategy

name	ai-testing-strategy
description	Design comprehensive testing strategies for AI-powered features and LLM applications. Outputs test taxonomy, evaluation harness, regression suite, human evaluation protocol, and CI integration.
argument-hint	["AI feature type","LLM provider","quality dimensions","production traffic volume"]
allowed-tools	Read, Write, Bash

AI Testing Strategy

Testing AI-powered features requires a fundamentally different approach than testing deterministic code. LLM outputs vary, quality is multidimensional, and failure modes are qualitative rather than binary. A good AI testing strategy covers automated evals, human evaluation, regression detection, and production monitoring.

Test Taxonomy

1. UNIT TESTS (deterministic components)
   Test prompt construction, tool call parsing, response processing
   Traditional unit tests — fast, no LLM calls

2. EVAL TESTS (LLM quality assessment)
   Test LLM output quality against rubrics
   Use an LLM-as-judge or human annotation
   Slow, expensive — run on PR and nightly

3. REGRESSION TESTS (guard against degradation)
   Fixed set of inputs with expected behaviour
   Binary pass/fail or score threshold
   Catch model/prompt changes that degrade quality

4. HUMAN EVALUATION (gold standard)
   Periodic human rating of random production samples
   Calibrates automated evals against human judgement
   Monthly or quarterly

5. PRODUCTION MONITORING
   Sample production traffic, classify quality
   Alert on quality degradation
   Continuous

Automated Evaluation Harness

 anthropic
 json
 dataclasses  dataclass
 typing  

client = anthropic.Anthropic()


 :
    : 
    expected_criteria: []   
    forbidden_criteria: []  
    context:  = 


 :
    case_id: 
    output: 
    passed: 
    scores: [, ]
    reasoning: 

 () -> EvalResult:
    
    criteria_text = .join(  c  criteria)
    forbidden_text = .join(  f  forbidden)

    judge_prompt = 

    response = client.messages.create(
        model=,
        max_tokens=,
        messages=[{: , : judge_prompt}],
    )

    judgment = json.loads(response.content[].text)
     judgment

 :
     ():
        .sut = system_under_test
        .results = []

     () -> :
         i,   (cases):
            output = .sut(.)
            judgment = llm_judge(output, .expected_criteria, .forbidden_criteria)

            .results.append({
                : ,
                : .,
                : output,
                : judgment[]  judgment[] >= pass_threshold,
                : judgment[],
                : judgment[],
            })

        pass_rate = (  r  .results  r[]) / (.results)
        avg_score = (r[]  r  .results) / (.results)

         {
            : pass_rate,
            : avg_score,
            : (.results),
            : ([r  r  .results  r[]]),
            : [r  r  .results   r[]],
        }