agentic-testing-patterns

AI-first testing methodology where autonomous agents plan, generate, execute, and maintain test suites with minimal human intervention, covering agent orchestration, feedback loops, and intelligent test prioritization.

仓库: PramodDutta/qaskills
最近来源活动: 2026年4月14日 07:59
检测到的 SKILL.md 语言: 英语
星标: 201
分支: 21

安装方式

默认使用会先检查来源的 Prompt；你也可以切换为直接命令，或下载本地副本。

检查来源文件

决定是否安装前，请先阅读 SKILL.md，以及 SkillsMP 当前展示的配套文件。

返回搜索返回安装

name	Agentic Testing Patterns
description	AI-first testing methodology where autonomous agents plan, generate, execute, and maintain test suites with minimal human intervention, covering agent orchestration, feedback loops, and intelligent test prioritization.
version	1.0.0
author	thetestingacademy
license	MIT
tags	["agentic-testing","ai-first","autonomous-testing","test-orchestration","intelligent-testing","feedback-loops","test-prioritization","autonomous-agents","ai-qa"]
testingTypes	["e2e","integration","unit","performance","accessibility"]
frameworks	["playwright","vitest","jest","cypress"]
languages	["typescript","javascript","python"]
domains	["web","backend","api","mobile"]
agents	["claude-code","cursor","github-copilot","windsurf","codex","aider","continue","cline","zed","bolt","gemini-cli","amp"]

Agentic Testing Patterns Skill

You are an expert in agentic testing methodology where AI agents autonomously plan, generate, execute, and maintain test suites. When the user asks you to implement agentic testing workflows, create autonomous test pipelines, or build AI-driven quality assurance systems, follow these detailed instructions.

Core Principles

Agent autonomy with human oversight -- Agents should operate independently for routine tasks but escalate to humans for ambiguous requirements, security-sensitive tests, and novel failure patterns.
Intent-driven test specification -- Tests are specified as high-level intents (what to verify) rather than step-by-step instructions. Agents determine the optimal implementation strategy.
Continuous learning from failures -- Every test failure feeds back into the agent's knowledge base. Agents improve their test generation and maintenance strategies over time.
Risk-based test prioritization -- Agents analyze code changes, historical failure data, and business impact to determine which tests to run and in what order.
Multi-agent collaboration -- Different agents specialize in different tasks: one plans, one generates, one executes, one analyzes results. They communicate through structured protocols.
Observable and auditable -- Every agent decision must be logged with reasoning. Humans must be able to trace why a test was generated, modified, or skipped.
Graceful degradation -- When AI services are unavailable, the system falls back to deterministic test execution. Agent-generated tests must be valid standalone tests.

Project Structure

agentic-tests/
  agents/
    coordinator/
      coordinator-agent.ts
      task-queue.ts
      priority-engine.ts
    analyzer/
      code-change-analyzer.ts
      failure-pattern-detector.ts
      coverage-gap-finder.ts
    generator/
      test-generator.ts
      fixture-generator.ts
      mock-generator.ts
    executor/
      test-runner.ts
      parallel-executor.ts
      result-collector.ts
    reporter/
      insight-generator.ts
      trend-analyzer.ts
      alert-system.ts
  knowledge/
    failure-patterns.json
    selector-mappings.json
    test-templates/
      unit-template.ts
      integration-template.ts
      e2e-template.ts
  pipelines/
    ci-pipeline.ts
    pr-review-pipeline.ts
    nightly-pipeline.ts
    deployment-pipeline.ts
  config/
    agent-config.ts
    pipeline-config.ts
    model-config.ts
  tests/
    agent-tests/
      coordinator.test.ts
      analyzer.test.ts
      generator.test.ts
  monitoring/
    agent-metrics.ts
    cost-dashboard.ts
    quality-tracker.ts

Coordinator Agent


 { ,  }  ;
 {  }  ;
 { ,  }  ;
 {  }  ;

   {
  : ;
  :  |  |  |  | ;
  : ;
  : <, >;
  :  |  |  | ;
  ?: ;
  ?: ;
  : ;
  ?: ;
}

   {
  : ;
  : ;
  : ;
  : ;
}

   {
   : [] = [];
   : ;
   : ;
   : ;
   : ;
   : ;

  () {
    . =  ();
    . =  ();
    . =  ();
    . =  ();
    . = {
      : ,
      : ,
      : ,
      : ,
      ...config,
    };
  }

   (: ): <> {
     startTime = .();
     : [] = [];

    
     analysis =  ..(trigger.);

    
     testPlan = .(analysis, trigger);

    
     ( gap  testPlan.) {
       task = .(, { gap, analysis });
      tasks.(task);
       generated =  ..(gap);
      task. = generated;
      task. = ;
    }

    
     testFiles = .(analysis, testPlan);
     executionResult =  ..(testFiles);

    
     insights =  ..(executionResult, analysis);

    
     (executionResult.. > ) {
       .(executionResult.);
    }

     {
      : .() - startTime,
      : executionResult.,
      : executionResult.,
      : executionResult..,
      : testPlan..,
      insights,
      tasks,
    };
  }

   (: , : ):  {
     coverageGaps = analysis..( ({
      : change.,
      : change.,
      : .(change),
      : .(change),
    }));

     existingTests = analysis..( ({
      : test,
      : analysis..(test),
    }));

     {
      : coverageGaps.( b. - a.),
      existingTests,
      : coverageGaps. *  + existingTests. * ,
    };
  }

   (: ):  {
     (change..() || change..())  ;
     (change..() || change..())  ;
     (change..() || change..())  ;
     ;
  }

   (: ):  {
     priority = ;
     (change..() || change..()) priority += ;
     (change. > ) priority += ;
     (change.) priority += ;
     .(, priority);
  }

   (: , : ): [] {
     tests =  <>();
     ( test  analysis.) tests.(test);
     ( test  analysis.) tests.(test);
     [...tests];
  }

    (: []): <> {
     ( failure  failures) {
       (failure.) {
        .();
      }   (failure.) {
        .();
      }
    }
  }

   (: [], : <, >):  {
     {
      : ,
      ,
      : ,
      payload,
      : ,
      :  ().(),
    };
  }
}

  {
  :  |  |  | ;
  : [];
  : ;
  : ;
}

  {
  : ;
  : ;
  : ;
  : ;
  : ;
  : ;
  : [];
}

  {
  : [];
  : [];
  : ;
}

// agentic-tests/agents/analyzer/code-change-analyzer.ts
import { execSync } from 'child_process';

export interface ChangeAnalysis {
  changedFiles: FileChange[];
  affectedTests: string[];
  directlyAffectedTests: string[];
  uncoveredChanges: FileChange[];
  brokenTests: string[];
  riskScore: number;
  summary: string;
}

export interface FileChange {
  file: string;
  linesChanged: number;
  functions: string[];
  isNewFile: boolean;
  isDeleted: boolean;
  changeType: 'feature' | 'bugfix' | 'refactor' | 'config' | 'test';
}

export class CodeChangeAnalyzer {
  async analyzeChanges(files: string[]): Promise<ChangeAnalysis> {
    const changedFiles = await this.getFileDetails(files);
    const affectedTests = this.findAffectedTests(changedFiles);
    const directlyAffectedTests = this.findDirectTests(changedFiles);
    const uncoveredChanges = this.findUncoveredChanges(changedFiles, affectedTests);
    const riskScore = this.calculateRiskScore(changedFiles);

    return {
      changedFiles,
      affectedTests,
      directlyAffectedTests,
      uncoveredChanges,
      brokenTests: [],
      riskScore,
      summary: this.buildSummary(changedFiles, riskScore),
    };
  }

  private async getFileDetails(files: string[]): Promise<FileChange[]> {
    return files.map((file) => {
      const isTest = file.includes('.test.') || file.includes('.spec.');
      return {
        file,
        linesChanged: this.countChangedLines(file),
        functions: this.extractChangedFunctions(file),
        isNewFile: this.isNewFile(file),
        isDeleted: false,
        changeType: isTest ? 'test' : this.inferChangeType(file),
      };
    });
  }

  private countChangedLines(file: string): number {
    try {
      const diff = execSync(`git diff --numstat HEAD~1 -- "${file}"`, { encoding: 'utf-8' });
      const parts = diff.trim().split('\t');
      return parseInt(parts[0] || '0', 10) + parseInt(parts[1] || '0', 10);
    } catch {
      return 0;
    }
  }

  private extractChangedFunctions(file: string): string[] {
    try {
      const diff = execSync(`git diff -U0 HEAD~1 -- "${file}"`, { encoding: 'utf-8' });
      const functionMatches = diff.match(/(?:function|const|class|export)\s+(\w+)/g) || [];
      return [...new Set(functionMatches.map((m) => m.split(/\s+/).pop()!))];
    } catch {
      return [];
    }
  }

  private isNewFile(file: string): boolean {
    try {
      execSync(`git log --oneline -1 HEAD~1 -- "${file}"`, { encoding: 'utf-8' });
      return false;
    } catch {
      return true;
    }
  }

  private inferChangeType(file: string): FileChange['changeType'] {
    if (file.includes('config') || file.endsWith('.json') || file.endsWith('.yaml')) return 'config';
    if (file.includes('fix') || file.includes('bug')) return 'bugfix';
    return 'feature';
  }

  private findAffectedTests(changes: FileChange[]): string[] {
    const tests: string[] = [];
    for (const change of changes) {
      const basename = change.file.replace(/\.(ts|js|tsx|jsx)$/, '');
      const possibleTests = [
        `${basename}.test.ts`,
        `${basename}.spec.ts`,
        `${basename}.test.tsx`,
        `__tests__/${basename.split('/').pop()}.test.ts`,
      ];
      tests.push(...possibleTests);
    }
    return [...new Set(tests)];
  }

  private findDirectTests(changes: FileChange[]): string[] {
    return changes
      .filter((c) => c.changeType === 'test')
      .map((c) => c.file);
  }

  private findUncoveredChanges(changes: FileChange[], tests: string[]): FileChange[] {
    return changes.filter((change) => {
      if (change.changeType === 'test') return false;
      const basename = change.file.replace(/\.(ts|js|tsx|jsx)$/, '');
      return !tests.some((test) => test.includes(basename.split('/').pop()!));
    });
  }

  private calculateRiskScore(changes: FileChange[]): number {
    let score = 0;
    for (const change of changes) {
      if (change.file.includes('auth')) score += 3;
      if (change.file.includes('payment') || change.file.includes('billing')) score += 3;
      if (change.file.includes('security')) score += 3;
      if (change.linesChanged > 100) score += 2;
      if (change.isNewFile) score += 1;
      score += change.functions.length * 0.5;
    }
    return Math.min(10, score);
  }

  private buildSummary(changes: FileChange[], riskScore: number): string {
    const total = changes.length;
    const newFiles = changes.filter((c) => c.isNewFile).length;
    const totalLines = changes.reduce((sum, c) => sum + c.linesChanged, 0);
    return `${total} files changed (${newFiles} new), ${totalLines} lines modified, risk score: ${riskScore}/10`;
  }
}

// agentic-tests/agents/generator/test-generator.ts
import Anthropic from '@anthropic-ai/sdk';
import { readFileSync } from 'fs';

export interface TestGenerationRequest {
  sourceFile: string;
  sourceCode: string;
  testType: 'unit' | 'integration' | 'e2e' | 'api';
  existingTests?: string;
  coverageReport?: string;
}

export interface GeneratedTestSuite {
  testFile: string;
  testCode: string;
  testCount: number;
  coverageEstimate: number;
  confidence: number;
  reasoning: string;
}

export class TestGenerator {
  private client: Anthropic;

  constructor() {
    this.client = new Anthropic();
  }

  async generateForGap(gap: any): Promise<GeneratedTestSuite> {
    const sourceCode = readFileSync(gap.file, 'utf-8');
    return this.generateTests({
      sourceFile: gap.file,
      sourceCode,
      testType: gap.suggestedTestType,
    });
  }

  async generateTests(request: TestGenerationRequest): Promise<GeneratedTestSuite> {
    const systemPrompt = `You are an expert test generator. Generate comprehensive tests following these rules:
1. Cover all public functions and edge cases
2. Use descriptive test names that explain the expected behavior
3. Follow the Arrange-Act-Assert pattern
4. Include both positive and negative test cases
5. Mock external dependencies appropriately
6. Use TypeScript with strict types
7. Generate tests appropriate for the test type (unit/integration/e2e/api)`;

    const prompt = `Generate ${request.testType} tests for this file:

File: ${request.sourceFile}
\`\`\`typescript
${request.sourceCode}
\`\`\`

${request.existingTests ? `Existing tests (avoid duplication):\n${request.existingTests}` : ''}
${request.coverageReport ? `Coverage gaps:\n${request.coverageReport}` : ''}

Generate a complete test file. Return ONLY valid TypeScript code.`;

    const response = await this.client.messages.create({
      model: 'claude-sonnet-4-20250514',
      max_tokens: 4096,
      temperature: 0,
      system: systemPrompt,
      messages: [{ role: 'user', content: prompt }],
    });

    const text = response.content[0].type === 'text' ? response.content[0].text : '';
    const codeMatch = text.match(/```typescript\n([\s\S]*?)```/);
    const testCode = codeMatch ? codeMatch[1] : text;

    const testCount = (testCode.match(/\bit\s*\(/g) || []).length;

    return {
      testFile: request.sourceFile.replace(/\.(ts|tsx)$/, '.test.ts'),
      testCode: testCode.trim(),
      testCount,
      coverageEstimate: Math.min(95, testCount * 15),
      confidence: 0.8,
      reasoning: `Generated ${testCount} ${request.testType} tests covering public API`,
    };
  }
}

// agentic-tests/agents/analyzer/failure-pattern-detector.ts
export interface FailurePattern {
  pattern: string;
  frequency: number;
  lastSeen: string;
  affectedTests: string[];
  rootCause: string;
  suggestedFix: string;
  isFlaky: boolean;
}

export class FailurePatternDetector {
  private patterns: Map<string, FailurePattern> = new Map();

  recordFailure(testName: string, errorMessage: string, stackTrace: string): void {
    const patternKey = this.extractPattern(errorMessage, stackTrace);
    const existing = this.patterns.get(patternKey);

    if (existing) {
      existing.frequency++;
      existing.lastSeen = new Date().toISOString();
      if (!existing.affectedTests.includes(testName)) {
        existing.affectedTests.push(testName);
      }
    } else {
      this.patterns.set(patternKey, {
        pattern: patternKey,
        frequency: 1,
        lastSeen: new Date().toISOString(),
        affectedTests: [testName],
        rootCause: this.inferRootCause(errorMessage, stackTrace),
        suggestedFix: this.suggestFix(errorMessage),
        isFlaky: false,
      });
    }
  }

  detectFlakyTests(history: Array<{ testName: string; passed: boolean }>): string[] {
    const testResults = new Map<string, boolean[]>();

    for (const run of history) {
      const results = testResults.get(run.testName) || [];
      results.push(run.passed);
      testResults.set(run.testName, results);
    }

    const flakyTests: string[] = [];
    for (const [testName, results] of testResults) {
      if (results.length >= 3) {
        const passRate = results.filter(Boolean).length / results.length;
        if (passRate > 0.1 && passRate < 0.9) {
          flakyTests.push(testName);
        }
      }
    }

    return flakyTests;
  }

  getTopPatterns(limit = 10): FailurePattern[] {
    return [...this.patterns.values()]
      .sort((a, b) => b.frequency - a.frequency)
      .slice(0, limit);
  }

  private extractPattern(error: string, stack: string): string {
    const normalized = error
      .replace(/\d+/g, 'N')
      .replace(/['"][^'"]*['"]/g, '"..."')
      .replace(/0x[a-f0-9]+/gi, '0xADDR')
      .trim();
    return normalized.substring(0, 200);
  }

  private inferRootCause(error: string, stack: string): string {
    if (error.includes('timeout')) return 'Element not found within timeout period';
    if (error.includes('ECONNREFUSED')) return 'Server not running or not accessible';
    if (error.includes('selector')) return 'DOM element selector changed';
    if (error.includes('assertion')) return 'Expected value does not match actual';
    return 'Unknown root cause - requires investigation';
  }

  private suggestFix(error: string): string {
    if (error.includes('timeout')) return 'Increase timeout or add explicit wait for element';
    if (error.includes('ECONNREFUSED')) return 'Verify server is running before test execution';
    if (error.includes('selector')) return 'Update selector to use data-testid or ARIA role';
    return 'Review test and application code for changes';
  }
}

// agentic-tests/pipelines/pr-review-pipeline.ts
import { CoordinatorAgent } from '../agents/coordinator/coordinator-agent';

export async function runPRReviewPipeline(prInfo: {
  branch: string;
  commit: string;
  changedFiles: string[];
  prTitle: string;
  prBody: string;
}): Promise<{
  approved: boolean;
  summary: string;
  details: any;
}> {
  const coordinator = new CoordinatorAgent({
    maxConcurrentTasks: 3,
    maxBudgetPerPipeline: 5.0,
    autoApproveConfidence: 0.9,
  });

  const result = await coordinator.runPipeline({
    type: 'pr',
    changes: prInfo.changedFiles,
    branch: prInfo.branch,
    commit: prInfo.commit,
  });

  const approved = result.testsFailed === 0 && result.testsRun > 0;

  return {
    approved,
    summary: approved
      ? `All ${result.testsRun} tests passed. ${result.testsGenerated} new tests generated.`
      : `${result.testsFailed}/${result.testsRun} tests failed. Review required.`,
    details: result,
  };
}

// agentic-tests/monitoring/agent-metrics.ts
export interface AgentMetrics {
  totalRuns: number;
  totalTestsGenerated: number;
  totalTestsHealed: number;
  totalCost: number;
  averageConfidence: number;
  healingSuccessRate: number;
  generationAccuracy: number;
  averagePipelineDuration: number;
}

export class AgentMetricsCollector {
  private metrics: AgentMetrics = {
    totalRuns: 0,
    totalTestsGenerated: 0,
    totalTestsHealed: 0,
    totalCost: 0,
    averageConfidence: 0,
    healingSuccessRate: 0,
    generationAccuracy: 0,
    averagePipelineDuration: 0,
  };

  private confidenceScores: number[] = [];
  private healingAttempts = 0;
  private healingSuccesses = 0;
  private pipelineDurations: number[] = [];

  recordRun(result: {
    testsGenerated: number;
    testsHealed: number;
    cost: number;
    confidence: number;
    duration: number;
    healingSuccess: boolean;
  }): void {
    this.metrics.totalRuns++;
    this.metrics.totalTestsGenerated += result.testsGenerated;
    this.metrics.totalTestsHealed += result.testsHealed;
    this.metrics.totalCost += result.cost;
    this.confidenceScores.push(result.confidence);
    this.pipelineDurations.push(result.duration);

    if (result.testsHealed > 0) {
      this.healingAttempts++;
      if (result.healingSuccess) this.healingSuccesses++;
    }

    this.metrics.averageConfidence =
      this.confidenceScores.reduce((a, b) => a + b, 0) / this.confidenceScores.length;
    this.metrics.healingSuccessRate =
      this.healingAttempts > 0 ? this.healingSuccesses / this.healingAttempts : 0;
    this.metrics.averagePipelineDuration =
      this.pipelineDurations.reduce((a, b) => a + b, 0) / this.pipelineDurations.length;
  }

  getMetrics(): AgentMetrics {
    return { ...this.metrics };
  }

  getReport(): string {
    const m = this.metrics;
    return [
      `=== Agentic Testing Metrics ===`,
      `Total runs: ${m.totalRuns}`,
      `Tests generated: ${m.totalTestsGenerated}`,
      `Tests healed: ${m.totalTestsHealed}`,
      `Total cost: $${m.totalCost.toFixed(2)}`,
      `Avg confidence: ${(m.averageConfidence * 100).toFixed(1)}%`,
      `Healing success rate: ${(m.healingSuccessRate * 100).toFixed(1)}%`,
      `Avg pipeline duration: ${(m.averagePipelineDuration / 1000).toFixed(1)}s`,
    ].join('\n');
  }
}