test-driven-ai-agents

name	test-driven-ai-agents
title	Test-Driven AI Agent Definition: Compiling Tool-Using Agents from Behavioral Specs
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2603.08806
keywords	["Agent Testing","Behavioral Verification","Test-Driven Development","Prompt Engineering","Agent Safety"]
description	Applies test-driven development to agent prompts by iteratively refining prompts against behavioral test suites until compliance is achieved. Enables measurable agent behavior validation through semantic mutation testing and specification evolution.

Test-Driven AI Agent Definition: Compiling Agent Prompts from Behavioral Specifications

Production LLM agents lack measurable behavioral compliance. Small prompt changes cause silent regressions, tool misuse goes undetected, and policy violations emerge only after deployment. Teams cannot verify agents behave correctly across specified scenarios without tedious manual testing. Test-Driven AI Agent Definition (TDAD) adapts test-driven development (TDD) to agents: write behavioral specs as executable tests, then iteratively refine prompts until all tests pass—like TDD for code but for agent prompts.

Core Concept

Traditional approach: Write agent prompt → Deploy → Hope it works

TDAD: Write behavioral tests → Iteratively refine prompt to pass tests → Verify robustness through mutation testing

Key insight: Prompts are compilation artifacts. Tests specify desired behavior; the compiler (you + LLM) generates a prompt that makes the agent behave correctly. Tests provide objective verification unavailable with manual inspection.

Architecture Overview

• Behavioral Test Specification: YAML specs defining must-do actions (MFT), invariants (INV), directives (DIR)
• Visible/Hidden Test Split: Visible tests guide prompt refinement; hidden tests measure generalization
• Semantic Mutation Testing: Generate faulty prompt variants to verify tests catch regressions
• Spec Evolution: Test robustness as specifications change over time
• Multi-Role Compilation: TestSmith generates tests, PromptSmith refines prompts, MutationSmith creates variants

Implementation Steps

Implement a test-driven prompt refinement system.

Behavioral Test Specification

import yaml
from typing import Dict, List, Any
from dataclasses import dataclass

@dataclass
class BehavioralSpec:
    """Specification of desired agent behavior."""
    task_description: str
    must_follow_tests: List[Dict[str, Any]]  
    invariants: [[, ]]  
    directives: [[, ]]  

     () -> :
        
         yaml.dump({
            : .task_description,
            : .must_follow_tests,
            : .invariants,
            : .directives
        })


     () -> :
        
        data = yaml.safe_load(yaml_str)
         cls(
            task_description=data.get(, ),
            must_follow_tests=data.get(, []),
            invariants=data.get(, []),
            directives=data.get(, [])
        )



AGENT_SPEC =