neuron-evaluation-engineer

# Neuron AI Evaluation Engineer This skill helps you create and run evaluations for AI systems in Neuron AI. The evaluation system provides dataset-driven testing with flexible assertions, comprehensive result reporting, and extensible output drivers. ## Core Concepts ### The Evaluation System Evaluations test AI systems using three main components: 1. **Evaluators** - Test classes that define what to run and how to validate 2. **Datasets** - Test data sources (arrays, JSON files) 3. **Assertions** - Validation rules for checking outputs ``` Dataset Items → Evaluator::run() → Output → Evaluator::evaluate() → Assertions → Results ``` ### Evaluation Flow For each dataset item: 1. `setUp()` - Initialize resources (once per evaluator) 2. `run(datasetItem)` - Execute your AI logic 3. `evaluate(output, datasetItem)` - Assert against expected results 4. Repeat for next item **Note:** Each evaluation starts with a fresh assertion executor - no manual reset needed. ## Creating Custom Evaluators ### Basic Evaluator ```php use NeuronAI\Evaluation\BaseEvaluator; use NeuronAI\Evaluation\Contracts\DatasetInterface; use NeuronAI\Evaluation\Assertions\StringContains; use NeuronAI\Evaluation\Dataset\ArrayDataset; use NeuronAI\Agent; use NeuronAI\Agent\SystemPrompt; class ContainsEvaluator extends BaseEvaluator { public function getDataset(): DatasetInterface { return new ArrayDataset([ [ 'text' => 'I love this product!', 'content' => 'product', ], [ 'text' => 'This is terrible.', 'content' => 'positive', ], ]); } public function run(array $datasetItem): mixed { $response = MyAgent::make()->chat( new UserMessage($datasetItem['text']) )->getMessage(); return $response->getContent(); } public function evaluate(mixed $output, array $datasetItem): void { $this->assert( new StringContains($datasetItem['content']), $output ); } } ``` ### JSON Dataset For larger datasets, use JSON files: ```php use NeuronAI\Evaluation\Dataset\JsonDataset; public function getDataset(): DatasetInterface { return new JsonDataset(__DIR__ . '/datasets/sentiment.json'); } ``` JSON format (`sentiment.json`): ```json [ {"text": "I love this!", "expected": "positive"}, {"text": "This is bad.", "expected": "negative"} ] ``` ## Built-in Assertions ### String Assertions #### StringContains Check if the output contains a substring: ```php $this->assert(new StringContains('positive'), $output); ``` #### StringContainsAll Check if the output contains all keywords: ```php $this->assert(new StringContainsAll(['hello', 'world']), $output); ``` #### StringContainsAny Check if the output contains any of the keywords: ```php $this->assert(new StringContainsAny(['success', 'completed']), $output); ``` #### StringStartsWith Check if the output starts with a prefix: ```php $this->assert(new StringStartsWith('Hello'), $output); ``` #### StringEndsWith Check if the output ends with a suffix: ```php $this->assert(new StringEndsWith('!'), $output); ``` #### StringLengthBetween Check if the string length is within range: ```php $this->assert(new StringLengthBetween(10, 100), $output); ``` #### StringDistance Check string similarity using Levenshtein distance: ```php $this->assert(new StringDistance( reference: 'expected text', threshold: 0.5, // Minimum similarity score maxDistance: 50 // Maximum allowed edits ), $output); ``` #### StringSimilarity Check string similarity using embeddings: ```php use NeuronAI\Evaluation\Assertions\StringSimilarity; use NeuronAI\RAG\Embeddings\OpenAI\OpenAIEmbeddings; $this->assert(new StringSimilarity( reference: 'The quick brown fox', embeddingsProvider: new OpenAIEmbeddings(key: 'YOUR_KEY'), threshold: 0.6 ), $output); ``` ### Pattern Assertions #### MatchesRegex Match against regular expression: ```php $this->assert(new MatchesRegex('/^\d{3}-\d{2}-\d{4}$/'), $output); ``` ### Structure Assertions #### IsValidJson Check if the output is valid JSON: ```php $this->assert(new IsValidJson(), $output); ``` ### AI Judge Assertions #### AgentJudge Use an AI agent to evaluate outputs with custom criteria: ```php use NeuronAI\Evaluation\Assertions\AgentJudge; use NeuronAI\Agent; $judge = Agent::make() ->setInstructions('You are an expert evaluator for customer support responses.'); // Reference-free evaluation (criteria only) $this->assert(new AgentJudge( judge: $judge, criteria: 'Response should be helpful, polite, and address the customer\'s question directly', threshold: 0.7 ), $output); // Reference-based evaluation (compare to expected) $this->assert(new AgentJudge( judge: $judge, criteria: 'The response should convey the same meaning as the reference', threshold: 0.8, reference: $datasetItem['expected_answer'] ), $output); // With few-shot examples for calibration $this->assert(new AgentJudge( judge: $judge, criteria: 'Rate the factual accuracy of the response', threshold: 0.7, examples: [ [ 'input' => 'What is 2+2?', 'output' => '2+2 equals 4', 'score' => 1.0, 'reasoning' => 'Mathematically correct and clear.', ], ] ), $output); ``` #### Pre-configured Judges Built-in judges for common evaluation scenarios: ```php use NeuronAI\Evaluation\Assertions\Judges\{FaithfulnessJudge, CorrectnessJudge, RelevanceJudge, HelpfulnessJudge}; // Faithfulness - check if output is grounded in context (no hallucinations) $this->assert(new FaithfulnessJudge( judge: $judge, context: $retrievedDocuments, threshold: 0.7 ), $output); // Correctness - compare to expected answer $this->assert(new CorrectnessJudge( judge: $judge, expected: $datasetItem['expected_answer'], threshold: 0.7 ), $output); // Relevance - check if output addresses the question $this->assert(new RelevanceJudge( judge: $judge, question: $datasetItem['question'], threshold: 0.7 ), $output); // Helpfulness - evaluate utility and actionability $this->assert(new HelpfulnessJudge( judge: $judge, threshold: 0.7 ), $output); ``` ### Creating Custom Assertions ```php use NeuronAI\Evaluation\Assertions\AbstractAssertion; use NeuronAI\Evaluation\AssertionResult; class GreaterThanAssertion extends AbstractAssertion { public function __construct( private readonly float $threshold ) {} public function evaluate(mixed $actual): AssertionResult { if (!is_numeric($actual)) { return AssertionResult::fail( 0.0, 'Expected numeric value, got ' . gettype($actual), ); } if ($actual > $this->threshold) { return AssertionResult::pass(1.0); } return AssertionResult::fail( 0.0, "Expected {$actual} to be greater than {$this->threshold}", ); } } ``` Use it: ```php $this->assert(new GreaterThanAssertion(0.8), $score); ``` ## Running Evaluations ### CLI Command ```bash # Run all evaluators in a directory vendor/bin/neuron evaluation /path/to/evaluators # Verbose output (shows evaluator names) vendor/bin/neuron evaluation --verbose /path/to/evaluators # Using --path flag vendor/bin/neuron evaluation --path=/path/to/evaluators # Help vendor/bin/neuron evaluation --help ``` ### Programmatic Execution ```php use NeuronAI\Evaluation\Runner\EvaluatorRunner; $runner = new EvaluatorRunner(); $evaluator = new MyEvaluator(); $summary = $runner->run($evaluator); echo "Passed: {$summary->getPassedCount()}\n"; echo "Failed: {$summary->getFailedCount()}\n"; echo "Success Rate: {$summary->getSuccessRate() * 100}%\n"; ``` ## Output Configuration ### Config File Create `evaluation.php` in project root: ```php <?php use NeuronAI\Evaluation\Output\ConsoleOutput; use NeuronAI\Evaluation\Output\JsonOutput; return [ 'output' => [ // Simple driver (no options) ConsoleOutput::class, // Driver with options (class as key) JsonOutput::class => [ 'path' => 'evaluation-results.json', ], ], ]; ``` **Default behavior**: If no config exists, uses `ConsoleOutput`. ### Built-in Output Drivers #### ConsoleOutput ```php ConsoleOutput::class => ['verbose' => true] ``` - `verbose` - Show detailed input/output for failures #### JsonOutput ```php // Write to file JsonOutput::class => ['path' => 'results.json'] // Write to stdout JsonOutput::class ``` ### Creating Custom Output Drivers ```php use NeuronAI\Evaluation\Contracts\EvaluationOutputInterface; use NeuronAI\Evaluation\Runner\EvaluatorSummary; class DatabaseOutput implements EvaluationOutputInterface { public function __construct( private readonly \PDO $pdo, private readonly string $table = 'evaluations' ) {} public function output(EvaluatorSummary $summary): void { $stmt = $this->pdo->prepare( "INSERT INTO {$this->table} (passed, failed, success_rate, total_time, created_at) VALUES (?, ?, ?, ?, NOW())" ); $stmt->execute([ $summary->getPassedCount(), $summary->getFailedCount(), $summary->getSuccessRate(), $summary->getTotalExecutionTime(), ]); } } ``` Register in config: ```php DatabaseOutput::class => [ 'pdo' => new \PDO('mysql:host=localhost;dbname=evaluations', 'user', 'pass'), 'table' => 'evaluations', ] ``` ## Project Setup ### Configuring Autoloader Add evaluators directory to `composer.json`: ```json { "autoload-dev": { "psr-4": { "App\\Evaluators\\": "evaluators/" } } } ``` ### Directory Structure ``` project/ ├── evaluators/ │ ├── SentimentEvaluator.php │ ├── SummarizationEvaluator.php │ └── datasets/ │ ├── sentiment.json │ └── summarization.json ├── evaluation.php └── vendor/bin/neuron ``` ## Result Analysis ### Accessing Results ```php $summary = $runner->run($evaluator); // Basic stats $summary->getPassedCount(); // int $summary->getFailedCount(); // int $summary->getTotalCount(); // int $summary->getSuccessRate(); // float (0.0 - 1.0) // Timing $summary->getTotalExecutionTime(); // float (seconds) $summary->getAverageExecutionTime(); // float (seconds) // Assertions $summary->getTotalAssertions(); // int $summary->getTotalAssertionsPassed(); // int $summary->getTotalAssertionsFailed(); // int $summary->getAssertionSuccessRate(); // float (0.0 - 1.0)