output-dev-evaluator-function

# Creating Evaluator Functions ## Overview This skill documents how to create evaluator functions in `evaluators.ts` for Output SDK workflows. Evaluators are used to assess quality, validate outputs, and provide confidence-scored judgments about workflow results. ## When to Use This Skill - Implementing quality assessment for workflow outputs - Adding validation logic with confidence scores - Creating LLM-powered content evaluation - Building reusable evaluation components ## File Organization ### Option 1: Flat File (Default) For smaller workflows, use a single `evaluators.ts` file: ``` src/workflows/{workflow-name}/ ├── workflow.ts ├── steps.ts ├── evaluators.ts # All evaluators in one file ├── types.ts └── ... ``` ### Option 2: Folder-Based (Large workflows) For larger workflows with many evaluators, use an `evaluators/` folder: ``` src/workflows/{workflow-name}/ ├── workflow.ts ├── steps.ts ├── evaluators/ # Evaluators split into individual files │ ├── quality.ts │ ├── accuracy.ts │ └── completeness.ts ├── types.ts └── ... ``` ## Component Location Rules **Important**: `evaluator()` calls MUST be in files containing 'evaluators' in the path: - `src/workflows/my_workflow/evaluators.ts` ✓ - `src/workflows/my_workflow/evaluators/quality.ts` ✓ - `src/shared/evaluators/common_evaluators.ts` ✓ - `src/workflows/my_workflow/helpers.ts` ✗ (cannot contain evaluator() calls) ## Activity Isolation Constraints Evaluators are Temporal activities with strict import rules to ensure deterministic replay. ### Evaluators CAN import from: - Local workflow files: `./utils.js`, `./types.js`, `./helpers.js` - Local subdirectories: `./lib/helpers.js` - Shared utilities: `../../shared/utils/*.js` - Shared clients: `../../shared/clients/*.js` - Shared services: `../../shared/services/*.js` ### Evaluators CANNOT import: - Other evaluator files (activity isolation) - Step files - Workflow files **Example of WRONG imports:** ```typescript // WRONG - evaluators cannot import other evaluators import { otherEvaluator } from '../../shared/evaluators/other.js'; // ✗ import { anotherEvaluator } from './other_evaluators.js'; // ✗ ``` ## Critical Import Patterns ### Core Imports ```typescript // CORRECT - Import from @outputai/core import { evaluator, z, EvaluationBooleanResult, EvaluationNumberResult, EvaluationStringResult, EvaluationFeedback } from '@outputai/core'; // WRONG - Never import z from zod import { z } from 'zod'; ``` ### LLM Client Import (for LLM-powered evaluators) ```typescript // CORRECT - Use @outputai/llm wrapper import { generateText, aiSdk } from '@outputai/llm'; // WRONG - Never call LLM providers directly import OpenAI from 'openai'; ``` ### ES Module Imports All imports MUST use `.js` extension: ```typescript // CORRECT import { BlogContent } from './types.js'; // WRONG - Missing .js extension import { BlogContent } from './types'; ``` ## Basic Structure ```typescript import { evaluator, z, EvaluationBooleanResult } from '@outputai/core'; export const myEvaluator = evaluator( { name: 'my_evaluator', description: 'Description of what this evaluator assesses', inputSchema: z.object( { /* input schema */ } ), fn: async input => { // Evaluation logic return new EvaluationBooleanResult( { value: true, confidence: 0.95 } ); } } ); ``` ## Required Properties ### name (string) Unique identifier for the evaluator. Use `snake_case`. ```typescript name: 'evaluate_content_quality' ``` ### description (string) Human-readable description of what the evaluator assesses. ```typescript description: 'Evaluate the quality and completeness of generated content' ``` ### inputSchema (Zod schema) Schema for validating evaluator input. ```typescript inputSchema: z.object( { content: z.string(), expectedLength: z.number() } ) ``` ### fn (async function) The evaluator execution function. Returns an evaluation result with value and confidence. ```typescript fn: async input => { const isValid = input.content.length >= input.expectedLength; return new EvaluationBooleanResult( { value: isValid, confidence: 0.95 } ); } ``` ## Result Types ### EvaluationBooleanResult Use for pass/fail or true/false evaluations: ```typescript import { EvaluationBooleanResult } from '@outputai/core'; return new EvaluationBooleanResult( { value: true, // boolean result confidence: 0.95, // 0.0 to 1.0 reasoning: 'Optional explanation of the evaluation' } ); ``` ### EvaluationNumberResult Use for numeric scores or ratings: ```typescript import { EvaluationNumberResult } from '@outputai/core'; return new EvaluationNumberResult( { value: 85, // numeric result (e.g., 0-100 score) confidence: 0.85, // 0.0 to 1.0 reasoning: 'Optional explanation of the score' } ); ``` ### EvaluationStringResult Use for categorical or text-based evaluations: ```typescript import { EvaluationStringResult } from '@outputai/core'; return new EvaluationStringResult( { value: 'positive', // string result (e.g., category, sentiment, label) confidence: 0.9, // 0.0 to 1.0 reasoning: 'Optional explanation of the classification' } ); ``` ## Result Properties | Property | Type | Required | Description | |----------|------|----------|-------------| | `value` | `boolean`, `number`, or `string` | Yes | The evaluation result | | `confidence` | `number` (0.0-1.0) | Yes | Confidence in the evaluation | | `reasoning` | `string` | No | Explanation of the evaluation | | `name` | `string` | No | Name for this specific result (useful in dimensions) | | `feedback` | `EvaluationFeedback[]` | No | Array of feedback objects with issues and suggestions | | `dimensions` | `EvaluationResult[]` | No | Nested results for multi-dimensional evaluation | ## Simple Evaluator Examples ### Boolean Evaluator - Content Validation ```typescript import { evaluator, z, EvaluationBooleanResult } from '@outputai/core'; export const evaluateCompleteness = evaluator( { name: 'evaluate_completeness', description: 'Check if content meets minimum length requirements', inputSchema: z.object( { content: z.string(), minLength: z.number().default( 100 ) } ), fn: async ( { content, minLength } ) => { const isComplete = content.length >= minLength; return new EvaluationBooleanResult( { value: isComplete, confidence: 1.0, reasoning: isComplete ? `Content has ${content.length} characters, meets minimum of ${minLength}` : `Content has ${content.length} characters, below minimum of ${minLength}` } ); } } ); ``` ### Boolean Evaluator - Pattern Detection ```typescript import { evaluator, z, EvaluationBooleanResult } from '@outputai/core'; export const evaluateGibberish = evaluator( { name: 'evaluate_gibberish', description: 'Check if a given string is gibberish', inputSchema: z.string(), fn: async content => { const gibberishPatterns = [ 'foo', 'bar', 'lorem', 'ipsum' ]; const isGibberish = gibberishPatterns.some( p => content.toLowerCase().includes( p ) ); return new EvaluationBooleanResult( { value: !isGibberish, confidence: 0.95 } ); } } ); ``` ### Number Evaluator - Quality Score ```typescript import { evaluator, z, EvaluationNumberResult } from '@outputai/core'; export const evaluateReadability = evaluator( { name: 'evaluate_readability', description: 'Calculate readability score based on sentence structure', inputSchema: z.object( { content: z.string() } ), fn: async ( { content } ) => { const sentences = content.split( /[.!?]+/ ).filter( s => s.trim() ); const words = content.split( /\s+/ ).filter( w => w.trim() ); const avgWordsPerSentence = words.length / Math.max( sentences.length, 1 ); // Simple readability score (lower avg words = more readable) const score = Math.max( 0, Math.min( 100, 100 - ( avgWordsPerSentence - 15 ) * 5 ) ); return new EvaluationNumberResult( { value: Math.round( score ), confidence: 0.8, reasoning: `Average ${avgWordsPerSentence.toFixed( 1 )} words per sentence` } ); } } ); ``` ### String Evaluator - Sentiment Classification ```typescript import { evaluator, z, EvaluationStringResult } from '@outputai/core'; export const evaluateSentiment = evaluator( { name: 'evaluate_sentiment', description: 'Classify the sentiment of content', inputSchema: z.object( { content: z.string() } ), fn: async ( { content } ) => { const positiveWords = [ 'great', 'excellent', 'amazing', 'good', 'love' ]; const negativeWords = [ 'bad', 'terrible', 'awful', 'hate', 'poor' ]; const lowerContent = content.toLowerCase(); const positiveCount = positiveWords.filter( w => lowerContent.includes( w ) ).length; const negativeCount = negativeWords.filter( w => lowerContent.includes( w ) ).length; const { sentiment, confidence } = positiveCount > negativeCount ? { sentiment: 'positive', confidence: Math.min( 0.95, 0.6 + positiveCount * 0.1 ) } : negativeCount > positiveCount ? { sentiment: 'negative', confidence: Math.min( 0.95, 0.6 + negativeCount * 0.1 ) } : { sentiment: 'neutral', confidence: 0.7 }; return new EvaluationStringResult( { value: sentiment, confidence, reasoning: `Found ${positiveCount} positive and ${negativeCount} negative indicators` } ); } } ); ``` ## LLM-Powered Evaluator Examples **Note**: Evaluators are self-contained components that don't share schemas across steps, so defining `aiSdk.Output.object()` schemas inline is acceptable here. For workflow steps that share schemas, define them in `types.ts` instead. `generateText` arguments: `prompt`, `promptDir`, `variables`, `tools`, `output`, `toolChoice`, `stopWhen`, `abortSignal`. ### Using generateText with aiSdk.Output.object() for Evaluation ```typescript import { evaluator, z, EvaluationNumberResult } from '@outputai/core'; import { generateText, aiSdk } from '@outputai/llm'; export const evaluateSignalToNoise = evaluator( { name: 'evaluate_signal_to_noise', description: 'Evaluate the signal-to-noise ratio of content', inputSchema: z.object( { title: z.string(), content: z.string() } ), fn: async ( { title, content } ) => { const { output } = await generateText( { prompt: 'signal_noise@v1', // References prompts/signal_noise@v1.prompt variables: { title, content }, output: aiSdk.Output.object( { schema: z.object( { score: z.number().describe( 'Signal-to-noise score 0-100' ) } ) } ) } ); return new EvaluationNumberResult( { value: output.score, confidence: 0.85 } ); } } ); ``` ### LLM Boolean Evaluation ```typescript import { evaluator, z, EvaluationBooleanResult } from '@outputai/core'; import { generateText, aiSdk } from '@outputai/llm'; export const evaluateFactualAccuracy = evaluator( { name: 'evaluate_factual_accuracy', description: 'Check if content contains factual claims that can be verified', inputSchema: z.object( { content: z.string(), topic: z.string() } ), fn: async ( { content, topic } ) => { const { output } = await generateText( { prompt: 'factual_check@v1', variables: { content, topic }, output: aiSdk.Output.object( { schema: z.object( { isFactual: z.boolean().describe( 'Whether content appears factually accurate' ), confidence: z.number().describe( 'Confidence in assessment 0-1' ), issues: z.array( z.string() ).optional().describe( 'Any factual issues found' ) } )