verify-samples-tool

# verify-samples Tool The `verify-samples` project (`dotnet/eng/verify-samples/`) is an automated tool that runs sample projects and verifies their output using deterministic checks and AI-powered verification. ## Running verify-samples **Important:** By default, samples must be pre-built before running verify-samples. Build the solution first, or pass `--build` to build samples during the run: ```bash cd dotnet dotnet build agent-framework-dotnet.slnx -f net10.0 ``` Then run verify-samples: ```bash # Run all samples across all categories dotnet run --project eng/verify-samples -- --log results.log --csv results.csv # Run a specific category dotnet run --project eng/verify-samples -- --category 02-agents --log results.log # Run specific samples by name dotnet run --project eng/verify-samples -- Agent_Step02_StructuredOutput Agent_Step09_AsFunctionTool # Control parallelism (default 8) dotnet run --project eng/verify-samples -- --parallel 8 --log results.log # Build samples during run (skips the need for a prior build step) # This may cause build conflicts as multiple samples are built in parallel, so use with caution dotnet run --project eng/verify-samples -- --build --log results.log # Combine options dotnet run --project eng/verify-samples -- --category 03-workflows --parallel 4 --log results.log --csv results.csv --md results.md ``` ### Required Environment Variables The tool itself needs: - `AZURE_OPENAI_ENDPOINT` — for the AI verification agent - `AZURE_OPENAI_DEPLOYMENT_NAME` (optional, defaults to `gpt-5-mini`) Individual samples require their own env vars (for example, `FOUNDRY_PROJECT_ENDPOINT`). The tool automatically checks and skips samples with missing env vars. ### Output Files - `--log results.log` — detailed per-sample log with stdout/stderr, AI reasoning, and a summary - `--csv results.csv` — tabular summary with Sample, ProjectPath, Status, FailedChecks, and Failures columns - `--md results.md` — Markdown summary with results table and collapsible failure details (suitable for GitHub PR comments) ## Sample Categories Definitions are in the `dotnet/eng/verify-samples/` directory: | Category | Config File | Registered Key | |----------|-------------|----------------| | 01-get-started | `GetStartedSamples.cs` | `01-get-started` | | 02-agents | `AgentsSamples.cs` | `02-agents` | | 03-workflows | `WorkflowSamples.cs` | `03-workflows` | Categories are registered in `VerifyOptions.cs` in the `s_sampleSets` dictionary. ## SampleDefinition Properties Each sample is defined as a `SampleDefinition` in the appropriate config file. Key properties: ```csharp new SampleDefinition { // Required: Display name for the sample Name = "Agent_Step02_StructuredOutput", // Required: Relative path from dotnet/ to the sample project directory ProjectPath = "samples/02-agents/Agents/Agent_Step02_StructuredOutput", // Environment variables the sample requires (throws if missing) RequiredEnvironmentVariables = ["AZURE_OPENAI_ENDPOINT"], // Environment variables with defaults that would prompt on console if unset OptionalEnvironmentVariables = ["AZURE_OPENAI_DEPLOYMENT_NAME"], // Skip this sample with a reason (for structural issues only) SkipReason = null, // or "Requires external service X." // Deterministic checks: substrings that must appear in stdout MustContain = ["=== Section Header ==="], // Substrings that must NOT appear in stdout MustNotContain = [], // If true, only MustContain checks are used (no AI verification) IsDeterministic = false, // AI verification: natural-language descriptions of expected output // Each entry describes one aspect to verify independently ExpectedOutputDescription = [ "The output should show structured person information with Name, Age, and Occupation fields.", "The output should not contain error messages or stack traces.", ], // Stdin inputs to feed to the sample (for interactive samples) Inputs = ["Y", "Y", "Y"], // Delay between stdin inputs in ms (default 2000, increase for LLM calls between inputs) InputDelayMs = 3000, } ``` ## How to Add a New Sample Definition 1. **Check the sample's Program.cs** to understand: - What environment variables it reads (look for `GetEnvironmentVariable`) - Whether it needs stdin input (look for `Console.ReadLine`, `Application.GetInput`) - Whether it has an external loop (look for `EXIT` patterns in YAML workflows) - What output it produces (section headers, markers, expected behavior) - Whether it exits on its own or runs as a server 2. **Choose the right verification strategy:** - **Deterministic** (`IsDeterministic = true`): Use `MustContain` for samples with fixed output strings. No AI verification. - **AI-verified** (default): Use `ExpectedOutputDescription` with semantic descriptions. Write expectations that are flexible enough for non-deterministic LLM output. - **Both**: Use `MustContain` for fixed markers AND `ExpectedOutputDescription` for LLM-generated content. 3. **Set `SkipReason` only for structural issues:** - Web servers that don't exit - Multi-process client/server architectures - Samples requiring external infrastructure (MCP servers you can't reach, Docker, etc.) - Do NOT skip for missing env vars — the tool checks those dynamically. 4. **For interactive samples, provide `Inputs`:** - Samples using `Application.GetInput(args)` need one initial input - Samples with `Console.ReadLine()` approval loops need `"Y"` inputs - YAML workflows with `externalLoop` need `"EXIT"` as the last input - Set `InputDelayMs` to 3000-8000ms for samples with LLM calls between inputs 5. **Add the definition** to the appropriate config file (e.g., `AgentsSamples.cs`) in the `All` list. 6. **Register new categories** (if needed) in `VerifyOptions.cs` `s_sampleSets` dictionary. ### Writing Good ExpectedOutputDescription - Write descriptions that are **semantically flexible** — LLM output varies between runs - Each array entry should describe **one independent aspect** to verify - Always include `"The output should not contain error messages or stack traces."` as the last entry - Avoid exact wording expectations — use "should mention", "should contain information about", "should show" - Bad: `"The output should say 'The weather in Amsterdam is cloudy with a high of 15°C'"` - Good: `"The output should contain weather information about Amsterdam mentioning cloudy weather with a high of 15°C."` ### Example: Simple LLM Sample ```csharp new SampleDefinition { Name = "Agent_With_AzureOpenAIChatCompletion", ProjectPath = "samples/02-agents/AgentProviders/azure/Agent_With_AzureOpenAIChatCompletion", RequiredEnvironmentVariables = ["AZURE_OPENAI_ENDPOINT"], OptionalEnvironmentVariables = ["AZURE_OPENAI_DEPLOYMENT_NAME"], ExpectedOutputDescription = [ "The output should contain a joke about a pirate.", "The output should not contain error messages or stack traces.", ], }, ``` ### Example: Deterministic Sample ```csharp new SampleDefinition { Name = "Workflow_Visualization", ProjectPath = "samples/03-workflows/Visualization", IsDeterministic = true, MustContain = ["Generating workflow visualization...", "Mermaid string:", "DiGraph string:"], ExpectedOutputDescription = ["The output should show workflow visualization in Mermaid and DiGraph formats."], }, ``` ### Example: Interactive Sample with Approval Loop ```csharp new SampleDefinition { Name = "FoundryAgent_Hosted_MCP", ProjectPath = "samples/02-agents/ModelContextProtocol/FoundryAgent_Hosted_MCP", RequiredEnvironmentVariables = ["FOUNDRY_PROJECT_ENDPOINT"], OptionalEnvironmentVariables = ["FOUNDRY_MODEL"], Inputs = ["Y", "Y", "Y", "Y", "Y"], InputDelayMs = 5000, ExpectedOutputDescription = ["The output should show an agent using the Microsoft Learn MCP tool with approval prompts."], }, ``` ### Example: Declarative Workflow with External Loop ```csharp new SampleDefinition { Name = "Workflow_Declarative_FunctionTools", ProjectPath = "samples/03-workflows/Declarative/FunctionTools", RequiredEnvironmentVariables = ["FOUNDRY_PROJECT_ENDPOINT"], OptionalEnvironmentVariables = ["FOUNDRY_MODEL"], Inputs = ["What are today's specials?", "EXIT"], InputDelayMs = 8000, ExpectedOutputDescription = ["The output should show a workflow calling function tools to answer a question about restaurant specials."], }, ``` ### Example: Skipped Sample ```csharp new SampleDefinition { Name = "Agent_MCP_Server", ProjectPath = "samples/02-agents/ModelContextProtocol/Agent_MCP_Server", RequiredEnvironmentVariables = ["AZURE_OPENAI_ENDPOINT"], OptionalEnvironmentVariables = ["AZURE_OPENAI_DEPLOYMENT_NAME"], SkipReason = "Runs as an MCP stdio server that does not exit on its own.", }, ```