用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
直接命令不会经过审查 Prompt；运行前请先检查来源。
npx skills add https://github.com/rudironsoni/Synaxis --skill agentic-eval命令会保持在同一行。复制前请横向滚动并检查完整内容。
想先保存到本地？可下载 SkillsMP 当前能够提供的文件。
AI-powered wiki generation for code repositories with commands, agents, and skills
Routes .NET/C# work to domain skills. Loads coding-standards for code paths.
Skill manifest management for dotnet-agent-harness. Tracks skill dependencies, conflicts, version compatibility, and provides validation and resolution tools. Triggers on: skill manifest, dependency resolution, skill compatibility, version conflicts, build manifest, validate dependencies.
基于 SOC 职业分类
正在显示 SKILL.md
| name | agentic-eval |
| category | developer-experience |
| subcategory | cli |
| location | file:///.rulesync/skills/agentic-eval/SKILL.md |
| description | Patterns and techniques for evaluating and improving AI agent outputs. |
| targets | ["*"] |
| license | MIT |
| metadata | {"author":"GitHub, Inc. (derived)","version":"0.0.1"} |
| claudecode | {} |
| opencode | {} |
| codexcli | {"short-description":"Toolkit guidance for agentic-eval"} |
| copilot | {} |
| geminicli | {} |
| antigravity | {} |
Portions derived from github/awesome-copilot (MIT License). Used under MIT License.
Patterns for self-improvement through iterative evaluation and refinement, built using the .NET and
Microsoft.Extensions.AI ecosystem.
Evaluation patterns enable agents to assess and improve their own outputs, moving beyond single-shot generation to iterative refinement loops.
Generate → Evaluate → Critique → Refine → Output
Agent evaluates and improves its own output through self-critique.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text.Json;
using System.Text.Json.Serialization;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Extensions.AI;
namespace DotNetAgentHarness.Evals.Engine;
public class BasicReflection(IChatClient chatClient)
{
public async Task<string> ReflectAndRefineAsync(string task, string[] criteria, int maxIterations = 3, CancellationToken cancellationToken = default)
{
var response = await chatClient.GetResponseAsync($"Complete this task:\n{task}", cancellationToken: cancellationToken);
output = response.Text ?? .Empty;
( i = ; i < maxIterations; i++)
{
prompt = , PASSFAIL;
critiqueResponse = chatClient.GetResponseAsync(
prompt,
ChatOptions { ResponseFormat = ChatResponseFormat.Json },
cancellationToken);
critiqueText = critiqueResponse.Text ?? ;
Dictionary<, CritiqueResult>? critiqueData = ;
{
critiqueData = JsonSerializer.Deserialize<Dictionary<, CritiqueResult>>(critiqueText);
}
(JsonException)
{
critiqueData = Dictionary<, CritiqueResult>();
}
(critiqueData != && critiqueData.Count > && critiqueData.Values.All(c => c.Status == ))
{
output;
}
failed = critiqueData?
.Where(kvp => kvp.Value.Status == )
.ToDictionary(kvp => kvp.Key, kvp => kvp.Value.Feedback) ?? Dictionary<, >();
(failed.Count == )
{
;
}
failedJson = JsonSerializer.Serialize(failed);
refinePrompt = ;
improvedResponse = chatClient.GetResponseAsync(refinePrompt, cancellationToken: cancellationToken);
output = improvedResponse.Text ?? .Empty;
}
output;
}
{
[]
Status { ; ; } = .Empty;
[]
Feedback { ; ; } = .Empty;
}
}
Key insight: Use structured JSON output for reliable parsing of critique results. In .NET, you can also use
IChatClient directly with structured output models to avoid manual deserialization checking.
Separate generation and evaluation into distinct components for clearer responsibilities.
using System;
using System.Collections.Generic;
using System.Text.Json;
using System.Text.Json.Serialization;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Extensions.AI;
namespace DotNetAgentHarness.Evals.Engine;
public class EvaluatorOptimizer(IChatClient chatClient, double scoreThreshold = 0.8)
{
public async Task<string> GenerateAsync(string task, CancellationToken cancellationToken = default)
{
var response = await chatClient.GetResponseAsync($"Complete: {task}", cancellationToken: cancellationToken);
return response.Text ?? string.Empty;
}
public async Task<EvaluationResult> EvaluateAsync(string output, string task, CancellationToken cancellationToken = default)
{
var prompt = $$"""
Evaluate output for task: {{task}}
Output:
{{output}}
Return JSON in this format: { "overall_score": 0.0, "dimensions": { "accuracy": 0.0, "clarity": 0.0 } }
""";
var response = await chatClient.GetResponseAsync(
prompt,
new ChatOptions { ResponseFormat = ChatResponseFormat.Json },
cancellationToken);
var jsonText = response.Text ?? "{}";
try
{
return JsonSerializer.Deserialize<EvaluationResult>(jsonText) ?? new EvaluationResult();
}
catch (JsonException)
{
return new EvaluationResult();
}
}
public async Task<string> OptimizeAsync(string output, EvaluationResult feedback, CancellationToken cancellationToken = default)
{
var feedbackJson = JsonSerializer.Serialize(feedback);
var prompt = $"Improve based on feedback: {feedbackJson}\nOutput: {output}";
var response = await chatClient.GetResponseAsync(prompt, cancellationToken: cancellationToken);
return response.Text ?? string.Empty;
}
public async Task<string> RunAsync(string task, int maxIterations = 3, CancellationToken cancellationToken = default)
{
var output = await GenerateAsync(task, cancellationToken);
for (int i = 0; i < maxIterations; i++)
{
var evaluation = await EvaluateAsync(output, task, cancellationToken);
if (evaluation.OverallScore >= scoreThreshold)
{
break;
}
output = await OptimizeAsync(output, evaluation, cancellationToken);
}
return output;
}
}
public class EvaluationResult
{
[JsonPropertyName("overall_score")]
public double OverallScore { get; set; }
[JsonPropertyName("dimensions")]
public Dictionary<string, double> Dimensions { get; set; } = new();
}
Test-driven refinement loop for code generation.
using System;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Extensions.AI;
namespace DotNetAgentHarness.Evals.Engine;
public class CodeReflector(IChatClient chatClient)
{
public async Task<string> ReflectAndFixAsync(string spec, int maxIterations = 3, CancellationToken cancellationToken = default)
{
var codeResponse = await chatClient.GetResponseAsync($"Write C# code for: {spec}", cancellationToken: cancellationToken);
var code = codeResponse.Text ?? string.Empty;
var testsResponse = await chatClient.GetResponseAsync($"Generate xUnit tests for: {spec}\nCode: {code}", cancellationToken: cancellationToken);
var tests = testsResponse.Text ?? string.Empty;
for (int i = 0; i < maxIterations; i++)
{
var result = await RunTestsAsync(code, tests, cancellationToken);
if (result.Success)
{
return code;
}
var fixResponse = await chatClient.GetResponseAsync($"Fix error: {result.Error}\nCode: {code}", cancellationToken: cancellationToken);
code = fixResponse.Text ?? string.Empty;
}
return code;
}
// Stub for actual test execution
private Task<TestResult> RunTestsAsync(string code, string tests, CancellationToken cancellationToken = default)
{
// In a real implementation, you would compile and run the tests dynamically
return Task.FromResult(new TestResult { Success = false, Error = "Mock test failure" });
}
private class TestResult
{
public bool Success { get; set; }
public string Error { get; set; } = string.Empty;
}
}
Evaluate whether output achieves the expected result.
public async Task<string> EvaluateOutcomeAsync(string task, string output, string expected, CancellationToken cancellationToken = default)
{
var response = await chatClient.GetResponseAsync(
$"Does output achieve expected outcome? Task: {task}, Expected: {expected}, Output: {output}",
cancellationToken: cancellationToken
);
return response.Text ?? string.Empty;
}
Use LLM to compare and rank outputs.
public async Task<string> LlmJudgeAsync(string outputA, string outputB, string criteria, CancellationToken cancellationToken = default)
{
var response = await chatClient.GetResponseAsync(
$"Compare outputs A and B for {criteria}. Which is better and why?\n\nOutput A:\n{outputA}\n\nOutput B:\n{outputB}",
cancellationToken: cancellationToken
);
return response.Text ?? string.Empty;
}
Score outputs against weighted dimensions.
public class RubricDimension
{
public double Weight { get; set; }
}
public async Task<double> EvaluateWithRubricAsync(string output, Dictionary<string, RubricDimension> rubric, CancellationToken cancellationToken = default)
{
var dimensions = string.Join(", ", rubric.Keys);
var prompt = $"Rate 1-5 for each dimension: {dimensions}\nOutput: {output}\n\nReturn ONLY a JSON dictionary where keys are dimensions and values are numbers.";
var response = await chatClient.GetResponseAsync(
prompt,
new ChatOptions { ResponseFormat = ChatResponseFormat.Json },
cancellationToken);
var jsonText = response.Text ?? "{}";
Dictionary<string, double> scores;
try
{
scores = JsonSerializer.Deserialize<Dictionary<string, double>>(jsonText) ?? new Dictionary<string, double>();
}
catch (JsonException)
{
scores = new Dictionary<string, double>();
}
double totalScore = 0;
foreach (var dimension in rubric.Keys)
{
if (scores.TryGetValue(dimension, out var score))
{
totalScore += score * rubric[dimension].Weight;
}
}
return totalScore / 5.0; // Normalize
}
| Practice | Rationale |
|---|---|
| Clear criteria | Define specific, measurable evaluation criteria upfront |
| Iteration limits | Set max iterations (3-5) to prevent infinite loops |
| Convergence check | Stop if output score isn't improving between iterations |
| Log history | Keep full trajectory for debugging and analysis |
| Structured output | Use JSON for reliable parsing of evaluation results |
GenerateAsync()EvaluateAsync() with structured outputOptimizeAsync()