AI-Eval-Skills

Adds cost (input + output tokens × model price) and latency (p50, p95) to eval results, plots model options on a Pareto frontier, and produces a quality-per-dollar composite score so production model selection is grounded in trade-offs, not just quality. Use when: model comparison, cost-aware evals, latency budget, quality-per-dollar, Pareto frontier, model selection, eval economics, picking a model for production.

Generates eval test cases for AI agents from an eval suite plan (output of /eval-suite-planner) or a plain-English agent description. Supports both single-response and conversation (multi-turn) evaluation modes. Outputs a test set table, a CSV file in the Copilot Studio import format (single-response only - used here as the primary worked example; trivially adaptable to any harness that accepts tabular test cases), and a docx report for human review.

Analyzes AI agent evaluation results - primarily from Copilot Studio (the worked example here, via its CSV export) but also from custom harnesses or any evaluator that produces per-case pass/fail rows - using Microsoft's Triage & Improvement Playbook. Returns a SHIP / ITERATE / BLOCK verdict with root cause classification, diagnostic triage, prioritized remediation, and pattern analysis.

Produces a concrete eval suite plan for AI agents - grounded in Microsoft's Eval Scenario Library and MS Learn agent evaluation guidance (Copilot Studio is the primary worked example, but the plan is platform-agnostic and adapts to any agent harness). Outputs scenario types, evaluation methods, quality signals, thresholds, and priority order - before any test cases are generated or evals are run.

Use this skill when AI agent evaluations have come back and the user needs to interpret scores, diagnose root causes of underperforming test cases, find remediation steps, or analyze patterns to improve their agent. Works against any agent platform - Copilot Studio is the primary worked example here, but the triage framework applies equally to custom harnesses, LangChain/LangGraph, AutoGen, Semantic Kernel, OpenAI Assistants, and other agent runtimes. Always use this skill when the user mentions: "eval failed", "why did this fail", "triage", "diagnose failure", "low pass rate", "fix evaluation results", "not passing", "failing test cases", "evaluation results", "improve my eval scores", or any situation where eval scores need interpretation and action.

Helps users discover and install agent skills when they ask questions like "how do I do X", "find a skill for X", "is there a skill that can...", or express interest in extending capabilities. This skill should be used when the user is looking for functionality that might exist as an installable skill.

Evaluation pattern for agents that call tools / functions - measures tool selection accuracy, argument correctness, recovery from tool errors, and avoidance of unnecessary tool calls. Pairs with eval-suite-planner. Use when: tool-use eval, function-calling eval, agent tool selection, tool invocation accuracy, tool argument validation, tool error recovery, agent tools, function-calling agent, MCP eval.