Menu
Scaffold a new EvalAgent plugin (config + implementation + registration).
The review-first prompt is selected by default. You can switch to a direct command or download a local copy.
Read SKILL.md and any companion files shown by SkillsMP before deciding whether to install.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
A direct command skips the review prompt. Inspect the source before running it.
npx skills add https://github.com/thetahealth/mirobody-eval --skill add-eval-agentThe command stays on one line. Scroll horizontally to inspect it before copying.
Prefer a local copy? Download the files currently available to SkillsMP.
Based on SOC occupation classification
Showing SKILL.md
| name | add-eval-agent |
| description | Scaffold a new EvalAgent plugin (config + implementation + registration). |
| argument-hint | ["agent-name"] |
Create a new EvalAgent plugin with all required boilerplate: Pydantic config, implementation class, and plugin registration.
本 skill 仅通过插件扩展点添加功能，严禁修改框架核心逻辑。文件按修改权限分为三级：
| 文件 | 操作 |
|---|---|
evaluator/plugin/eval_agent/<name>_eval_agent.py | 创建新文件（plugin 实现） |
evaluator/plugin/eval_agent/__init__.py | 仅追加: 添加 import + __all__ 条目 + docstring 映射。不得删除/修改已有内容 |
evaluator/core/schema.py 中的 EvalInfo 采用 BeforeValidator 动态分发机制，从 AbstractEvalAgent._params_registry 自动路由到对应插件的配置模型。新增 EvalAgent 无需修改 schema.py。
配置模型直接定义在插件文件中，通过 params_model 参数自动注册：
class MyEvalInfo(BaseModel):
evaluator: Literal["my_eval"] = ...
...
class MyEvalAgent(AbstractEvalAgent, name=, params_model=MyEvalInfo):
...
以下文件为框架核心，任何修改都可能破坏全局功能：
evaluator/core/orchestrator.py — 编排引擎evaluator/core/bench_schema.py — Benchmark 数据模型evaluator/core/interfaces/abstract_*.py — 抽象基类evaluator/utils/*.py — 通用工具层（llm, benchmark_reader, report_reader, agent_inspector, config）benchmark/basic_runner.py — 跑分执行器web/ — Web UI（通过 agent_inspector 自动适配新 plugin）如果你发现需要修改 🔴 文件才能完成需求，请停下来通知用户 — 这通常意味着需求理解有误，或框架需要由维护者升级扩展点。
完成以下步骤后，Web UI / CLI 会自动适配新 plugin：
agent_inspector 从 AbstractEvalAgent._params_registry 自动派生 config map，无需手动维护_display_meta 类属性自定义（可选）Ask the user (via AskUserQuestion) for the following if not provided in $ARGUMENTS:
Plugin name (snake_case, e.g. response_time) — used as:
__init_subclass__ registration nameevaluator: Literal["<name>"] discriminator valueevaluator/plugin/eval_agent/<name>_eval_agent.py<PascalCase>EvalAgentEvaluation logic description — what this evaluator measures and how (LLM-based or deterministic)
Config fields — what parameters the user needs to provide in test case JSON (beyond the standard evaluator, model, threshold)
Before generating code, verify:
evaluator/plugin/eval_agent/__init__.py.Create evaluator/plugin/eval_agent/<name>_eval_agent.py following this template:
"""
<PascalCase>EvalAgent — <中文描述>
注册名称: "<name>"
<详细说明评估逻辑>
"""
import logging
from typing import Literal, Optional
from pydantic import BaseModel, ConfigDict, Field
from evaluator.core.interfaces.abstract_eval_agent import AbstractEvalAgent
from evaluator.core.schema import EvalResult, SessionInfo, TestAgentMemory
logger = logging.getLogger(__name__)
# ============================================================
# 配置模型（与插件实现同文件，通过 params_model 自动注册）
# ============================================================
class <NewEvalInfo>(BaseModel):
"""<中文描述>"""
model_config = ConfigDict(
extra="forbid",
json_schema_extra={"examples": [{"evaluator": "<name>"}]},
)
evaluator: Literal["<name>"] = Field(default="<name>", description="评估器类型")
model: Optional[str] = Field(None, description="LLM 模型")
threshold: float = Field(default=0.7, ge=0.0, le=1.0, description="通过阈值")
# ... 自定义字段
# ============================================================
# 插件实现
# ============================================================
class <PascalCase>EvalAgent(AbstractEvalAgent, name="<name>", params_model=<NewEvalInfo>):
"""<中文一句话描述>"""
# 可选：自定义 Web UI 展示元数据（不声明则使用默认图标/颜色）
# _display_meta = {"color": "#8b5cf6", "features": ["LLM 驱动", "自定义评估"]}
_cost_meta = {
"est_cost_per_case": 0.010, # LLM-based ~$0.01，纯规则填 0
}
def __init__(self, eval_config: <NewEvalInfo>, **kwargs):
super().__init__(eval_config, **kwargs)
self.config: <NewEvalInfo> = eval_config
async def run(
self,
memory_list: list[TestAgentMemory],
session_info: SessionInfo | None = None,
) -> EvalResult:
"""执行评估"""
...
Key patterns to follow (from existing implementations):
semantic_eval_agent.py (SemanticEvalAgent) — uses do_execute(), tracks cost via self._cost, exposes self.model and self.cost propertypreset_answer_eval_agent.py — no LLM, no cost tracking, pure logicmemory_list (each entry has .test_reaction for user action and .target_response for AI response)self.history, self.user_info, self.case_idEvalResult(result="pass"|"fail", score=0.0~1.0, feedback="...", trace=...)Edit evaluator/plugin/eval_agent/__init__.py:
__all__Run the following to verify:
# 插件注册检查
uv run python -c "import evaluator.plugin.eval_agent; from evaluator.core.interfaces.abstract_eval_agent import AbstractEvalAgent; print(AbstractEvalAgent.get_all())"
# Web UI schema 自动发现检查
uv run python -c "from evaluator.utils.agent_inspector import list_eval_agents; print([(a.name, list(a.config_schema.get('properties', {}).keys())) for a in list_eval_agents()])"
# Lint
ruff check evaluator/core/schema.py evaluator/plugin/eval_agent/
ruff format evaluator/core/schema.py evaluator/plugin/eval_agent/
新增 EvalAgent 后，更新以下文档中的评估器列表/表格，保持信息同步：
| 文件 | 更新内容 |
|---|---|
web/guides/develop-eval-agent.md | 「现有评估器」表格追加新行 + 「关键文件」表格追加参考实现 |
web/guides/overview.md | 「核心评估能力」表格追加新行 |
CLAUDE.md | Plugin System 表格中 EvalAgent 行追加新名称 + Key Modules 追加说明 |
README.md | 「已注册插件」表格 EvalAgent 区域追加新行 |
每个文件只需追加一行到已有表格，不要修改其他内容。
新 plugin 默认使用灰色图标和空特性标签。若需自定义 Web UI 展示，在 plugin 类上声明 _display_meta:
class <PascalCase>EvalAgent(AbstractEvalAgent, name="<name>"):
_display_meta = {
"icon": "M9 12.75L11.25 15 15 9.75M21 12a9 9 0 11-18 0 9 9 0 0118 0z", # heroicons SVG path
"color": "#8b5cf6", # CSS 颜色值
"features": ["LLM 驱动", "自定义评估"], # 特性标签
}
费用预估通过 _cost_meta 声明，前端自动读取并用于费用预估显示：
_cost_meta = {
"est_cost_per_case": 0.035, # 单 case 评估费用 (USD)，封装了模型 + 调用模式
}
| 评估器类型 | 参考值 | 说明 |
|---|---|---|
| LLM 单次调用 (semantic) | 0.010 | 1-2 次 LLM 调用 |
| LLM 多次调用 (healthbench) | 0.035 | N 条 rubric × 独立 LLM grading |
| 纯规则 (keyword/preset_answer) | 0 | 无 LLM 调用 |
_display_meta 和 _cost_meta 中的字段均为可选，未指定的使用默认值。
| Name | Config | File | Type |
|---|---|---|---|
semantic | SemanticEvalInfo | semantic_eval_agent.py | LLM-based |
indicator | IndicatorEvalInfo | indicator_eval_agent.py | LLM + API |
keyword | KeywordEvalInfo | keyword_eval_agent.py | Deterministic |
preset_answer | PresetAnswerEvalInfo | preset_answer_eval_agent.py | Deterministic |
healthbench | HealthBenchEvalInfo | healthbench_eval_agent.py | LLM-based (rubric) |
medcalc | MedCalcEvalInfo | medcalc_eval_agent.py | LLM + deterministic |