validate-eval

Validate that a NEW eval is correct and runnable by running it and iterating in a loop until green — fixing infra/config/task bugs (not the model's score), with opt-in hands-off and unlimited/self-healing modes; invoke when authoring or vetting a task before setting `validated: true`.

跳到安装查看文件

仓库: kubernetes-sigs/devops-bench
最近来源活动: 2026年8月28日 23:02
检测到的 SKILL.md 语言: 英语
星标: 35
Forks: 17

安装方式

默认使用会先检查来源的 Prompt；你也可以切换为直接命令，或下载本地副本。

用 AI 助手安装

复制这段 Prompt，发给你正在使用的 AI 助手。

Follow https://skillsmp.com/skill-install/prompt.md to install only the Agent Skill at https://github.com/kubernetes-sigs/devops-bench/tree/main/.agents/skills/validate-eval.

直接命令不会经过审查 Prompt；运行前请先检查来源。

npx skills add https://github.com/kubernetes-sigs/devops-bench --skill validate-eval

命令会保持在同一行。复制前请横向滚动并检查完整内容。

想先保存到本地？可下载 SkillsMP 当前能够提供的文件。

检查来源文件

决定是否安装前，请先阅读 SKILL.md，以及 SkillsMP 当前展示的配套文件。

SKILL.md

来源说明 · 只读预览

name	validate-eval
description	Validate that a NEW eval is correct and runnable by running it and iterating in a loop until green — fixing infra/config/task bugs (not the model's score), with opt-in hands-off and unlimited/self-healing modes; invoke when authoring or vetting a task before setting `validated: true`.

返回搜索返回安装

diagnose-eval-failure

Explain WHY a model scored low on an eval — read the judge's reasons and the agent's trajectory, compare against the task rubric, and produce a rationale (capability gap vs. task/rubric problem). Distinct from infrastructure failures. Invoke when someone asks "why did the model fail this task?", "why is OutcomeValidity low?", "explain this low score", or "did the agent actually do the work?".

2026-10-09

task-review

Use when the user adds or changes a benchmark TASK and asks to review it — e.g. "review this new task", "is this task parallel-safe", "review this task.yaml + stack", "will this task collide under the matrix", "check my new tf/prebuilt stack". Reviews a newly added/changed `tasks/**/task.yaml` together with its TF stack across schema/metadata, spec parsing, outcome rubric, parallel-safety (the emphasis), infra leak prevention, placeholders, and flags — returning ranked findings with severity + file:line evidence + a concrete fix. Review-only: static analysis plus unit tests / spec-parse checks; it NEVER provisions infra or runs an eval. For a review of harness/library CODE, use the sibling `devops-bench-review` skill instead.

2026-09-19

docs-sync

Keep the docs current after a code change — map changed code areas to the docs that describe them, update those docs in place, and resolve known-issues rows the code has verifiably fixed. Invoke after editing the pipeline, adding a model/agent/task/metric, moving a directory, or whenever someone asks to "sync the docs", "update the docs for this change", or "do the docs still match the code?".

2026-08-28

查看这个仓库的全部 Skills

validate-eval

来源信息

安装方式

用 AI 助手安装

检查来源文件

同仓库更多 Skills

用 AI 助手安装

同仓库更多 Skills