一键在 Manus 中运行任何 Skill

scaffold-agentic-evals

星标2

分支0

更新时间2026年7月13日 22:42

Use when setting up a project-local eval harness for prompts, agents, judges, RAG, tools, or other LLM behavior, especially when the default should be free/OSS promptfoo with repo-owned artifacts.

安装

用 Codex 或 Claude 帮你安装复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。

在 Manus 中运行

来源

jwilger

jwilger/ai-plugins

打开 GitHub 仓库查看创作者相关仓库

下载

在 Manus 中运行

Scaffold Agentic Evals

Use this skill to add a local-first eval harness.

Default Stack

Load references/scaffold.md.

Default to promptfoo for OSS, CI-friendly behavior evals.
Ensure promptfoo@0.121.18 is installed on PATH for both local eval commands and optional MCP use. Prefer the consuming project's existing tool provisioning: if it has flake.nix and nixpkgs provides a compatible pkgs.promptfoo, add that package there so updates flow through the flake lockfile; otherwise use the project-local package manager sandbox.
Store configs under evals/promptfoo/.
Store fixtures under evals/fixtures/.
Store generated artifacts under evals/out/.
Generate JSON, HTML, and JUnit outputs.
Build repo-owned static reports under site/evals/.
Do not rely on hosted sharing as the durable record.
Pin promptfoo once the harness is committed, and disable prompt response caching for provider-backed behavior evidence.
For coding-agent plugins, prefer Promptfoo's native harness providers such as openai:codex-sdk and anthropic:claude-agent-sdk before writing custom wrappers.
Separate canaries that prove plugin/skill loading from natural behavior prompts that measure whether the system chooses the right guidance unaided.
Treat Promptfoo's MCP server as optional agent tooling for validation, focused runs, and result inspection. Treat Promptfoo's mcp provider as a separate choice for evaluating MCP servers as systems under test.

Trusted And Untrusted Runs

Run secret-free config checks on pull requests.
Skip provider-backed live evals on untrusted PRs.
Run live evals on trusted scheduled, manual, or main-branch workflows when required secrets are present.
Upload artifacts on every CI run that executes evals.
For this marketplace, point users to eval-case-reporter when they find a behavior that should become a future fixture.

同仓库更多 Skills

同仓库

final-review

jwilger/ai-plugins

Use when preparing local changes, a branch, pull request, merge request, or merge-to-main for final review before publishing, opening a PR, merging, or claiming readiness.

2026-07-142

test-driven-development

jwilger/ai-plugins

Use when implementing any feature, bugfix, behavior change, or refactor before writing production code.

2026-07-142

advisor

jwilger/ai-plugins

Use when the user asks to shape fuzzy planning, product/design/engineering tradeoffs, scope/spec/ticket plans, or says "challenge this/help me think"; delegate to an advisor subagent. Good for rough ideas, second opinions, load-bearing decisions, or scope cuts. Skip scoped implementation, narrow bugs, reviews, and "just build" requests.

2026-07-132

babysit-pr

jwilger/ai-plugins

Use when a pull request or merge request needs to be driven to merge — watching CI, responding to review feedback, and merging once green and approved, across GitHub, Forgejo, or GitLab.

2026-07-132

tiber

jwilger/ai-plugins

Use when the user wants repository task tracking, shared agent task state, cross-worktree coordination, tiber setup/install/scaffold guidance, or task create/list/show/prioritize/validate/close workflows. Plugin install and session start are non-mutating; setup integration starts with dry-run previews.

2026-07-132

new-task

jwilger/ai-plugins

Use when the user asks to add, capture, file, or record a new repository task or backlog ticket through Tiber from chat.

2026-07-082

name	scaffold-agentic-evals
description	Use when setting up a project-local eval harness for prompts, agents, judges, RAG, tools, or other LLM behavior, especially when the default should be free/OSS promptfoo with repo-owned artifacts.

Scaffold Agentic Evals

Use this skill to add a local-first eval harness.

Default Stack

Load references/scaffold.md.

Default to promptfoo for OSS, CI-friendly behavior evals.
Ensure promptfoo@0.121.18 is installed on PATH for both local eval commands and optional MCP use. Prefer the consuming project's existing tool provisioning: if it has flake.nix and nixpkgs provides a compatible pkgs.promptfoo, add that package there so updates flow through the flake lockfile; otherwise use the project-local package manager sandbox.
Store configs under evals/promptfoo/.
Store fixtures under evals/fixtures/.
Store generated artifacts under evals/out/.
Generate JSON, HTML, and JUnit outputs.
Build repo-owned static reports under site/evals/.
Do not rely on hosted sharing as the durable record.
Pin promptfoo once the harness is committed, and disable prompt response caching for provider-backed behavior evidence.
For coding-agent plugins, prefer Promptfoo's native harness providers such as openai:codex-sdk and anthropic:claude-agent-sdk before writing custom wrappers.
Separate canaries that prove plugin/skill loading from natural behavior prompts that measure whether the system chooses the right guidance unaided.
Treat Promptfoo's MCP server as optional agent tooling for validation, focused runs, and result inspection. Treat Promptfoo's mcp provider as a separate choice for evaluating MCP servers as systems under test.

Trusted And Untrusted Runs

Run secret-free config checks on pull requests.
Skip provider-backed live evals on untrusted PRs.
Run live evals on trusted scheduled, manual, or main-branch workflows when required secrets are present.
Upload artifacts on every CI run that executes evals.
For this marketplace, point users to eval-case-reporter when they find a behavior that should become a future fixture.