원클릭으로 Manus에서 모든 스킬 실행

scaffold-agentic-evals

스타2

포크0

업데이트2026년 7월 13일 22:42

Use when setting up a project-local eval harness for prompts, agents, judges, RAG, tools, or other LLM behavior, especially when the default should be free/OSS promptfoo with repo-owned artifacts.

설치

Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.

Manus에서 실행

출처

jwilger

jwilger/ai-plugins

GitHub 저장소 열기 Creator 저장소 보기

다운로드

Manus에서 실행

Scaffold Agentic Evals

Use this skill to add a local-first eval harness.

Default Stack

Load references/scaffold.md.

Default to promptfoo for OSS, CI-friendly behavior evals.
Ensure promptfoo@0.121.18 is installed on PATH for both local eval commands and optional MCP use. Prefer the consuming project's existing tool provisioning: if it has flake.nix and nixpkgs provides a compatible pkgs.promptfoo, add that package there so updates flow through the flake lockfile; otherwise use the project-local package manager sandbox.
Store configs under evals/promptfoo/.
Store fixtures under evals/fixtures/.
Store generated artifacts under evals/out/.
Generate JSON, HTML, and JUnit outputs.
Build repo-owned static reports under site/evals/.
Do not rely on hosted sharing as the durable record.
Pin promptfoo once the harness is committed, and disable prompt response caching for provider-backed behavior evidence.
For coding-agent plugins, prefer Promptfoo's native harness providers such as openai:codex-sdk and anthropic:claude-agent-sdk before writing custom wrappers.
Separate canaries that prove plugin/skill loading from natural behavior prompts that measure whether the system chooses the right guidance unaided.
Treat Promptfoo's MCP server as optional agent tooling for validation, focused runs, and result inspection. Treat Promptfoo's mcp provider as a separate choice for evaluating MCP servers as systems under test.

Trusted And Untrusted Runs

Run secret-free config checks on pull requests.
Skip provider-backed live evals on untrusted PRs.
Run live evals on trusted scheduled, manual, or main-branch workflows when required secrets are present.
Upload artifacts on every CI run that executes evals.
For this marketplace, point users to eval-case-reporter when they find a behavior that should become a future fixture.

이 저장소의 다른 Skills

같은 저장소

final-review

jwilger/ai-plugins

Use when preparing local changes, a branch, pull request, merge request, or merge-to-main for final review before publishing, opening a PR, merging, or claiming readiness.

2026-07-142

test-driven-development

jwilger/ai-plugins

Use when implementing any feature, bugfix, behavior change, or refactor before writing production code.

2026-07-142

advisor

jwilger/ai-plugins

Use when the user asks to shape fuzzy planning, product/design/engineering tradeoffs, scope/spec/ticket plans, or says "challenge this/help me think"; delegate to an advisor subagent. Good for rough ideas, second opinions, load-bearing decisions, or scope cuts. Skip scoped implementation, narrow bugs, reviews, and "just build" requests.

2026-07-132

babysit-pr

jwilger/ai-plugins

Use when a pull request or merge request needs to be driven to merge — watching CI, responding to review feedback, and merging once green and approved, across GitHub, Forgejo, or GitLab.

2026-07-132

tiber

jwilger/ai-plugins

Use when the user wants repository task tracking, shared agent task state, cross-worktree coordination, tiber setup/install/scaffold guidance, or task create/list/show/prioritize/validate/close workflows. Plugin install and session start are non-mutating; setup integration starts with dry-run previews.

2026-07-132

new-task

jwilger/ai-plugins

Use when the user asks to add, capture, file, or record a new repository task or backlog ticket through Tiber from chat.

2026-07-082

name	scaffold-agentic-evals
description	Use when setting up a project-local eval harness for prompts, agents, judges, RAG, tools, or other LLM behavior, especially when the default should be free/OSS promptfoo with repo-owned artifacts.

Scaffold Agentic Evals

Use this skill to add a local-first eval harness.

Default Stack

Load references/scaffold.md.

Default to promptfoo for OSS, CI-friendly behavior evals.
Ensure promptfoo@0.121.18 is installed on PATH for both local eval commands and optional MCP use. Prefer the consuming project's existing tool provisioning: if it has flake.nix and nixpkgs provides a compatible pkgs.promptfoo, add that package there so updates flow through the flake lockfile; otherwise use the project-local package manager sandbox.
Store configs under evals/promptfoo/.
Store fixtures under evals/fixtures/.
Store generated artifacts under evals/out/.
Generate JSON, HTML, and JUnit outputs.
Build repo-owned static reports under site/evals/.
Do not rely on hosted sharing as the durable record.
Pin promptfoo once the harness is committed, and disable prompt response caching for provider-backed behavior evidence.
For coding-agent plugins, prefer Promptfoo's native harness providers such as openai:codex-sdk and anthropic:claude-agent-sdk before writing custom wrappers.
Separate canaries that prove plugin/skill loading from natural behavior prompts that measure whether the system chooses the right guidance unaided.
Treat Promptfoo's MCP server as optional agent tooling for validation, focused runs, and result inspection. Treat Promptfoo's mcp provider as a separate choice for evaluating MCP servers as systems under test.

Trusted And Untrusted Runs

Run secret-free config checks on pull requests.
Skip provider-backed live evals on untrusted PRs.
Run live evals on trusted scheduled, manual, or main-branch workflows when required secrets are present.
Upload artifacts on every CI run that executes evals.
For this marketplace, point users to eval-case-reporter when they find a behavior that should become a future fixture.