一键在 Manus 中运行任何 Skill

designing-llm-evals

星标1

分支0

更新时间2026年6月17日 20:23

Use when building or reviewing an evaluation for an LLM feature — assembling a representative test set, choosing pass criteria (exact match, programmatic checks, rubric, or LLM-as-judge), and catching regressions. Use when asking "how do I know this prompt or model change is better?"

安装

用 Codex 或 Claude 帮你安装复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。

在 Manus 中运行

来源

meaganewaller

meaganewaller/rosetta

打开 GitHub 仓库查看创作者相关仓库

下载

在 Manus 中运行

Designing LLM evals

If you can't measure it, you're tuning by vibes. Decide how you'll know the output is good before you start changing prompts or models.

Build a real test set

Draw cases from real inputs, including edge cases and known past failures. A handful of representative cases beats a thousand synthetic near-duplicates.
Keep a holdout you don't tune against, so you can detect overfitting to the eval.
Label the expected behavior (exact answer, or what "good" means) per case.

Choose the cheapest valid criterion

In order of preference — use the strongest that fits the task:

Exact / structural match — classification, extraction, JSON shape. Cheap and objective.
Programmatic checks — does it compile, parse, satisfy invariants, stay under length?
Rubric — score against explicit criteria for open-ended output.
LLM-as-judge — last resort for open-ended quality. Calibrate it against human labels and watch for judge bias (favoring length, position, its own style).

Score honestly

Report the pass rate and the specific failures, broken down by category — not a single aggregate number that hides which class of input is broken.
Re-run the eval on every prompt or model change (regression). Pin the test set.
Offline eval is necessary but not sufficient: also sample and review production outputs.

Anti-patterns to flag

Vibes-only evaluation; tiny non-representative sets.
Judging on the same examples you tuned on (overfit).
One aggregate score masking a category that regressed.
An uncalibrated LLM-judge treated as ground truth.

Out of scope: which model to use, pricing, and batch/eval APIs are provider-specific — this skill is about eval design, not the runner.

同仓库更多 Skills

同仓库

agent-memory-patterns

meaganewaller/rosetta

Use when designing memory or persistent context for an agent — deciding what to persist vs. recompute, how to retrieve only what's relevant, and how to summarize/compact history to fit the context window. Use when an agent forgets, bloats its context, or repeats work.

2026-06-171

configure-rosetta

meaganewaller/rosetta

Interactive installer for Rosetta. Guides users through selecting and installing skills and language-specific rules at user-level or project-level, verifies paths, and optionally optimizes installed files.

2026-06-171

continuous-learning

meaganewaller/rosetta

Instinct-based learning system that observes sessions via hooks, creates atomic instincts with confidence scoring, and evolves them into skills, commands, and agents.

2026-06-171

effective-prompting

meaganewaller/rosetta

Use when writing or reviewing a prompt for an LLM — making it specific, structured, and testable (clear task, role/context, examples, an explicit output contract, decomposition, room to reason) and avoiding the anti-patterns that make prompts brittle. Provider-agnostic.

2026-06-171

iterative-retrieval

meaganewaller/rosetta

Pattern for progressively refining context retrieval to solve the subagent context problem.

2026-06-171

plugin-authoring

meaganewaller/rosetta

Use when creating or structuring a Claude Code plugin for this marketplace — the on-disk layout, the plugin.json manifest fields, the frontmatter for skills/commands/agents, and the authoring rules (no absolute paths, least-privilege tools, one coherent capability).

2026-06-171

name	designing-llm-evals
description	Use when building or reviewing an evaluation for an LLM feature — assembling a representative test set, choosing pass criteria (exact match, programmatic checks, rubric, or LLM-as-judge), and catching regressions. Use when asking "how do I know this prompt or model change is better?"

Designing LLM evals

If you can't measure it, you're tuning by vibes. Decide how you'll know the output is good before you start changing prompts or models.

Build a real test set

Draw cases from real inputs, including edge cases and known past failures. A handful of representative cases beats a thousand synthetic near-duplicates.
Keep a holdout you don't tune against, so you can detect overfitting to the eval.
Label the expected behavior (exact answer, or what "good" means) per case.

Choose the cheapest valid criterion

In order of preference — use the strongest that fits the task:

Exact / structural match — classification, extraction, JSON shape. Cheap and objective.
Programmatic checks — does it compile, parse, satisfy invariants, stay under length?
Rubric — score against explicit criteria for open-ended output.
LLM-as-judge — last resort for open-ended quality. Calibrate it against human labels and watch for judge bias (favoring length, position, its own style).

Score honestly

Report the pass rate and the specific failures, broken down by category — not a single aggregate number that hides which class of input is broken.
Re-run the eval on every prompt or model change (regression). Pin the test set.
Offline eval is necessary but not sufficient: also sample and review production outputs.

Anti-patterns to flag

Vibes-only evaluation; tiny non-representative sets.
Judging on the same examples you tuned on (overfit).
One aggregate score masking a category that regressed.
An uncalibrated LLM-judge treated as ground truth.

Out of scope: which model to use, pricing, and batch/eval APIs are provider-specific — this skill is about eval design, not the runner.