Run the 6 Wonder Toys evals against a Rosetta Stone test project. Phoenix path runs the built-in `npm run evals` script. AX path ensures the stable space-level rosetta-e2e-* evaluators exist (creating only missing ones from the templates in evals/README.md), then creates and triggers a per-run eval task scoped to the project. Part of the rosetta-test e2e flow; can also be invoked standalone.
Instalar com Codex ou Claude Copie este prompt, cole no Codex, Claude ou outro assistente e deixe que ele revise a página da skill e instale para você.
Um comando direto ignora o prompt de revisão. Verifique a origem antes de executá-lo.
Instruções da origem · Visualização somente leitura
name
rosetta-test-evals
description
Run the 6 Wonder Toys evals against a Rosetta Stone test project. Phoenix path runs the built-in `npm run evals` script. AX path ensures the stable space-level rosetta-e2e-* evaluators exist (creating only missing ones from the templates in evals/README.md), then creates and triggers a per-run eval task scoped to the project. Part of the rosetta-test e2e flow; can also be invoked standalone.
Rosetta Test — Evals Phase
Inputs
PROJECT_DIR — absolute path to <platform>/<framework>/
PROJECT_NAME — the test project name
PLATFORM — ax or phoenix
For AX: ARIZE_SPACE_ID already sourced from .env.local
Phoenix path
Phoenix evals are already programmatic via the shared harness — just run them with the env overlay loaded:
(
set -a
source"$PROJECT_DIR/.env.local"source"$PROJECT_DIR/.env.test-local"set +a
cd"$PROJECT_DIR" && npm run evals
)
The harness (evals/run-phoenix-evals.ts) fetches root spans from the project, runs all 6 evaluators, and writes results back as span annotations. It uses PHOENIX_PROJECT_NAME from the env, so the overlay routes it to the test project automatically.
Done — proceed to verify.
AX path
AX has no programmatic eval runner of its own — the rosetta-test flow drives the ax CLI directly.
1. Define the stable evaluator set
These six evaluators live at space level and are reused across every rosetta-e2e run. Names use a rosetta-e2e- prefix so they're easy to find in the UI and never collide with the user's other evaluators.
Build a map {name → id} of the ones that already exist. Anything missing from the table above gets created in step 3.
3. Create the missing LLM evaluators
For each missing LLM evaluator, use ax evaluators create with the prompt template, classification choices, and trace granularity from evals/README.md. Consult the arize-evaluator skill for the exact flag combinations and AI integration ID resolution.
Use Claude Sonnet via an Anthropic AI integration if one exists in the space; otherwise default to GPT-4o via OpenAI. List integrations:
ax ai-integrations list --space "$ARIZE_SPACE_ID" --output json
4. Handle the code evaluators
The ax evaluators create CLI only supports LLM (template-based) evaluators. The two code evaluators (rosetta-e2e-image-url-correctness, rosetta-e2e-tool-call-count) must be created once manually via the AX console using the Python code in evals/README.md §§ Eval 5 and Eval 6 — they then persist across runs.
On a run where one or both code evaluators are missing from the space, print a clear warning:
WARNING: code evaluator "<name>" not found in space.
Create it once via the AX console using the Python code in evals/README.md.
This run will proceed with the remaining N evaluators only.
Continue with whatever evaluators do exist. The verify phase will receive the actual list and only check coverage for those.
5. Build the column mappings
Per evals/README.md, the LLM evaluators map prompt template variables to span attributes. The mappings are evaluator-specific:
