Run any Skill in Manus with one click

rosetta-test-traces

Stars4

Forks0

UpdatedMay 18, 2026 at 19:17

Run the 25 synthetic Wonder Toys requests against a Rosetta Stone framework to generate traces in an isolated Arize AX or Phoenix project. Loads the .env.test-local overlay so the project name override takes effect without touching .env.local. Part of the rosetta-test e2e flow; can also be invoked standalone if setup already ran.

Installation

Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.

Run Skill in Manus

Source

Arize-ai

Arize-ai/project-rosetta-stone

View GitHub Repository View Creator Repositories

Download

Run Skill in Manus

Related occupationsSOC

Based on SOC occupation classification

Software DevelopersComputer and Mathematical Occupations·SOC 15-1252

SKILL.md

readonly

name	rosetta-test-traces
description	Run the 25 synthetic Wonder Toys requests against a Rosetta Stone framework to generate traces in an isolated Arize AX or Phoenix project. Loads the .env.test-local overlay so the project name override takes effect without touching .env.local. Part of the rosetta-test e2e flow; can also be invoked standalone if setup already ran.

Rosetta Test — Traces Phase

Sends 25 synthetic requests to a freshly provisioned project so the platform has trace data to evaluate.

Inputs

PROJECT_DIR — absolute path to <platform>/<framework>/
PROJECT_NAME — the unique name emitted by setup
PLATFORM — ax or phoenix

Steps

1. Install npm deps if missing

test -d "$PROJECT_DIR/node_modules" || (cd "$PROJECT_DIR" && npm install)

The eval harness shell scripts (evals/run-synthetic-requests.sh) handle the dev server and Chroma startup themselves — don't pre-start those here.

2. Run synthetic requests with the env overlay

The harness scripts only auto-load .env and .env.local. To get .env.test-local into the npm child process, export it into the parent shell first:

(
  set -a
  [ -f "$PROJECT_DIR/.env.local" ]      && source "$PROJECT_DIR/.env.local"
  [ -f "$PROJECT_DIR/.env.test-local" ] && source "$PROJECT_DIR/.env.test-local"
  set +a
  cd "$PROJECT_DIR" && npm run synthetic-requests
)

Order matters: .env.test-local sourced last so the project-name override wins.

This call:

Starts the dev server in the background if not already running (with its own trap to kill on exit).
For Python framework tiers, also starts the FastAPI backend on port 8001.
Sends all 25 requests sequentially to /api/chat.
Sleeps 20s afterward for trace ingestion.

Expect ~5–15 minutes of wall time depending on framework and LLM latency. Do not background or timeout-truncate this — let it complete.

3. Confirm trace ingestion

For AX, give the ingestion pipeline an extra buffer beyond the harness's built-in sleep before the evals phase runs. The harness already sleeps 20s; add 30s more here:

[ "$PLATFORM" = "ax" ] && sleep 30

Phoenix ingestion is typically faster — no extra wait needed.

4. Quick sanity check

Before handing off, confirm at least one trace landed:

AX: ax traces list "$PROJECT_NAME" --space "$ARIZE_SPACE_ID" --limit 1 --output json
Phoenix: hit ${PHOENIX_BASE_URL}/v1/projects with the API key and look for the project name, where PHOENIX_BASE_URL = ${PHOENIX_COLLECTOR_ENDPOINT%/v1/traces}.

If the sanity check shows zero traces, abort — re-running the harness from this point is safer than continuing to evals against an empty project.

Failure modes

Dev server fails to start within 60s → harness aborts and dumps /tmp/nextjs-evals.log. Surface that log to the user.
ChromaDB indexing fails → no traces will land. The harness still exits 0 in some cases. Sanity check (step 4) catches this.
HTTP non-200 from /api/chat → likely missing EVAL_SECRET (the harness generates one but the route must validate it). Check the project's src/app/api/chat/route.ts for the bypass logic.

More from this repository

same repository

rosetta-add-framework

Arize-ai/project-rosetta-stone

Add a new agent framework to the Rosetta Stone repo — researches the framework, builds all three observability tiers (no-observability, phoenix, ax), tests each, runs a Playwright smoke against the UI, updates README + TODO, and raises a PR. Trigger when the user asks to "add the <framework> framework", "implement <framework>", "wire up <framework>", or similar. The framework must be one of the Arize-supported agent frameworks (see TODO below).

2026-06-304

rosetta-add-framework-tier-build

Arize-ai/project-rosetta-stone

Build a single tier (no-observability, phoenix, or ax) for a new framework. Clones the closest existing tier, swaps in framework-specific agent.py / tools.py / requirements.txt, and (for observability tiers) adds tracing.py + main.py wiring + eval-harness scripts. Part of the rosetta-add-framework flow; can be invoked standalone to rebuild a single tier from scratch.

2026-06-304

check-models

Arize-ai/project-rosetta-stone

Find and update out-of-date OpenAI and Anthropic model references (in docs, MDX, notebooks, and code) to the latest size-equivalent models, and apply the code changes each new generation requires (e.g. max_tokens → max_completion_tokens for GPT-5). Use when asked to "check the models", "update model versions", "are these models current", "migrate the models in the docs/tutorials", or before publishing content that names a model.

2026-06-254

rosetta-add-framework-playwright

Arize-ai/project-rosetta-stone

Run a public-flow Playwright smoke test against a freshly-built framework tier's Next.js frontend. Covers home page rendering + product browsing — the parts that don't require X/Twitter OAuth. The Playwright project (package.json, config, tests) lives inside this skill directory and is checked into the repo. Part of the rosetta-add-framework flow.

2026-05-274

rosetta-pr-screenshots

Arize-ai/project-rosetta-stone

Capture AX trace UI, Phoenix trace UI, and Wonder Toys app UI screenshots for a framework's PR, then upload as GitHub release assets and embed in the PR body. Called automatically by rosetta-add-framework-docs as part of new-framework PRs; can also be invoked standalone to retrofit existing PRs. Cross-platform — uses Playwright end-to-end.

2026-05-274

rosetta-add-framework-docs

Arize-ai/project-rosetta-stone

Finalise a newly-added framework — updates the README's supported-frameworks table, directory tree, and per-framework "what differs" section, marks off the framework in the orchestrator skill's embedded TODO, commits per tier, and raises a PR. Part of the rosetta-add-framework flow.

2026-05-204

name	rosetta-test-traces
description	Run the 25 synthetic Wonder Toys requests against a Rosetta Stone framework to generate traces in an isolated Arize AX or Phoenix project. Loads the .env.test-local overlay so the project name override takes effect without touching .env.local. Part of the rosetta-test e2e flow; can also be invoked standalone if setup already ran.

Rosetta Test — Traces Phase

Sends 25 synthetic requests to a freshly provisioned project so the platform has trace data to evaluate.

Inputs

PROJECT_DIR — absolute path to <platform>/<framework>/
PROJECT_NAME — the unique name emitted by setup
PLATFORM — ax or phoenix

Steps

1. Install npm deps if missing

test -d "$PROJECT_DIR/node_modules" || (cd "$PROJECT_DIR" && npm install)

The eval harness shell scripts (evals/run-synthetic-requests.sh) handle the dev server and Chroma startup themselves — don't pre-start those here.

2. Run synthetic requests with the env overlay

The harness scripts only auto-load .env and .env.local. To get .env.test-local into the npm child process, export it into the parent shell first:

(
  set -a
  [ -f "$PROJECT_DIR/.env.local" ]      && source "$PROJECT_DIR/.env.local"
  [ -f "$PROJECT_DIR/.env.test-local" ] && source "$PROJECT_DIR/.env.test-local"
  set +a
  cd "$PROJECT_DIR" && npm run synthetic-requests
)

Order matters: .env.test-local sourced last so the project-name override wins.

This call:

Starts the dev server in the background if not already running (with its own trap to kill on exit).
For Python framework tiers, also starts the FastAPI backend on port 8001.
Sends all 25 requests sequentially to /api/chat.
Sleeps 20s afterward for trace ingestion.

Expect ~5–15 minutes of wall time depending on framework and LLM latency. Do not background or timeout-truncate this — let it complete.

3. Confirm trace ingestion

For AX, give the ingestion pipeline an extra buffer beyond the harness's built-in sleep before the evals phase runs. The harness already sleeps 20s; add 30s more here:

[ "$PLATFORM" = "ax" ] && sleep 30

Phoenix ingestion is typically faster — no extra wait needed.

4. Quick sanity check

Before handing off, confirm at least one trace landed:

AX: ax traces list "$PROJECT_NAME" --space "$ARIZE_SPACE_ID" --limit 1 --output json
Phoenix: hit ${PHOENIX_BASE_URL}/v1/projects with the API key and look for the project name, where PHOENIX_BASE_URL = ${PHOENIX_COLLECTOR_ENDPOINT%/v1/traces}.

If the sanity check shows zero traces, abort — re-running the harness from this point is safer than continuing to evals against an empty project.

Failure modes

Dev server fails to start within 60s → harness aborts and dumps /tmp/nextjs-evals.log. Surface that log to the user.
ChromaDB indexing fails → no traces will land. The harness still exits 0 in some cases. Sanity check (step 4) catches this.
HTTP non-200 from /api/chat → likely missing EVAL_SECRET (the harness generates one but the route must validate it). Check the project's src/app/api/chat/route.ts for the bypass logic.