rosetta-test

name

rosetta-test

description

End-to-end test a Rosetta Stone framework × platform combination. Provisions a fresh isolated project on Arize AX or Phoenix, generates 25 synthetic traces, runs the 6 evals, verifies every trace was evaluated, then deletes the project. Trigger when the user asks to "test the <framework> <platform> project", "run e2e on <framework> <platform>", "verify <framework> works on <platform>", or any similar phrasing. Frameworks and platforms are discovered from the directory layout — no hardcoded list, so this works for any new framework added under ax/ or phoenix/.

Rosetta Stone End-to-End Test

Orchestrates a full lifecycle test of one framework × platform combination from this repo.

Inputs

Parse from the user's invocation:

• <framework> — e.g. mastra, langchain-py, microsoft-agent-py, vercel-ai-sdk, or any new framework directory
• <platform> — ax or phoenix
• --keep (optional) — skip the cleanup phase so the project remains for inspection

The skill is generic: it never hardcodes the framework list. New frameworks added to ax/ or phoenix/ work automatically as long as they follow the existing convention.

Convention contract

A <platform>/<framework>/ directory is testable if:

1. It exists.
2. Its package.json defines synthetic-requests as an npm script.
3. For phoenix only: package.json also defines evals.

Validate this before doing anything else. Abort early with a clear error if violated — list the directories that available under that platform so the user can correct typos.