Collects comparison test run artifacts and answers the user's questions based on the trajectories of each run. WHEN TO USE: collect comparison test artifacts
Standardmäßig ist der Prompt ausgewählt, der zuerst die Quelle prüft. Sie können zu einem direkten Befehl wechseln oder eine lokale Kopie herunterladen.
Quelldateien prüfen
Lesen Sie SKILL.md und alle von SkillsMP angezeigten Begleitdateien, bevor Sie sich für eine Installation entscheiden.
Mit Codex oder Claude installieren Kopieren Sie diesen Prompt, fügen Sie ihn in Codex, Claude oder einen anderen Assistant ein und lassen Sie die Skill-Seite prüfen und installieren.
Ein direkter Befehl überspringt den Prüf-Prompt. Prüfen Sie die Quelle, bevor Sie ihn ausführen.
Collects comparison test run artifacts and answers the user's questions based on the trajectories of each run. WHEN TO USE: collect comparison test artifacts
license
MIT
metadata
{"author":"Microsoft","version":"1.0.0"}
Steps
Collect run artifacts
Execute the collect-artifacts script to download the test run artifacts.
The user must provide an JSON file to correlate each comparison test run with the GitHub Actions run. The script expects one input argument as the path to this JSON file. The JSON input is supposed to be the JSON output when queuing the comparison test runs using the npm run compare:run command.
cd tests/
npm run compare:collect -- input.json
The collect-artifacts script will download the test run artifacts to a directory named comparison-artifacts in the current working directory. Before executing the script, check if there is already such an directory. If so, skip executing the script and proceed to step 2.
Extract insights
The downloaded artifacts will have the following folder structure:
Each <branch-name>/<stimulus-name>/<model>-with-skill or <branch-name>/<stimulus-name>/<model>-without-skill directory contains the test run trajectories for that stimulus and model on that branch, with or without skills. Each trajectory is a markdown file that records user prompts, tool call requests, tool execution results, assistant responses that happened during the run. It also contains statistics such as token usage and turns. Based on the trajectories, answer the user's questions for each test run. Generate a report following the report-template to show your answers.
