Menu
Run the LangSmith evaluation suite and display pass/fail results
Instalar com Codex ou Claude Copie este prompt, cole no Codex, Claude ou outro assistente e deixe que ele revise a página da skill e instale para você.
Baseado na classificação ocupacional SOC
Add a new LangChain tool to the Medox ReAct agent. Use when creating a new tool file, registering it in the graph, and writing its test.
Create a git commit with Medox conventions (co-author, conventional type/scope). Invoke manually with /commit — do NOT trigger automatically.
Audit code changes on the current branch and update docs/ + .claude/rules/ to stay in sync. Run before /pr.
Create or update a GitHub pull request following Medox conventions (author spicode-bot, reviewer spideystreet). Invoke manually with /pr — do NOT trigger automatically.
Fetch and display the full status of the current PR — CI checks, GitHub Actions runs, reviews, and code comments — using GitHub MCP tools. Invoke manually with /pr-status.
Rebuild ChromaDB Gold layer indexes after source data changes. Invoke manually — deletes and recreates collections, do NOT trigger automatically.
| name | run-eval |
| description | Run the LangSmith evaluation suite and display pass/fail results |
| disable-model-invocation | true |
Ensure Docker is running
docker compose ps
If PostgreSQL or ChromaDB is not up:
docker compose up -d
Run the evaluation suite
uv run dotenv -f .env run -- python scripts/run_eval.py
Note the experiment name printed (e.g. medox-<hash>).
Fetch and display results
Write the following script to /tmp/check_eval.py then run it:
from langsmith import Client
client = Client()
runs = list(client.list_runs(project_name='<experiment_name>', is_root=True))
print(f'Eval cases: {len(runs)}')
print()
passed, failed = 0, 0
for run in runs:
fb = list(client.list_feedback(run_ids=[str(run.id)]))
score = fb[0].score if fb else None
comment = fb[0].comment if fb else ''
prompt = (run.inputs or {}).get('prompt', '').strip()[:75]
status = 'PASS' if score == 1 else 'FAIL'
if score == 1:
passed += 1
else:
failed += 1
print(f'[{status}] {prompt}')
if comment and comment != 'OK':
print(f' -> {comment}')
print()
print(f'Result: {passed} passed, {failed} failed out of {len(runs)}')
Replace <experiment_name> with the value printed in step 2, then:
uv run dotenv -f .env run -- python3 /tmp/check_eval.py
Investigate failures
For any [FAIL], read the comment and:
src/medox/agent/scripts/run_eval.py/add-eval-case to add a regression case if a new edge case was foundReport summary
Print the final Result: N passed, M failed out of X line to the user.