Menu
Run the LangSmith evaluation suite and display pass/fail results
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
Based on SOC occupation classification
Add a new LangChain tool to the Medox ReAct agent. Use when creating a new tool file, registering it in the graph, and writing its test.
Create a git commit with Medox conventions (co-author, conventional type/scope). Invoke manually with /commit — do NOT trigger automatically.
Audit code changes on the current branch and update docs/ + .claude/rules/ to stay in sync. Run before /pr.
Create or update a GitHub pull request following Medox conventions (author spicode-bot, reviewer spideystreet). Invoke manually with /pr — do NOT trigger automatically.
Fetch and display the full status of the current PR — CI checks, GitHub Actions runs, reviews, and code comments — using GitHub MCP tools. Invoke manually with /pr-status.
Rebuild ChromaDB Gold layer indexes after source data changes. Invoke manually — deletes and recreates collections, do NOT trigger automatically.
| name | run-eval |
| description | Run the LangSmith evaluation suite and display pass/fail results |
| disable-model-invocation | true |
Ensure Docker is running
docker compose ps
If PostgreSQL or ChromaDB is not up:
docker compose up -d
Run the evaluation suite
uv run dotenv -f .env run -- python scripts/run_eval.py
Note the experiment name printed (e.g. medox-<hash>).
Fetch and display results
Write the following script to /tmp/check_eval.py then run it:
from langsmith import Client
client = Client()
runs = list(client.list_runs(project_name='<experiment_name>', is_root=True))
print(f'Eval cases: {len(runs)}')
print()
passed, failed = 0, 0
for run in runs:
fb = list(client.list_feedback(run_ids=[str(run.id)]))
score = fb[0].score if fb else None
comment = fb[0].comment if fb else ''
prompt = (run.inputs or {}).get('prompt', '').strip()[:75]
status = 'PASS' if score == 1 else 'FAIL'
if score == 1:
passed += 1
else:
failed += 1
print(f'[{status}] {prompt}')
if comment and comment != 'OK':
print(f' -> {comment}')
print()
print(f'Result: {passed} passed, {failed} failed out of {len(runs)}')
Replace <experiment_name> with the value printed in step 2, then:
uv run dotenv -f .env run -- python3 /tmp/check_eval.py
Investigate failures
For any [FAIL], read the comment and:
src/medox/agent/scripts/run_eval.py/add-eval-case to add a regression case if a new edge case was foundReport summary
Print the final Result: N passed, M failed out of X line to the user.