メニュー
LLM-as-Judge trace quality scorer. Evaluates Hermes conversation traces in Langfuse using a local Ollama model and posts quality scores back to Langfuse. Cron schedule: weekdays 12pm/8pm, weekends 10pm KST.
デフォルトでは、最初にソースを確認する Prompt が選択されています。直接コマンドに切り替えるか、ローカルコピーをダウンロードすることもできます。
インストールを決める前に、SKILL.md と SkillsMP に表示されている付属ファイルをお読みください。
Codex または Claude でインストール この Prompt をコピーして Codex、Claude、または他のアシスタントに貼り付けると、Skill ページを確認してインストールできます。
直接コマンドでは確認用 Prompt が省略されます。実行前にソースを確認してください。
npx skills add https://github.com/lukemcqueen/hermes-cortex --skill llm-judge-scorerコマンドは1行のまま表示されます。コピー前に横へスクロールして全体を確認してください。
ローカルで確認しますか？SkillsMP が現在取得できるファイルをダウンロードできます。
Cross-server agent health monitoring using binary status vectors — deploy health endpoints on each agent, poll from orchestrator, alert on state transitions.
Wire a self-hosted Langfuse instance to Hermes Agent — generate API keys, configure env vars, enable the bundled plugin, install SDK, and verify traces flow.
Use before enforcement code changes or shared-repo commits.
SOC 職業分類に基づく
SKILL.md を表示中
| name | llm-judge-scorer |
| version | 1.0.0 |
| category | devops |
| description | LLM-as-Judge trace quality scorer. Evaluates Hermes conversation traces in Langfuse using a local Ollama model and posts quality scores back to Langfuse. Cron schedule: weekdays 12pm/8pm, weekends 10pm KST. |
| author | Hermes Cortex |
| license | MIT |
| metadata | {"hermes":{"tags":["langfuse","scoring","quality","eval","ollama","cron"]}} |
A no_agent cron script that evaluates Hermes conversation traces in Langfuse using qwen2.5:3b as the judge model. Scores are posted back to Langfuse under helpfulness, clarity, depth, and overall.
Langfuse (traces)
│
▼
llm-judge-scorer.py (no_agent cron)
│
├── Fetches unscored traces from past 14 days
├── Sends input/output to Ollama (qwen2.5:3b)
├── Parses JSON response: helpfulness, clarity, depth, overall
└── Posts scores back to Langfuse API
| Dependency | Check | Fix |
|---|---|---|
| Langfuse running | curl -s http://localhost:3000/api/public/health | docker compose -f ~/langfuse/docker-compose.yml up -d |
| Ollama running | curl -s http://localhost:11434/api/tags | brew services start ollama |
| Judge model | ollama list | grep qwen2.5:3b | ollama pull qwen2.5:3b |
| Env file | ~/.hermes-cortex/.env (symlink to ~/langfuse/.env) or --env-path | ln -sf ~/langfuse/.env ~/.hermes-cortex/.env |
# Default (finds unscored traces, evaluates, posts scores)
python3 ~/.hermes/scripts/llm-judge-scorer.py
# Dry run (no scores posted)
python3 ~/.hermes/scripts/llm-judge-scorer.py --dry-run
# Specific trace
python3 ~/.hermes/scripts/llm-judge-scorer.py --trace-id=<id>
# Quiet mode (silent when no traces to score)
python3 ~/.hermes/scripts/llm-judge-scorer.py --quiet
# Custom env path
python3 ~/.hermes/scripts/llm-judge-scorer.py --env-path ~/langfuse/.env
| Score | Name | Meaning |
|---|---|---|
| 1-5 | helpfulness | How well the response addresses the user's request |
| 1-5 | clarity | Structure, formatting, readability |
| 1-5 | depth | Substance beyond surface-level |
| 1-10 | overall | Holistic quality assessment |
When overall < 5 on your traces:
| Symptom | Fix |
|---|---|
FileNotFoundError: .env | Create symlink: ln -sf ~/langfuse/.env ~/.hermes-cortex/.env |
HTTP 401 from Langfuse | Check LANGFUSE_INIT_PROJECT_PUBLIC_KEY and SECRET_KEY in .env |
ollama: connection refused | brew services start ollama or ollama serve |
Model not found | ollama pull qwen2.5:3b |
| Zero traces scored | Normal if all traces already scored. Check Langfuse has traces. |
Script lives at ops/scripts/manage/llm-judge-scorer.py in the cortex repo.
Deployed to ~/.hermes/scripts/llm-judge-scorer.py via cortex-update.