菜单
LLM-as-Judge trace quality scorer. Evaluates Hermes conversation traces in Langfuse using a local Ollama model and posts quality scores back to Langfuse. Cron schedule: weekdays 12pm/8pm, weekends 10pm KST.
默认使用会先检查来源的 Prompt；你也可以切换为直接命令，或下载本地副本。
决定是否安装前，请先阅读 SKILL.md，以及 SkillsMP 当前展示的配套文件。
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
直接命令不会经过审查 Prompt；运行前请先检查来源。
npx skills add https://github.com/lukemcqueen/hermes-cortex --skill llm-judge-scorer命令会保持在同一行。复制前请横向滚动并检查完整内容。
想先保存到本地？可下载 SkillsMP 当前能够提供的文件。
Cross-server agent health monitoring using binary status vectors — deploy health endpoints on each agent, poll from orchestrator, alert on state transitions.
Wire a self-hosted Langfuse instance to Hermes Agent — generate API keys, configure env vars, enable the bundled plugin, install SDK, and verify traces flow.
Use before enforcement code changes or shared-repo commits.
基于 SOC 职业分类
正在显示 SKILL.md
| name | llm-judge-scorer |
| version | 1.0.0 |
| category | devops |
| description | LLM-as-Judge trace quality scorer. Evaluates Hermes conversation traces in Langfuse using a local Ollama model and posts quality scores back to Langfuse. Cron schedule: weekdays 12pm/8pm, weekends 10pm KST. |
| author | Hermes Cortex |
| license | MIT |
| metadata | {"hermes":{"tags":["langfuse","scoring","quality","eval","ollama","cron"]}} |
A no_agent cron script that evaluates Hermes conversation traces in Langfuse using qwen2.5:3b as the judge model. Scores are posted back to Langfuse under helpfulness, clarity, depth, and overall.
Langfuse (traces)
│
▼
llm-judge-scorer.py (no_agent cron)
│
├── Fetches unscored traces from past 14 days
├── Sends input/output to Ollama (qwen2.5:3b)
├── Parses JSON response: helpfulness, clarity, depth, overall
└── Posts scores back to Langfuse API
| Dependency | Check | Fix |
|---|---|---|
| Langfuse running | curl -s http://localhost:3000/api/public/health | docker compose -f ~/langfuse/docker-compose.yml up -d |
| Ollama running | curl -s http://localhost:11434/api/tags | brew services start ollama |
| Judge model | ollama list | grep qwen2.5:3b | ollama pull qwen2.5:3b |
| Env file | ~/.hermes-cortex/.env (symlink to ~/langfuse/.env) or --env-path | ln -sf ~/langfuse/.env ~/.hermes-cortex/.env |
# Default (finds unscored traces, evaluates, posts scores)
python3 ~/.hermes/scripts/llm-judge-scorer.py
# Dry run (no scores posted)
python3 ~/.hermes/scripts/llm-judge-scorer.py --dry-run
# Specific trace
python3 ~/.hermes/scripts/llm-judge-scorer.py --trace-id=<id>
# Quiet mode (silent when no traces to score)
python3 ~/.hermes/scripts/llm-judge-scorer.py --quiet
# Custom env path
python3 ~/.hermes/scripts/llm-judge-scorer.py --env-path ~/langfuse/.env
| Score | Name | Meaning |
|---|---|---|
| 1-5 | helpfulness | How well the response addresses the user's request |
| 1-5 | clarity | Structure, formatting, readability |
| 1-5 | depth | Substance beyond surface-level |
| 1-10 | overall | Holistic quality assessment |
When overall < 5 on your traces:
| Symptom | Fix |
|---|---|
FileNotFoundError: .env | Create symlink: ln -sf ~/langfuse/.env ~/.hermes-cortex/.env |
HTTP 401 from Langfuse | Check LANGFUSE_INIT_PROJECT_PUBLIC_KEY and SECRET_KEY in .env |
ollama: connection refused | brew services start ollama or ollama serve |
Model not found | ollama pull qwen2.5:3b |
| Zero traces scored | Normal if all traces already scored. Check Langfuse has traces. |
Script lives at ops/scripts/manage/llm-judge-scorer.py in the cortex repo.
Deployed to ~/.hermes/scripts/llm-judge-scorer.py via cortex-update.