メニュー
Use when monitoring, waiting on, cancelling, or fetching logs for an EvalHub evaluation job. Covers the full job lifecycle after submission. For submitting new jobs, use evalhub-eval.
Codex または Claude でインストール この Prompt をコピーして Codex、Claude、または他のアシスタントに貼り付けると、Skill ページを確認してインストールできます。
SOC 職業分類に基づく
| name | evalhub-jobs |
| description | Use when monitoring, waiting on, cancelling, or fetching logs for an EvalHub evaluation job. Covers the full job lifecycle after submission. For submitting new jobs, use evalhub-eval. |
| user-invocable | false |
Monitor, poll, cancel, and fetch logs for EvalHub evaluation jobs.
| Variable | Purpose | Example |
|---|---|---|
EVALHUB_BASE_URL | EvalHub API base URL | https://evalhub.apps.cluster.example.com |
EVALHUB_TOKEN | Bearer token for auth | sha256~... (from oc whoami -t) |
EVALHUB_TENANT | Namespace / tenant | eval-test |
pending → running → completed | failed | cancelled | partially_failed
When evalhub MCP server is connected:
| MCP Tool | Replaces |
|---|---|
get_job_status | evalhub_status.py JOB_ID |
cancel_job | evalhub_status.py JOB_ID --cancel |
MCP resources for listing:
| MCP Resource URI | Replaces |
|---|---|
evalhub://jobs | evalhub_status.py --list |
evalhub://jobs?status=running | evalhub_status.py --list --status running |
evalhub://jobs/{id} | evalhub_status.py JOB_ID |
Evaluation jobs can take minutes to hours. Choose the right approach:
| Situation | Approach |
|---|---|
| Short job (< 5 min expected) | --wait --timeout 300 or MCP get_job_status loop |
| Long job (> 5 min) | MCP get_job_status every 30s; show progress updates |
| Context compression risk | Save job ID with state.py, recover on next session |
| Job already running | Check state.py status first — job ID may already be saved |
When polling via MCP, stop when state is completed, failed, cancelled, or partially_failed.
uv run ~/.claude/skills/evalhub/scripts/evalhub_status.py JOB_ID 2>/dev/null
uv run ~/.claude/skills/evalhub/scripts/evalhub_status.py JOB_ID --wait 2>/dev/null
uv run ~/.claude/skills/evalhub/scripts/evalhub_status.py JOB_ID --wait --timeout 1800 2>/dev/null
uv run ~/.claude/skills/evalhub/scripts/evalhub_status.py --list 2>/dev/null
uv run ~/.claude/skills/evalhub/scripts/evalhub_status.py --list --status running 2>/dev/null
uv run ~/.claude/skills/evalhub/scripts/evalhub_status.py --list --status pending 2>/dev/null
uv run ~/.claude/skills/evalhub/scripts/evalhub_status.py JOB_ID --cancel 2>/dev/null
uv run ~/.claude/skills/evalhub/scripts/evalhub_logs.py JOB_ID 2>/dev/null
Before polling a long-running job, save the job ID to survive context compression:
uv run ~/.claude/skills/evalhub/scripts/state.py set job_id JOB_ID 2>/dev/null
On session resume, check for in-flight jobs:
uv run ~/.claude/skills/evalhub/scripts/state.py status 2>/dev/null
Clear after a job reaches a terminal state:
uv run ~/.claude/skills/evalhub/scripts/state.py set job_id "" 2>/dev/null
After completed, read the provider's agent metadata for context:
uv run ~/.claude/skills/evalhub/scripts/evalhub_providers.py PROVIDER_ID 2>/dev/null
Use agent.result_interpretation to explain metric direction and baselines. Use agent.complements to suggest follow-up evaluations.
oc whoami -t--wait timeout is 600s. Use --timeout 3600 for heavy benchmarksnum_examples: Jobs submitted with --num-examples run faster — check the submission parametersScans Go repositories for CVEs with govulncheck and upgrades the Go toolchain only when a matching tag exists on registry.access.redhat.com/ubi9/go-toolset. Use when checking Go CVEs, remediating stdlib or module vulnerabilities, bumping go.mod or Dockerfile Go versions, or validating Red Hat UBI9 go-toolset availability. Make sure to use this skill whenever the user mentions Go CVEs, govulncheck, golang version bumps, or ubi9/go-toolset image availability.
Use when evaluating AI/ML models, running benchmarks, checking evaluation status, listing providers or collections, or performing model quality and safety assessments. Covers EvalHub provider discovery, benchmark execution, job lifecycle, and Evaluation-Driven Development (EDD) workflows.
Use when discovering EvalHub providers, benchmarks, or collections — listing what evaluation options exist, filtering by capability (e.g. safety, reasoning), or reading agent metadata. Does NOT cover job submission or status monitoring.
Use when submitting an evaluation job to EvalHub — running a model against benchmarks or collections. Covers job creation only. For status monitoring, use evalhub-jobs. For provider/benchmark discovery, use evalhub-discovery.