メニュー
Operate the live props cluster as evaluator — fetch credentials from k8s, call the API at props.allegedly.works, trigger critic/grader runs, and inspect results.
デフォルトでは、最初にソースを確認する Prompt が選択されています。直接コマンドに切り替えるか、ローカルコピーをダウンロードすることもできます。
インストールを決める前に、SKILL.md と SkillsMP に表示されている付属ファイルをお読みください。
Codex または Claude でインストール この Prompt をコピーして Codex、Claude、または他のアシスタントに貼り付けると、Skill ページを確認してインストールできます。
直接コマンドでは確認用 Prompt が省略されます。実行前にソースを確認してください。
npx skills add https://github.com/agentydragon/ducktape --skill props-evaluatorコマンドは1行のまま表示されます。コピー前に横へスクロールして全体を確認してください。
ローカルで確認しますか？SkillsMP が現在取得できるファイルをダウンロードできます。
Audit every Flux Kustomization and HelmRelease over a window (default 7d): classify broken / slow / miswired / propagating / suspended / healthy with per-item culprit, failure counts, and p99 reconcile duration. Use for "what's slow / stuck / been broken this week".
Reference for querying the BuildBuddy API: build/test logs, invocations, RBE execution details, cache stats, undeclared test outputs, target pass/fail history for bisecting, workflow re-runs. Use for any why-did-CI-fail, show-me-the-log, or when-did-this-test-break task.
Check CI/CD pipeline health: devel status, per-commit checks, release artifacts, pin staleness, image publishing, scheduled jobs. Fixes trivial issues as PRs, proposes diagnosis plans for deeper ones. Use when CI seems broken, red, lagging, or stuck.
SOC 職業分類に基づく
SKILL.md を表示中
| name | props_evaluator |
| description | Operate the live props cluster as evaluator — fetch credentials from k8s, call the API at props.allegedly.works, trigger critic/grader runs, and inspect results. |
| allowed-tools | Bash, Read, Grep, Glob, WebFetch, Task |
Operate the production props deployment at https://props.allegedly.works using
evaluator credentials retrieved from the cluster.
The evaluator password lives in a k8s secret scoped to the props namespace:
EVALUATOR_PASSWORD=$(kubectl get secret props-evaluator-credentials -n props \
-o jsonpath='{.data.password}' | base64 -d)
EVALUATOR_CREDS=$(echo -n "evaluator:$EVALUATOR_PASSWORD" | base64 -w0)
Use as a Bearer token on every request:
curl -s -H "Authorization: Bearer $EVALUATOR_CREDS" \
https://props.allegedly.works/api/stats/overview
https://props.allegedly.works
Health check: GET /health → {"status":"ok"}
@props/docs/backend_api.md
All /api/gt/* and /api/stats/* endpoints use per-caller RLS via the caller's
Postgres credentials — there is no agent-type gate at the HTTP layer.
| Caller | /api/gt/* | /api/stats/* | /api/runs/* |
|---|---|---|---|
| Admin (postgres user) | All data | All data | Full access |
| Evaluator | All data (BYPASSRLS) | All data (BYPASSRLS) | Read-only |
critic_dev_optimize | TRAIN split only | TRAIN split only | Read-only |
critic | Own run only | Own run only | Read-only |
grader | Assigned snapshot only | Assigned snapshot only | Read-only |
Full endpoint list and request shapes: fetch /openapi.json from the live server.
# POST /api/runs/critic — trigger a critic run
curl -s -X POST https://props.allegedly.works/api/runs/critic \
-H "Authorization: Bearer $EVALUATOR_CREDS" \
-H "Content-Type: application/json" \
-d '{
"definition_id": "latest",
"example": {
"kind": "file_set",
"snapshot_slug": "ducktape/2025-09-03-00",
"files_hash": "8e2209f20bd1df0c5bc4073dfff739fe"
},
"critic_model": "gpt-oss-20b-128k",
"timeout_seconds": 1800,
"budget_usd": 0.0
}'
budget_usd = 0.0 for gpt-oss-20b-128k (cluster inference is free).
Top file-set examples (fastest to run):
| Rank | snapshot_slug | files_hash | TPs | Occurrences |
|---|---|---|---|---|
| 1 | ducktape/2025-09-03-00 | 8e2209f20bd1df0c5bc4073dfff739fe | 33 | 39 |
| 2 | ducktape/2025-11-20-00 | bb8aff17944a6348a8089790457e3094 | 15 | 31 |
| 3 | ducktape/2025-11-26-00 | 6e416fb1d095abc7fdc79131434c7dac | 20 | 21 |
| 4 | ducktape/2025-11-21-00 | 15702f4d16234db852e973e31323fbdd | 21 | 21 |
| 5 | gmail-archiver/2025-12-17-00 | 9e218584782810e5a65195da8f63931a | 14 | 21 |
# List recent runs
curl -s -H "Authorization: Bearer $EVALUATOR_CREDS" \
"https://props.allegedly.works/api/runs?limit=10" | python3 -m json.tool
# Get specific run
curl -s -H "Authorization: Bearer $EVALUATOR_CREDS" \
"https://props.allegedly.works/api/runs/<run_id>" | python3 -m json.tool
# Active runs
curl -s -H "Authorization: Bearer $EVALUATOR_CREDS" \
https://props.allegedly.works/api/runs/active | python3 -m json.tool
# Overview (definitions + example counts)
curl -s -H "Authorization: Bearer $EVALUATOR_CREDS" \
https://props.allegedly.works/api/stats/overview | python3 -m json.tool
# Per-definition performance by image digest
curl -s -H "Authorization: Bearer $EVALUATOR_CREDS" \
"https://props.allegedly.works/api/stats/definitions/<image_digest>" | python3 -m json.tool