with one click
auto-judge-starter-kit
auto-judge-starter-kit contains 3 collected skills from trec-auto-judge, with repository-level occupation coverage and site-owned skill detail pages.
Skills in this repository
Set up a development environment to start building a new TREC AutoJudge from this starter kit. Use when the user wants to get the starter kit running, install dependencies, configure the LLM endpoint, or asks "how do I start / set up / begin developing my judge". Walks through venv, install, LLM endpoint configuration, and a verification run on the kiddie dataset.
Submit a TREC AutoJudge to TIRA via a code submission. Use when the user wants to submit, upload, or run tira-cli code-submission for their judge, or asks "how do I submit to TIRA / ship my judge". Walks through local verification, LLM environment, git hygiene, authentication, a dry-run, and the real upload.
Implement, run, and evaluate a TREC AutoJudge. Use when the user wants to build a judge, scaffold a judge directory, write judge()/create_nuggets()/create_qrels(), run workflows or variants, debug prompt-cache misses, or meta-evaluate against ground truth. Covers the develop, run, cache, and meta-evaluation activities.