codeprobe

Run the codeprobe calibration gate and emit a curator profile when the R11 validity thresholds are met. Compares two curators over a holdout and enforces minimum tasks, minimum repos, and Pearson correlation before accepting. Triggers on calibrate curator, calibration gate, validity gate, curator profile, r11 gate, pearson correlation. Use this when a new curator version needs to be qualified before it is used in mining or scoring pipelines.

Diagnose mined-task infrastructure for drift and offline readiness. Compares metadata.json capability snapshots to live capabilities and runs credential-TTL preflight for airgapped runs. Triggers on check infra, capability drift, preamble drift, offline preflight, credential ttl, airgapped run readiness. Use this before running mined tasks that were produced on a different machine or weeks ago.

Analyze eval results from codeprobe runs. Compares configurations statistically, ranks by score and cost-efficiency, and produces actionable recommendations in JSON or pretty text. Triggers on interpret results, analyze eval results, compare configurations, rank agents, score regression, plot regression. Use this when the agent needs to turn a `codeprobe run` output directory into structured analysis.

Mine eval tasks from a repository's history using the codeprobe CLI. Extracts real code-change tasks from merged PRs/MRs with ground truth, test scripts, and scoring rubrics. Triggers on mine tasks, extract tasks, propose tasks, benchmark my repo, eval my repo, discover tasks. Use this when the agent needs to produce a reusable task suite from a codebase.

Execute eval tasks against an AI coding agent using the codeprobe CLI. Spawns isolated per-task sessions, scores with automated tests, and emits NDJSON events plus a terminal envelope. Triggers on run eval, run tasks, benchmark agent, evaluate agent, score agent, compare agents. Use this when the agent needs to produce scored results on a mined or scaffolded task suite.