inference-sim

Formulate a single testable hypothesis by scanning the project for untested behaviors, gaps, and claims

Guided hypothesis-driven experimentation. Choose a project, generate hypotheses, select which to test, run experiments in parallel or sequentially, document findings.

Generate iteratively-reviewed research ideas from a problem statement. Fully guided - no arguments required.

Send Claude plan to an LLM for external technical review

Create research document with problem statement and context from multiple sources (repositories, papers, files)

Populate FINDINGS.md with results and analysis, update HYPOTHESIS.md status

Execute experiment run.sh, run analysis script, parse output, and interpret results

Design experiment and generate run.sh, analyze script, and FINDINGS.md template

Dispatch parallel review perspectives and enforce convergence via persistent state file and automatic Phase A/B loop. Supports 8 gate types — design doc (8), macro plan (8), PR plan (10), PR code (10), pr-docs (7), hypothesis design (5), hypothesis code (5), hypothesis FINDINGS (10). Automatically loops through Phase A (review/tally) and Phase B (fix/verify) until 0 CRITICAL + 0 IMPORTANT or round limit.

Run the full hypothesis experiment workflow (Steps 0-10) — worktree, classify, design review (5 perspectives), human gate, implement, code review (5 perspectives), run, analyze, FINDINGS review (10 perspectives), self-audit, commit + PR. Enforces convergence protocol and hard gates.

Create a PR from a GitHub issue. Reads the issue, plans implementation, writes code following project standards (TDD, invariants, parity), creates the PR, then triggers blis-pr-review for self-review.

Project-specific PR self-review for BLIS. Evaluates correctness, invariants, cross-path parity (run/replay/observe), preemption safety, timeout consistency, and adherence to project standards. Run after creating a PR to catch issues before human review.

Thoroughly review a GitHub issue for correctness, evidence quality, scope, parity, and actionability. Produces a VALID / NEEDS WORK / SUPERSEDED / DUPLICATE verdict.

Analyze sim2real pipeline run results. Offers a catalog of standard analyses (per-workload latency comparison, per-request scatter) plus a free-form loop for custom charts, distributions, HTML reports, and cross-run comparisons.

Bootstrap an experiment repo into a sim2real transfer bundle. Two modes: (1) BLIS mode (default) — for BLIS-generated folders with algorithms/, workloads/, and config; derives transfer.yaml, baselines/, and a component submodule with user approval at each gate. (2) --byo mode — for operators arriving with pre-built EPP images plus a full baseline scenario and per-algorithm overlays; scaffolds transfer.yaml with `byo: true` markers and emits a copy-pasteable batched `sim2real translation register` command.

Translates simulation-discovered algorithms into production Go plugins for the target component (e.g. llm-d-inference-scheduler). Reads workspace/translations/<hash>/skill_input.json (written by `sim2real translate`), spawns a three-agent team (expert + writer + reviewer) per algorithm, and writes plugin source + treatment overlay to workspace/translations/<hash>/generated/<algo>/{cmd,pkg,<algo>_config.yaml,<algo>_output.json}. Use after `sim2real translate` reaches the translation checkpoint; follow up with `sim2real translate --resume` to validate outputs.

Validate a sim2real translation bundle against its simulation bundle. Checks workloads, configs, signals, policies, and runtime health.

Automates BLIS LLM data collection pipeline on Tekton. Handles cluster validation, pipeline deployment, monitoring, and data retrieval. Supports custom vLLM images and observability features (journey tracing, step tracing, KV events). Use for benchmarking (/blis) or deep debugging (/blis --observability).

Automates BLIS LLM benchmarking pipeline with inference-perf harness on Tekton. Handles cluster validation, pipeline deployment, monitoring, and data retrieval. Defaults to stock vLLM; observability features (journey tracing, step tracing, KV events) are opt-in via --observability/--obs. Use for detailed request lifecycle analysis (/blis-inference-perf, /blis-ip). Outputs inference-perf metrics: per_request_lifecycle_metrics.json, stage metrics, summary.

Use when asked to view, analyze, or display results from a BLIS data collection experiment. Triggers include "show results", "download data", "view benchmark", or referencing an experiment ID.