RLEnv_audit

Audit a Prime Intellect / verifiers RL environment before training on it. Runs six judgment-based checks (integrity, problem-statement alignment, reward design, latency, rollout quality, contamination) and produces a per-check scorecard with scores and written justifications. Use when the user wants to audit, review, vet, or quality-check an RL environment, or asks "is my env good / ready to train on?".

Contamination check — compare the environment's dataset against the HuggingFace datasets the user explicitly provided. N/A (carries no weight in the rating) if the user provided none. Matching instances lower the score; a clean dataset scores high.

Integrity check for an RL environment — verify it is written properly and actually runs. Confirms the dataset loads and is well-formed, the reward function is present and callable, the code follows verifiers/prime-intellect conventions, and there are no missing fields or broken imports. The "does it even run and is it shaped right" check.

Repair an audited RL environment from its scorecard feedback — only when the user explicitly asks to rewrite/repair/fix the env. Applies mechanical fixes to a local copy under rlenv_audit_repairs/ (never the installed package, never the Hub), triages design-level findings into recommendations, and validates the repaired copy by re-running the cheap checks. Publishing the result is the user's job.

Problem-statement alignment check — given the user's problem statement (a required audit input), judge whether the environment actually tests what they claim they're trying to solve.

Reward-design check — stress-test the reward function without the policy model. Sample ~20 prompts, write synthetic completions for each (correct, wrong, edge cases, format perturbations), pass them through the reward function, and check (a) the reward varies and discriminates with sensible ordering, and (b) the assigned reward matches your own judgment of each completion's quality.

Latency check — measure how long rollouts take end to end. Requires a model endpoint. Reads the shared cached rollout set (8 rollouts over ~20 samples) and reports timing; does not run its own rollouts.

Rollout-quality check — inspect actual model rollouts and judge whether the environment is set up well in practice. Is the system prompt right, is anything missing from it, are the model's outputs sensible given the prompts, are there obvious failure modes the env setup is causing. Requires a model endpoint; reads the shared cached rollout set.