菜单
Problem-statement alignment check — given the user's problem statement (a required audit input), judge whether the environment actually tests what they claim they're trying to solve.
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
基于 SOC 职业分类
Audit a Prime Intellect / verifiers RL environment before training on it. Runs six judgment-based checks (integrity, problem-statement alignment, reward design, latency, rollout quality, contamination) and produces a per-check scorecard with scores and written justifications. Use when the user wants to audit, review, vet, or quality-check an RL environment, or asks "is my env good / ready to train on?".
Contamination check — compare the environment's dataset against the HuggingFace datasets the user explicitly provided. N/A (carries no weight in the rating) if the user provided none. Matching instances lower the score; a clean dataset scores high.
Integrity check for an RL environment — verify it is written properly and actually runs. Confirms the dataset loads and is well-formed, the reward function is present and callable, the code follows verifiers/prime-intellect conventions, and there are no missing fields or broken imports. The "does it even run and is it shaped right" check.
Repair an audited RL environment from its scorecard feedback — only when the user explicitly asks to rewrite/repair/fix the env. Applies mechanical fixes to a local copy under rlenv_audit_repairs/ (never the installed package, never the Hub), triages design-level findings into recommendations, and validates the repaired copy by re-running the cheap checks. Publishing the result is the user's job.
Reward-design check — stress-test the reward function without the policy model. Sample ~20 prompts, write synthetic completions for each (correct, wrong, edge cases, format perturbations), pass them through the reward function, and check (a) the reward varies and discriminates with sensible ordering, and (b) the assigned reward matches your own judgment of each completion's quality.
Latency check — measure how long rollouts take end to end. Requires a model endpoint. Reads the shared cached rollout set (8 rollouts over ~20 samples) and reports timing; does not run its own rollouts.
| name | env-audit-problem-alignment |
| description | Problem-statement alignment check — given the user's problem statement (a required audit input), judge whether the environment actually tests what they claim they're trying to solve. |
Question: does this environment actually test the thing the user says they want to solve?
The problem statement is a required audit input. If you don't have it, stop and tell the user: "A problem statement must be specified — what are you trying to train or test with this environment?" Do not score this check, do not substitute a guess from the env itself, and do not silently skip it — wait for the user's answer. This check needs no model — only your judgement over the env.
Read the inspect JSON (/tmp/envaudit_inspect.json): system prompt, sample
tasks (prompts + answers), and the reward function source.
Hold these against the user's problem statement and judge alignment:
dataset_size from the inspect JSON.Difficulty calibration. Judge whether the difficulty band fits the stated goal: tasks a base model already aces give no training signal at the claimed level (reward will saturate at 1), tasks far beyond it give none either (saturates at 0). You're judging the intent here from the samples — the rollout-quality check measures the saturation empirically.
Call out concrete mismatches (over-broad, too easy/hard, measures the wrong thing, off-domain examples, wrong interaction shape, too small to train on) and genuine alignment.
Score 0–10 for how well the env tests the stated problem:
{"name": "problem_alignment", "status": "PASS|WARN|FAIL", "score": <int>,
"justification": "<one line: aligned on X, misaligned on Y>"}