메뉴
Latency check — measure how long rollouts take end to end. Requires a model endpoint. Reads the shared cached rollout set (8 rollouts over ~20 samples) and reports timing; does not run its own rollouts.
Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.
SOC 직업 분류 기준
Audit a Prime Intellect / verifiers RL environment before training on it. Runs six judgment-based checks (integrity, problem-statement alignment, reward design, latency, rollout quality, contamination) and produces a per-check scorecard with scores and written justifications. Use when the user wants to audit, review, vet, or quality-check an RL environment, or asks "is my env good / ready to train on?".
Contamination check — compare the environment's dataset against the HuggingFace datasets the user explicitly provided. N/A (carries no weight in the rating) if the user provided none. Matching instances lower the score; a clean dataset scores high.
Integrity check for an RL environment — verify it is written properly and actually runs. Confirms the dataset loads and is well-formed, the reward function is present and callable, the code follows verifiers/prime-intellect conventions, and there are no missing fields or broken imports. The "does it even run and is it shaped right" check.
Repair an audited RL environment from its scorecard feedback — only when the user explicitly asks to rewrite/repair/fix the env. Applies mechanical fixes to a local copy under rlenv_audit_repairs/ (never the installed package, never the Hub), triages design-level findings into recommendations, and validates the repaired copy by re-running the cheap checks. Publishing the result is the user's job.
Problem-statement alignment check — given the user's problem statement (a required audit input), judge whether the environment actually tests what they claim they're trying to solve.
Reward-design check — stress-test the reward function without the policy model. Sample ~20 prompts, write synthetic completions for each (correct, wrong, edge cases, format perturbations), pass them through the reward function, and check (a) the reward varies and discriminates with sensible ordering, and (b) the assigned reward matches your own judgment of each completion's quality.
| name | env-audit-latency |
| description | Latency check — measure how long rollouts take end to end. Requires a model endpoint. Reads the shared cached rollout set (8 rollouts over ~20 samples) and reports timing; does not run its own rollouts. |
Question: how expensive is a rollout end to end?
Requires a model endpoint. If none was configured, output:
{"name": "latency", "status": "N/A", "score": null, "justification": "no model endpoint provided"}
Read the shared rollout cache the orchestrator already generated
(/tmp/envaudit_rollouts.json, produced by rlenv-audit rollouts ... -n 20 -k 8,
which drives verifiers' own vf-eval engine under the hood). Do not roll
out again — checks 4 and 5 share this one cache. If the file is missing but an
endpoint was configured, generate it once with
rlenv-audit rollouts <env> --endpoint <url> --model <name> -n 20 -k 8 --out /tmp/envaudit_rollouts.json.
From its timing block read mean_s, p50_s, p90_s, max_s, total_s
and calls — per-rollout end-to-end times vf-eval recorded (generation plus
scoring). Note max_concurrent in the cache: timings reflect generation at
that concurrency (the realistic batched-rollout setting), so read them as
throughput-under-load rather than single-request latency.
If the cache was generated with --dummy ("dummy": true, empty timing),
there is no real timing to judge — output N/A with justification
"dummy rollouts — no real timing".
Judge the numbers in context: a per-rollout mean of a few seconds is normal for a hosted model; tens of seconds, heavy tail (p90 ≫ p50), or frequent errors mean the env's verification/tooling is slow and will bottleneck training throughput.
This check is informational — reserve FAIL for pathological cases (errors on most rollouts, or absurd latency). Score reflects throughput health.
{"name": "latency", "status": "PASS|WARN|FAIL", "score": <int>,
"justification": "<one line: mean/p90 per rollout, total, any error rate>"}