菜单
Validate that an EAGLE3 pipeline run completed successfully end-to-end. Checks all 4 steps produced expected artifacts, verifies acceptance rate meets threshold (>= 2.1), and produces a summary report. Use when user wants to verify a pipeline run or check benchmark results.
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
基于 SOC 职业分类
Evaluates accuracy of quantized or unquantized LLMs using NeMo Evaluator Launcher (NEL). Triggers on "evaluate model", "benchmark accuracy", "run MMLU", "evaluate quantized model", "run nel". Handles deployment, config generation, and evaluation execution. Not for quantizing models (use ptq), deploying/serving models (use deployment), or comparing completed baseline-vs-quantized results (use compare-results).
Serve a quantized or unquantized LLM checkpoint as an OpenAI-compatible API endpoint using vLLM, SGLang, or TRT-LLM. Use when user says "deploy model", "serve model", "start vLLM server", "launch SGLang", "TRT-LLM deploy", "AutoDeploy", "benchmark throughput", "serve checkpoint", or needs an inference endpoint from a HuggingFace or ModelOpt-quantized checkpoint. Do NOT use for quantizing models (use ptq) or evaluating accuracy (use evaluation).
This skill should be used when the user asks to "quantize a model", "run PTQ", "post-training quantization", "NVFP4 quantization", "FP8 quantization", "INT8 quantization", "INT4 AWQ", "quantize LLM", "quantize MoE", "quantize VLM", or needs to produce a quantized HuggingFace or TensorRT-LLM checkpoint from a pretrained model using ModelOpt.
Deterministic end-to-end driver for day-0 quantized-checkpoint releases — chains PTQ → evaluation → comparison with enforced gates between stages (the evaluation stage deploys the checkpoint itself), and returns a publish decision (ACCEPT / REGRESSION / ANOMALOUS / INFEASIBLE). Use when the user asks to "release a model at day-0", "quantize and validate model X is within N% of baseline and tell me if it's publishable", or "run the full day-0 workflow". Do NOT use for single-stage requests — quantizing only (use ptq), serving only (use deployment), evaluating only (use evaluation), or comparing two existing runs (use compare-results).
Establish baseline-vs-candidate evaluation plans, delegate missing evaluations, compare validated results, and decide quantization feasibility. Use when the user asks to compare baseline vs quantized runs, explain an accuracy drop/regression, verify whether a quantized checkpoint is acceptable, or compare NEL/MLflow evaluation outputs. Do NOT use for generic single-model evaluation without comparison intent (use evaluation), live NEL status/debugging (use launching-evals), or generic MLflow browsing without a comparison goal (use accessing-mlflow).
Add a new model to the EAGLE3 offline pipeline. Generates an hf_offline_eagle3.yaml launcher config for a new model checkpoint, choosing the right hidden state dump backend (TRT-LLM / HF / vLLM) and GPU configuration. Use when user wants to run EAGLE3 on a model that does not yet have a YAML in tools/launcher/examples/ or asks how to configure the pipeline for a new checkpoint.
| name | eagle3-validate |
| description | Validate that an EAGLE3 pipeline run completed successfully end-to-end. Checks all 4 steps produced expected artifacts, verifies acceptance rate meets threshold (>= 2.1), and produces a summary report. Use when user wants to verify a pipeline run or check benchmark results. |
| user_invocable | true |
Verify that an EAGLE3 pipeline run completed successfully and meets quality criteria.
Find the most recent experiment directory (or ask the user for the path):
ls -td experiments/cicd/cicd_* | head -5
Each experiment directory has one subdirectory per task (numbered 0–3), each containing a
log file whose name varies by launch mode (Slurm: sbatch_*.out, local Docker: *.log).
Match the log files generally and read the tail of each:
find experiments/<exp_id>/ -type f \( -name '*.out' -o -name '*.log' \) | sort | while read -r f; do
echo "=== $f ==="; tail -50 "$f"; echo
done
All 4 tasks must complete without error. Look for:
exit code: 0 or no error — successDUE TO TIME LIMIT — timeoutFAILED / signal / exception traceback — failureIf any task failed, suggest running /eagle3-triage instead.
Check each step produced the expected output (artifacts live on the cluster at /scratchspace/).
Confirm via log messages:
| Step | Expected log evidence | Artifact |
|---|---|---|
| task_0 | "Saved N samples" or progress bar completing | /scratchspace/data/*.jsonl |
| task_1 | "Successfully processed N conversations" | /scratchspace/offline_hidden_states/*.pt |
| task_2 | Training loss decreasing, "export complete" | /scratchspace/eagle3/model.safetensors, /scratchspace/export/ |
| task_3 | Average Acceptance Length ... ratio: X.XX | JSON result files |
In the task_3 log, find:
Average Acceptance Length {'accept': X, 'count': Y, 'ratio': Z.ZZ}
The ratio field is the acceptance rate (AR).
| Criterion | Threshold | Status |
|---|---|---|
| AR (MT-Bench) | >= 2.1 | PASS / FAIL |
If the log shows AR ... < lower bound, the run already triggered a threshold failure (exit code 1).
In the task_2 log look for:
training.ar_validate_steps was set)## EAGLE3 Pipeline Validation Report
**Experiment:** <exp_dir>
**Model:** <model_name>
**Date:** <date>
**Pipeline config:** <yaml_path>
### Step Status
| Step | Task | Status | Notes |
|------|------|--------|-------|
| 0 | Data synthesis | PASS/FAIL/TIMEOUT | N samples generated |
| 1 | Hidden state dump | PASS/FAIL | N .pt files |
| 2 | Training + export | PASS/FAIL | Final loss: X.XX |
| 3 | Benchmark | PASS/FAIL | AR: X.XX |
### Acceptance Rate
- MT-Bench AR: X.XX (threshold: >= 2.1) — PASS/FAIL
### Training Summary
- Final loss: X.XX
- Training steps: N
- AR during training: X.XX (if validated)
### Overall: PASS / FAIL
<one-line summary>
If PASS:
If FAIL:
/eagle3-triage for diagnosis