trainer-optimize

# Prompt Optimization with Agent Lightning Use this skill to improve a single markdown prompt file by running Agent Lightning prompt optimization against explicit JSONL train and validation datasets, while keeping authored eval cases in the official `evals/evals.json` layout. The runtime is single-shot: it returns one optimized prompt result and does not coordinate leader election internally. ## When to use this skill - The user says "optimize my prompt", "improve this prompt", or starts with `/trainer-optimize` - The target artifact is a single `.md` (markdown) prompt template file - The user provides, or can provide, train and validation JSONL datasets with expected outputs Do not use this skill for general code optimization or non-markdown prompts. ## Required inputs The skill always needs a prompt path. It also needs train and validation datasets. In this repository, authored eval cases live under `evals/evals.json`, while APO datasets are passed explicitly. - `prompt`: path to the markdown prompt file to optimize - `train`: path to the JSONL training dataset - `val`: path to the JSONL validation dataset If those files do not exist, stop and report the missing path instead of guessing. Do not synthesize missing datasets inside `trainer-optimize`. Any dataset creation or conversion must happen before this skill runs. ## Invocation syntax ``` /trainer-optimize prompt=<path> train=<path> val=<path> [options] ``` Required arguments: - `prompt` — path to the markdown prompt file to optimize - `train` — path to the JSONL training dataset - `val` — path to the JSONL validation dataset Optional arguments: - `iterations` (default `3`) — number of APO beam rounds - `algorithm` (default `apo`) — supported values are `apo` and `verl` - `output` — optional path to write the optimized markdown without touching the source prompt - `report` — where to write the JSON optimization report - `beam_width` (default `4`) - `branch_factor` (default `4`) - `n_runners` (default `4`) - `judge_mode` (default `deterministic`) — supported values are `deterministic`, `custom`, and `llm_judge` - `judge_prompt_file` — optional path for `llm_judge`; falls back to `assets/judge-default.md` - `debug_only` — run a smoke-test pass only, do not write output files - `in_place` — overwrite the original prompt file with the optimized result ## Current implementation behavior - Supported algorithms: `apo` and `verl`, selected through the same optimization pipeline. - Supported judge modes: `deterministic`, `custom`, and `llm_judge`. - The optimizer uses the repository root `.env` `COPILOT_MODEL` setting for inference, gradient, and apply-edit requests. - By default the optimized prompt is returned in the JSON result and CLI stdout without writing files. - The source prompt is only overwritten when `in_place` is requested. - `output` writes a separate optimized prompt file when requested. - `report` writes a JSON report only when requested. - `scripts/train.py` can run a Microsoft Trace self-training loop over one or more explicit prompt/train/val cases to tune the single-shot optimize policy. - Require at least one row in both `train` and `val`. - The repository's authored skill eval cases live under `evals/evals.json` next to the prompt or skill. - The generator utility `scripts/generate_jsonl.py` can create explicit `train.jsonl` and `val.jsonl` files from CSV input. - Dataset generation and conversion belong outside this runtime. Use [skills/trainer-research/SKILL.md](../trainer-research/SKILL.md) and [skills/trainer-synthesize/SKILL.md](../trainer-synthesize/SKILL.md) before calling `trainer-optimize` when explicit JSONL files do not exist yet. ## Process 1. Parse the `/trainer-optimize` arguments from the user's message. 2. Resolve explicit `train` and `val` dataset paths. 3. If datasets still do not exist, stop and report the missing explicit dataset arguments or paths. 4. Validate that all resolved file paths exist and are readable. 5. Read the prompt file and inspect a sample row from the training dataset. 6. Verify the train and validation files each contain at least one JSON object row. 7. Extract template placeholders and verify they are satisfiable from the sample task schema. 8. Keep evaluator-only fields such as `expected`, `expected_json`, `reference`, `criteria`, and `scoring` out of the prompt rendering path. 9. If the prompt or dataset shape is incompatible, stop and explain the mismatch clearly. 10. Run `scripts/run_optimize.py` with the resolved arguments. 11. Return the optimized prompt in JSON/stdout unless explicit persistence flags request file output. 12. Report the optimized prompt path, report path, and optimization metadata from the JSON result. ## Algorithm guidance - Default to `apo` for most prompt-file optimization tasks. - Use `verl` when the user explicitly wants the alternative algorithm or when repo policy prefers it. - Prefer `deterministic` when exact or rule-based scoring is available. - Use `custom` for normalization or schema-based scoring. - Use `llm_judge` only when deterministic or custom scoring is not a good fit. ## Output Modes The runtime produces a single optimized prompt result. Persistence is explicit: - **Default**: return the optimized prompt in JSON/stdout only. - **`output` provided**: write a separate optimized prompt file. - **`in_place` enabled**: overwrite the source `prompt_file`. - **`report` provided**: write a JSON report file. If a higher-level workflow wants to compare multiple optimize results, that comparison belongs outside this skill. ## Dataset format Each line in the JSONL file must be a JSON object. See `references/dataset-format.md` for details. Recommended form: ```json {"input": "user request", "expected": "ideal answer", "scoring": "exact_match"} ``` Repository-authored eval convention: ```text skills/ trainer-optimize/ SKILL.md evals/ evals.json files/ ``` For APO runs, keep `train.jsonl` and `val.jsonl` as explicit dataset files and pass them on the command line. Use `evals/evals.json` for authored evaluation cases, and keep runtime-only optimization output separate from the published eval manifest. Trace self-training: ```bash python scripts/train.py \ --prompt-file ../../examples/first-run/prompts/classify_support.md \ --train-file ../../examples/first-run/datasets/train.jsonl \ --val-file ../../examples/first-run/datasets/val.jsonl \ --epochs 2 \ --report-file /tmp/trace-train-report.json ``` ## Outputs After a successful run, the JSON result always contains the optimized prompt text and optimization metadata. Files are written only when requested via `output`, `report`, or `in_place`. `scripts/train.py` writes a JSON training report only when `--report-file` is passed; otherwise it prints the report to stdout. ## Dashboard During an active optimization run, Agent Lightning starts a local dashboard server at `http://localhost:4747`. - Open the dashboard root URL in a browser while the run is active. - The dashboard uses `http://localhost:4747/v1/agl/health` for health checks and API connectivity. - Available pages include `Rollouts`, `Resources`, `Traces`, `Runners`, and `Settings`. - If the UI shows `Offline`, verify the backend base URL in `Settings` and confirm the optimization process is still running. ## Placeholder validation Template placeholders such as `{input}` must match keys the optimizer can validate from the task rows. Double-brace escapes like `{{literal}}` are ignored. If any placeholder is missing from the dataset schema, stop and explain the mismatch to the user. ## Error cases - Missing or unreadable files → report the specific path and stop - Empty train or val dataset → require at least one task row in each - Placeholder mismatch → list the unmatched placeholders and the actual dataset fields - Missing dataset paths → report the missing explicit arguments or unreadable paths and stop - Unsupported judge mode → explain the supported modes: `deterministic`, `custom`, and `llm_judge` - Unsupported algorithm → explain the supported algorithms: `apo` and `verl` ## Running the script ```bash python scripts/run_optimize.py \ --prompt-file <path> \ [--train-file <path>] \ [--val-file <path>] \ [--iterations 3] \ [--output-file <path>] \ [--report-file <path>] \ [--beam-width 4] \ [--branch-factor 4] \ [--n-runners 4] \ [--algorithm apo|verl] \ [--judge-mode deterministic|custom|llm_judge] \ [--judge-prompt-file assets/judge-default.md] \ [--debug-only] \ [--in-place] ``` Use `--help` if you need to confirm available flags in the current script version. Copilot `.env` example at the repository root: ```dotenv COPILOT_MODEL=default ``` Use [/.env.sample](/workspaces/copilot-apo/.env.sample) as the canonical template for the supported Copilot setting. See `references/dataset-format.md` for the dataset contract and `assets/examples.md` for worked examples.