llm4ad-task-builder

name	llm4ad-task-builder
description	Use when a user wants to build an LLM4AD_Next task package — a runnable directory that lets the LLM4AD platform evolve an algorithm for their problem. Covers what files a task package contains, each file's contract, and how the package is run on the LLM4AD platform.

Building an LLM4AD_Next Task Package

What LLM4AD_Next is

LLM4AD_Next is an automated algorithm-design platform: it combines an LLM with evolutionary optimization to discover and improve algorithms automatically. The user describes a problem (e.g. "evolve a solver for the Traveling Salesman Problem"), marks the code region to evolve with # EVOLVE_START / # EVOLVE_END, and the platform repeatedly asks an LLM to propose better code inside that region, scores each candidate with a custom evaluator, and keeps the best across generations. Better algorithms emerge through guided search rather than manual trial and error.

Your job with this skill: turn a user's problem into a complete, runnable task package, then verify it actually runs before handing it over.

What a task package is

A task package is a self-contained directory with everything the platform needs to evolve algorithms for one problem:

File	Purpose
config.yaml	Master config for the whole pipeline (evolution params, providers, evaluator, dataset). This is what gets run.
<name>_evaluator.py	A BaseEvaluator subclass that runs a candidate algorithm on test data and returns a score + metrics. Defines what "better" means.
<algo_dir>/<algo>.py	The algorithm, with the function to evolve wrapped in # EVOLVE_START / # EVOLVE_END. Reads input as a JSON CLI arg, prints a JSON result.
requirements.txt	Third-party dependencies (auto-generated by build engine based on task description and code analysis).
debug_run.py	Runs the full pipeline once (LLM4AD("config.yaml").run()) — a smoke test.
test_evaluator.py	Loads the evaluator via the dispatcher to confirm it imports and is wired correctly.
data/sample/*.json	2-3 small test instances the evaluator scores algorithms against.

The contracts each file must satisfy

Algorithm file (<algo_dir>/<algo>.py):

• The algorithm directory is the one named in version_control.local_path; the platform copies it into a git worktree for each candidate and edits only the code between the markers.
• The function to evolve is wrapped exactly in # EVOLVE_START and # EVOLVE_END comment markers (the platform only edits code between them).
• It reads its input as a single JSON string from sys.argv[1], and prints a JSON result to stdout — so the evaluator can run it as a subprocess.
• The file name must match the name the evaluator looks for (see below). In the TSP example both sides use solve.py.

Evaluator (<name>_evaluator.py):

• Before writing evaluator code, read reference/api-contract.md for the complete BaseEvaluator contract and common pitfalls.
• Subclasses BaseEvaluator, decorated @BaseEvaluator.register("<name>"), with a no-argument __init__(self) (the base __init__ takes no config).
• Implements metrics (list of Metric(name, type=MetricType.MINIMIZE, weight, description) — note MetricType.MINIMIZE, not bare MINIMIZE), name, and async def evaluate(self, cfg: EvalContext) -> EvaluationResult.
• cfg: EvalContext carries cfg.project_root (the candidate's worktree dir), cfg.data_path (the current data file), and cfg.timeout (seconds per instance).
• evaluate locates the algorithm file by hardcoded name under cfg.project_root (e.g. Path(cfg.project_root) / "solve.py"), runs it as a subprocess on cfg.data_path, parses its JSON output, validates it, and returns EvaluationResult(score, metrics, success, ...) (negative cost for a minimization objective). That hardcoded name must equal the algorithm file's actual name — this is the most common wiring mistake.

config.yaml (10 sections — generated programmatically, not hand-written):

• providers use ${LLM_BASE_URL} / ${LLM_API_KEY} / ${LLM_MODEL} env placeholders (the platform fills them in).
• The evaluator module reference (<file>.py:<ClassName>), the metrics list, the dataset path (data/sample), and the coder prompt_template's EVOLVE block must all be consistent with the other files.

debug_run.py / test_evaluator.py: standard boilerplate that runs config.yaml end-to-end and loads the evaluator, respectively.

See reference/api-contract.md for the BaseEvaluator contract and reference/templates/ for two complete, copyable template packages demonstrating different problem patterns (see reference/templates/README.md).

The information to gather before building

A correct package needs these decided (ask the user; let the agent fill sensible defaults where the user has no preference):

1. Problem description — what is being optimized. (Required — ask.)
2. Function/algorithm to evolve + input/output format — ask if the user has any constraints; if not, the agent designs it.
3. Evaluation metric(s) — what to minimize/maximize. (Required — ask clearly.)
4. Reuse existing code/evaluator? — user provides existing code, or generate from scratch. (Ask.)
5. Data source — user provides data, or generate sample instances. (Ask.)
6. Programming language — default Python. (Offer as a choice.)
7. Project name — the agent proposes one from the description; confirm with user.

Build workflow

1. Understand the problem and the 7 items above (inspect any user-provided code/data).
2. Write the algorithm file first; verify it runs standalone and prints valid JSON.
3. Write 2-3 sample data files under data/sample/.
4. Write the evaluator; keep its metrics consistent with config.yaml.
5. Write config.yaml, debug_run.py, test_evaluator.py.
6. Self-test (required): run test_evaluator.py (evaluator loads — no LLM, no network, cheap and always runnable). Then run debug_run.py, which executes the full pipeline and does call the LLM: it needs the LLM_BASE_URL / LLM_API_KEY / LLM_MODEL env vars set (the config uses these placeholders) and consumes tokens over the network. Read every error, fix the relevant file, re-run. Not done until both run cleanly. If no provider credentials are available, say so — a missing-credential failure of debug_run.py is not a package defect.

Automated algorithm design methods (reference — only when the user asks)

Always build new task packages with the default Island GA method. Do NOT proactively suggest switching to another method — let the user bring it up.

When the user asks about methods or explicitly requests a switch, help them using the reference below. After applying a switch, ask whether they want to continue adjusting parameters or are satisfied.

Each method requires a matching pair of evolution.type + planner.type. Switching methods means **replacing the entire evolution section(parameters differ per method) and updatingplanner.type`. A fresh run is required — you cannot resume a prior run after changing the evolution type.

---

Island GA (default — use this for all new builds)

evolution:
  type: "island_ga"
  max_generations: 50
  num_islands: 5              # number of parallel populations
  island_population_size: 20  # individuals per island
  mutation_rate: 0.3
  crossover_rate: 0.5
  tournament_size: 3
  early_stop_patience: 10
  migration_interval: 5       # gens between cross-island exchanges
  migration_rate: 0.1         # fraction of individuals migrated
  migration_strategy: "best"  # best | random | elite | worst
  migration_topology: "ring"  # ring | full | hierarchy | mesh
  parallel_islands: true      # run islands concurrently
planner:
  type: "llm_evolution"

EoH — Evolution of Heuristics (ICML 2024)
Single-objective, rank-based truncation. Runs E1 (and optionally E2/M1/M2) operators each generation.

evolution:
  type: "eoh"
  max_generations: 50
  population_size: 5          # top-k individuals kept after truncation
  selection_num: 2            # parents selected per E1/E2 crossover
  max_sample_nums: 100        # LLM call budget cap for the run
  num_samplers: 1             # parallel candidates per operator per gen
  use_e2_operator: true       # enable E2 (backbone-motivated crossover)
  use_m1_operator: true       # enable M1 (structural mutation)
  use_m2_operator: true       # enable M2 (parameter mutation)
  seed_path: null             # optional path to a seed heuristic file
planner:
  type: "eoh_evolution"

ReEvo — Reflective Evolution (NeurIPS 2024)
Adds two reflection signals to genetic search: short-term (comparing worse/better parent pairs → better crossover) and long-term (accumulated history → better elite mutation).

evolution:
  type: "reevo"
  max_generations: 50
  population_size: 8          # individuals in the population
  mutation_rate: 0.5          # fraction of pop used for elite mutations per gen
  max_sample_nums: 100        # LLM call budget cap for the run
  num_samplers: 1             # parallel crossover candidates per step
  seed_path: null             # optional path to a seed heuristic file
planner:
  type: "reevo_evolution"

MCTS-AHD — Monte Carlo Tree Search for AHD (ICML 2025)
Tree search over algorithm space: selects nodes via UCT, expands with e1/e2/m1/m2/s1 operators. Not generational; max_generations is interpreted as MCTS iterations.

evolution:
  type: "mcts_ahd"
  max_generations: 1000       # MCTS iterations (not generations)
  init_size: 4                # initial child nodes expanded under root
  population_size: 10         # active algorithm pool size
  selection_num: 2            # parents per e1/e2 operator
  max_sample_nums: 100        # LLM call budget cap for the search
  alpha: 0.5                  # UCT progressive-widening parameter
  lambda_0: 0.1               # UCT exploration constant (decays with budget)
  max_depth: 10               # maximum tree depth
  seed_path: null             # optional path to a seed heuristic file
planner:
  type: "mcts_ahd_evolution"

Switching checklist (only when the user actively requests it)

1. Replace evolution section — remove the old method's specific params, write the new method's complete section from the templates above. Do NOT carry over params that don't exist in the new method's schema (e.g. num_islands is IGA-only).
2. Update planner.type — must match the new method.
3. Keep everything else — providers, evaluator, coder, dataset, version_control stay unchanged.
4. Warn the user — switching evolution type means a fresh run; you cannot resume an old checkpoint under a different method.
5. Re-verify — after editing config.yaml, re-run debug_run.py to confirm the new method loads correctly.
6. Ask whether to continue — after the switch is applied and verified, ask the user if they want to adjust any parameters further or are satisfied.

How the package is used on the LLM4AD platform

Once the package exists, the user runs it to evolve their algorithm. Two paths:

CLI:

llm4ad run path/to/config.yaml
# resume an interrupted run: pass a real checkpoint file written under
# ./runs/<project>/<run_id>/checkpoints/ (Island GA names them
# iga_checkpoint_gen_<N>_<hash>.json; MEoH names them meoh_<N>.json).
llm4ad run config.yaml --resume ./runs/<project>/<run_id>/checkpoints/iga_checkpoint_gen_5_a1b2c3d4.json

It launches the evolution pipeline and prints per-generation progress (best score each generation).

Web platform (this project's Web UI):

1. Create or open a task (the package's config.yaml + files back it).
2. Configure an LLM provider (the user's own key/model — OpenAI-compatible or Anthropic). The package's config.yaml references providers by the platform's env placeholders; the platform injects the real credentials at run time.
3. Start the evolution run; monitor progress in the browser — current generation, best individual's score, metrics, and logs — in real time.
4. When it finishes, inspect the best evolved algorithm and its score.

Tuning after generation (optional): users often adjust config.yaml evolution parameters to trade cost vs. quality — e.g. max_generations, num_islands, island_population_size, mutation_rate / crossover_rate, early_stop_patience, the provider model, and evaluator.timeout.

Where results go: each run writes to ./runs/{project}/{run_id}/:

• best/code/ — the evolved algorithm source (the main deliverable)
• best/metadata.json, best/summary.txt — score, generation, metrics
• state/evolution_state.json — full generation history (drives the Web UI dashboards)
• logs/llm4ad.log — full execution log for troubleshooting
• checkpoints/ — snapshots to resume from

Completion criteria

The package is done only when test_evaluator.py loads the evaluator AND debug_run.py runs without raising. Then summarize what was built (files + the 7 decisions) and the verification result.

After a successful build, always close by conveying this message (translate to the user's language; do NOT proactively recommend a specific method — just mention that the option exists):

The task package has been built and verified. I can help you choose the evolution method (such as EoH, ReEvo, or MCTS-AHD), and adjust evolution parameters (such as max_generations, population size, etc.), or feel free to let me know if you have other needs!