# Llm4ad Task Builder > Use when a user wants to build an LLM4AD_Next task package — a runnable directory that lets the LLM4AD platform evolve an algorithm for their problem. Covers what files a task package contains, each file's contract, and how the package is run on the LLM4AD platform. - Skill: `optima-cityu/llm4ad-task-builder` (Agent Skill, multi-file: 20 files) - Install (CLI): `npx skillmds@latest add optima-cityu/llm4ad-task-builder` - Raw SKILL.md: https://api.skillmd.com/api/skills/optima-cityu/llm4ad-task-builder/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Productivity - Author: optima-cityu (https://skillmd.com/u/optima-cityu) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/optima-cityu/llm4ad-task-builder --- # Building an LLM4AD_Next Task Package ## What LLM4AD_Next is LLM4AD_Next is an automated algorithm-design platform: it combines an LLM with evolutionary optimization to discover and improve algorithms automatically. The user describes a problem (e.g. "evolve a solver for the Traveling Salesman Problem"), marks the code region to evolve with `# EVOLVE_START` / `# EVOLVE_END`, and the platform repeatedly asks an LLM to propose better code inside that region, scores each candidate with a custom evaluator, and keeps the best across generations. Better algorithms emerge through guided search rather than manual trial and error. Your job with this skill: turn a user's problem into a **complete, runnable task package**, then verify it actually runs before handing it over. ## What a task package is A task package is a self-contained directory with everything the platform needs to evolve algorithms for one problem: | File | Purpose | |---|---| | `config.yaml` | Master config for the whole pipeline (evolution params, providers, evaluator, dataset). This is what gets run. | | `_evaluator.py` | A `BaseEvaluator` subclass that runs a candidate algorithm on test data and returns a score + metrics. Defines what "better" means. | | `/.py` | The algorithm, with the function to evolve wrapped in `# EVOLVE_START` / `# EVOLVE_END`. Reads input as a JSON CLI arg, prints a JSON result. | | `requirements.txt` | Third-party dependencies (auto-generated by build engine based on task description and code analysis). | | `debug_run.py` | Runs the full pipeline once (`LLM4AD("config.yaml").run()`) — a smoke test. | | `test_evaluator.py` | Loads the evaluator via the dispatcher to confirm it imports and is wired correctly. | | `data/sample/*.json` | 2-3 small test instances the evaluator scores algorithms against. | ## The contracts each file must satisfy **Algorithm file** (`/.py`): - The algorithm directory is the one named in `version_control.local_path`; the platform copies it into a git worktree for each candidate and edits only the code between the markers. - The function to evolve is wrapped exactly in `# EVOLVE_START` and `# EVOLVE_END` comment markers (the platform only edits code between them). - It reads its input as a single JSON string from `sys.argv[1]`, and prints a JSON result to stdout — so the evaluator can run it as a subprocess. - The file name must match the name the evaluator looks for (see below). In the TSP example both sides use `solve.py`. **Evaluator** (`_evaluator.py`): - **Before writing evaluator code**, read `reference/api-contract.md` for the complete BaseEvaluator contract and common pitfalls. - Subclasses `BaseEvaluator`, decorated `@BaseEvaluator.register("")`, with a no-argument `__init__(self)` (the base `__init__` takes no config). - Implements `metrics` (list of `Metric(name, type=MetricType.MINIMIZE, weight, description)` — note `MetricType.MINIMIZE`, not bare `MINIMIZE`), `name`, and `async def evaluate(self, cfg: EvalContext) -> EvaluationResult`. - `cfg: EvalContext` carries `cfg.project_root` (the candidate's worktree dir), `cfg.data_path` (the current data file), and `cfg.timeout` (seconds per instance). - `evaluate` locates the algorithm file **by hardcoded name** under `cfg.project_root` (e.g. `Path(cfg.project_root) / "solve.py"`), runs it as a subprocess on `cfg.data_path`, parses its JSON output, validates it, and returns `EvaluationResult(score, metrics, success, ...)` (negative cost for a minimization objective). That hardcoded name **must** equal the algorithm file's actual name — this is the most common wiring mistake. **config.yaml** (10 sections — generated programmatically, not hand-written): - `providers` use `${LLM_BASE_URL}` / `${LLM_API_KEY}` / `${LLM_MODEL}` env placeholders (the platform fills them in). - The evaluator `module` reference (`.py:`), the `metrics` list, the `dataset` path (`data/sample`), and the coder `prompt_template`'s EVOLVE block must all be consistent with the other files. **debug_run.py** / **test_evaluator.py**: standard boilerplate that runs `config.yaml` end-to-end and loads the evaluator, respectively. See `reference/api-contract.md` for the BaseEvaluator contract and `reference/templates/` for two complete, copyable template packages demonstrating different problem patterns (see `reference/templates/README.md`). ## The information to gather before building A correct package needs these decided (ask the user; let the agent fill sensible defaults where the user has no preference): 1. **Problem description** — what is being optimized. (Required — ask.) 2. **Function/algorithm to evolve + input/output format** — ask if the user has any constraints; if not, the agent designs it. 3. **Evaluation metric(s)** — what to minimize/maximize. (Required — ask clearly.) 4. **Reuse existing code/evaluator?** — user provides existing code, or generate from scratch. (Ask.) 5. **Data source** — user provides data, or generate sample instances. (Ask.) 6. **Programming language** — default Python. (Offer as a choice.) 7. **Project name** — the agent proposes one from the description; confirm with user. ## Build workflow 1. Understand the problem and the 7 items above (inspect any user-provided code/data). 2. Write the algorithm file first; verify it runs standalone and prints valid JSON. 3. Write 2-3 sample data files under `data/sample/`. 4. Write the evaluator; keep its metrics consistent with `config.yaml`. 5. Write `config.yaml`, `debug_run.py`, `test_evaluator.py`. 6. **Self-test (required):** run `test_evaluator.py` (evaluator loads — no LLM, no network, cheap and always runnable). Then run `debug_run.py`, which executes the full pipeline and **does call the LLM**: it needs the `LLM_BASE_URL` / `LLM_API_KEY` / `LLM_MODEL` env vars set (the config uses these placeholders) and consumes tokens over the network. Read every error, fix the relevant file, re-run. Not done until both run cleanly. If no provider credentials are available, say so — a missing-credential failure of `debug_run.py` is not a package defect. ## Automated algorithm design methods (reference — only when the user asks) Always build new task packages with the default **Island GA** method. Do NOT proactively suggest switching to another method — let the user bring it up. When the user asks about methods or explicitly requests a switch, help them using the reference below. After applying a switch, ask whether they want to continue adjusting parameters or are satisfied. Each method requires a matching pair of `evolution.type` + `planner.type`. Switching methods means **replacing the entire `evolution` section` (parameters differ per method) and updating `planner.type`. A fresh run is required — you cannot resume a prior run after changing the evolution type. --- **Island GA** (default — use this for all new builds) ```yaml evolution: type: "island_ga" max_generations: 50 num_islands: 5 # number of parallel populations island_population_size: 20 # individuals per island mutation_rate: 0.3 crossover_rate: 0.5 tournament_size: 3 early_stop_patience: 10 migration_interval: 5 # gens between cross-island exchanges migration_rate: 0.1 # fraction of individuals migrated migration_strategy: "best" # best | random | elite | worst migration_topology: "ring" # ring | full | hierarchy | mesh parallel_islands: true # run islands concurrently planner: type: "llm_evolution" ``` **EoH** — Evolution of Heuristics *(ICML 2024)* Single-objective, rank-based truncation. Runs E1 (and optionally E2/M1/M2) operators each generation. ```yaml evolution: type: "eoh" max_generations: 50 population_size: 5 # top-k individuals kept after truncation selection_num: 2 # parents selected per E1/E2 crossover max_sample_nums: 100 # LLM call budget cap for the run num_samplers: 1 # parallel candidates per operator per gen use_e2_operator: true # enable E2 (backbone-motivated crossover) use_m1_operator: true # enable M1 (structural mutation) use_m2_operator: true # enable M2 (parameter mutation) seed_path: null # optional path to a seed heuristic file planner: type: "eoh_evolution" ``` **ReEvo** — Reflective Evolution *(NeurIPS 2024)* Adds two reflection signals to genetic search: short-term (comparing worse/better parent pairs → better crossover) and long-term (accumulated history → better elite mutation). ```yaml evolution: type: "reevo" max_generations: 50 population_size: 8 # individuals in the population mutation_rate: 0.5 # fraction of pop used for elite mutations per gen max_sample_nums: 100 # LLM call budget cap for the run num_samplers: 1 # parallel crossover candidates per step seed_path: null # optional path to a seed heuristic file planner: type: "reevo_evolution" ``` **MCTS-AHD** — Monte Carlo Tree Search for AHD *(ICML 2025)* Tree search over algorithm space: selects nodes via UCT, expands with e1/e2/m1/m2/s1 operators. Not generational; `max_generations` is interpreted as MCTS iterations. ```yaml evolution: type: "mcts_ahd" max_generations: 1000 # MCTS iterations (not generations) init_size: 4 # initial child nodes expanded under root population_size: 10 # active algorithm pool size selection_num: 2 # parents per e1/e2 operator max_sample_nums: 100 # LLM call budget cap for the search alpha: 0.5 # UCT progressive-widening parameter lambda_0: 0.1 # UCT exploration constant (decays with budget) max_depth: 10 # maximum tree depth seed_path: null # optional path to a seed heuristic file planner: type: "mcts_ahd_evolution" ``` ### Switching checklist (only when the user actively requests it) 1. **Replace `evolution` section** — remove the old method's specific params, write the new method's complete section from the templates above. Do NOT carry over params that don't exist in the new method's schema (e.g. `num_islands` is IGA-only). 2. **Update `planner.type`** — must match the new method. 3. **Keep everything else** — `providers`, `evaluator`, `coder`, `dataset`, `version_control` stay unchanged. 4. **Warn the user** — switching evolution type means a fresh run; you cannot resume an old checkpoint under a different method. 5. **Re-verify** — after editing `config.yaml`, re-run `debug_run.py` to confirm the new method loads correctly. 6. **Ask whether to continue** — after the switch is applied and verified, ask the user if they want to adjust any parameters further or are satisfied. ## How the package is used on the LLM4AD platform Once the package exists, the user runs it to evolve their algorithm. Two paths: **CLI:** ```bash llm4ad run path/to/config.yaml # resume an interrupted run: pass a real checkpoint file written under # ./runs///checkpoints/ (Island GA names them # iga_checkpoint_gen__.json; MEoH names them meoh_.json). llm4ad run config.yaml --resume ./runs///checkpoints/iga_checkpoint_gen_5_a1b2c3d4.json ``` It launches the evolution pipeline and prints per-generation progress (best score each generation). **Web platform (this project's Web UI):** 1. Create or open a task (the package's `config.yaml` + files back it). 2. Configure an LLM provider (the user's own key/model — OpenAI-compatible or Anthropic). The package's `config.yaml` references providers by the platform's env placeholders; the platform injects the real credentials at run time. 3. Start the evolution run; monitor progress in the browser — current generation, best individual's score, metrics, and logs — in real time. 4. When it finishes, inspect the best evolved algorithm and its score. **Tuning after generation (optional):** users often adjust `config.yaml` evolution parameters to trade cost vs. quality — e.g. `max_generations`, `num_islands`, `island_population_size`, `mutation_rate` / `crossover_rate`, `early_stop_patience`, the provider `model`, and `evaluator.timeout`. **Where results go:** each run writes to `./runs/{project}/{run_id}/`: - `best/code/` — the evolved algorithm source (the main deliverable) - `best/metadata.json`, `best/summary.txt` — score, generation, metrics - `state/evolution_state.json` — full generation history (drives the Web UI dashboards) - `logs/llm4ad.log` — full execution log for troubleshooting - `checkpoints/` — snapshots to resume from ## Completion criteria The package is done only when `test_evaluator.py` loads the evaluator AND `debug_run.py` runs without raising. Then summarize what was built (files + the 7 decisions) and the verification result. After a successful build, always close by conveying this message (translate to the user's language; do NOT proactively recommend a specific method — just mention that the option exists): > The task package has been built and verified. I can help you choose the > evolution method (such as EoH, ReEvo, or MCTS-AHD), and adjust evolution > parameters (such as max_generations, population size, etc.), or feel free > to let me know if you have other needs!