Slurm Apptainer

Curation-Bench Protocol (skill-grounded)

You are an AI agent working inside a dataset-curation benchmark. Your goal is to curate training data based on the task to maximize downstream evaluation performance after fine-tuning.

The benchmark manages the suite lifecycle, training, and evaluation. You manage the data curation strategy.

Read only this file as protocol. No other protocol files are available to you.

Filesystem layout

• /workspace — your working directory and the only place you may write. It holds curate.py (the one file you edit) and anything you choose to create (output/curated/, your own results.tsv, a .git repo, notes).
• /data/raw — the curation input dataset (read-only).
• /workspace/skills/<category>/<paper-slug>/SKILL.md — the paper-derived skill-card library (read-only); ground each strategy in these (see Strategy must be skill-grounded).
• /code/benchmark, /code/train, /code/eval, /code/scripts — benchmark / training / eval / script source, read-only, for reference and for the train/eval stage scripts.
• /PROTOCOL.md — this file.

Run train/eval stages through /code/scripts/slurm/* exactly as shown below. Do not override BENCH_REPO_SCRIPT_ROOT or BENCH_REPO_COMPUTE; the launcher sets them so wrappers can read scripts from /code while Slurm executes from the compute-visible repository checkout.

The benchmark owns the per-iteration run directory (run_dir, reported by next) where it writes finetune/ and eval/ outputs and, on each completed iteration, a snapshot of your /workspace.

Session initialization

First of all, ground yourself in the research skills for this session:

• ls /workspace/skills/*/ to see entries across all four categories (data-acquisition, data-curation, data-selection, data-synthesis).
• Pick at least 3 skills per category whose titles suggest relevance to VLM instruction-tuning data selection.
• Read those SKILL.md files in full before touching /workspace/curate.py. Spawn Agent/Explore tools in parallel to scan quickly. Listing titles is not enough — you must understand the procedures so you can adapt them on budget.

Then start the workflow immediately.

BENCHMARK_CURRENT_TASK tells you which task you own — only that task, not any other task in the suite.

echo "My assigned task: $BENCHMARK_CURRENT_TASK"

If "$BENCHMARK_ROOT_VENV/bin/datacuration-bench" next ever returns a task_id different from $BENCHMARK_CURRENT_TASK, exit immediately — that's another agent session's responsibility.

At the start of every session, your working directory is /workspace. If you want a clean git history for your own bookkeeping, you may (optionally) reset it:

cd /workspace
rm -rf .git
git init
git add curate.py
git commit -m "fresh start: $BENCHMARK_CURRENT_TASK"

git is entirely your own choice — the benchmark snapshots all of /workspace each iteration regardless. If you use git, manage any ignore rules yourself and do not commit output/, .venv/, *.log, runs/, checkpoints, or other generated artifacts.

Setup

To set up a new experiment:

1. Read the task: "$BENCHMARK_ROOT_VENV/bin/datacuration-bench" task — task definition: task_id, goal, target_rows, model_key, target_evals, strategy_timeout_seconds, dataset_path, default_submission_path.
2. Verify vendor venvs (see "Venv setup" below).
3. Initialize results.tsv: create /workspace/results.tsv with the header row (see Logging results).
4. Proceed.

After setup, start the workflow immediately.

The Dataset

The LLaVA-665K dataset is a multimodal instruction-following dataset with ~665K samples. Each sample has:

• images: list of PIL images
• texts: list of conversation turns (user/assistant pairs)

Inspect ds.column_names before relying on optional metadata. The current Container mounts expose images, texts, and original_id; your submitted dataset must include at least images and texts.

You are selecting a subset (of size target_rows) that maximizes downstream VLM benchmark performance after fine-tuning on the selected data.

IMPORTANT: The dataset is on a read-only mount. Avoid operations that write cache files beside the input dataset. Prefer column-level access to build index lists, then use ds.select(indices).

Strategy must be skill-grounded

Every iteration of /workspace/curate.py MUST be grounded in at least one SKILL.md under /workspace/skills/. Before editing /workspace/curate.py:

1. Name a failure mode from the last result (e.g., "model predicts wrong MC option — suggests low-signal examples dominate training", not "MMVet is low").
2. Pick 1–3 SKILL.md files relevant to that failure mode. Read them in full (not just the title). Use Agent/Explore tools in parallel if you need to scan many.
3. State in the commit message: <skill-dirname> — <one-line adaptation>. Example: el2n-…-2107-07075v2 — proxy-loss scoring via forward pass on LLaVA-base to keep high-error samples.

Skills fall into four buckets (see skills/data-{acquisition,curation,selection,synthesis}/). Before repeating a bucket, attempt at least one iteration from each of the other three.

Forbidden strategies (CANNOT do)

These require zero research and will be rejected regardless of accuracy:

• Tweaking per-subset sample counts based on subset name (coco/gqa/ocr_vqa/textvqa/…) — "empirical adjustment on domain ratios."
• Length / word-count / turn-count thresholds (e.g., "short answer < N chars", SHORT_ANSWER_THRESHOLD).
• Regex keyword gates on question text (e.g., MATH_WORDS, REASONING_WORDS).
• Rephrase prompt tuning without a skill justification.
• Any strategy whose commit message cites a skill as "inspired by" vibes rather than an adapted procedure from a specific paper.

If your next idea fits the list above, discard it and pick a skill instead.

Venv setup (one-time)

The cluster uses scratch-backed venvs prepared before launch. Do not run uv sync, uv pip install, or uv add unless the operator explicitly asks you to repair an environment. The normal agent workflow only verifies the existing venvs and runs commands through the scratch venv executables.

PROJECT_ROOT=$(pwd)
test -x "${BENCHMARK_ROOT_VENV:?}/bin/python"
test -x "${BENCHMARK_TRAIN_VENV:?}/bin/python"
test -x "${BENCHMARK_EVAL_VENV:?}/bin/python"
"$BENCHMARK_ROOT_VENV/bin/datacuration-bench" --help >/dev/null

Pre-init baseline

The pre-init baseline is the base model (no finetune) evaluated on iteration 0. It is deterministic per (base model, benchmark set) and is normally pre-seeded: your first "$BENCHMARK_ROOT_VENV/bin/datacuration-bench" next returns iteration 1, and the baseline artifacts already exist at <run_dir>/../iter_000/eval/results/ (results.json + per-question <model_name>/*.xlsx). Before your first curation, read those baseline predictions to see where the base model fails. Never run eval0 yourself when next returns iteration 1.

Fallback (not seeded): only if next returns status baseline_eval_activated and stage eval_base, run eval as a blocking foreground call:

/code/scripts/slurm/run_judge_eval_srun.sh 2>&1

This evaluates the base model path from the selected profile, not a fine-tuned checkpoint. Do not run VLMEvalKit directly, do not run datacuration-bench eval directly in this container, and do not add inline API credentials.

After the eval stage reports {"status": "completed", ...}, call "$BENCHMARK_ROOT_VENV/bin/datacuration-bench" next again to activate iter1.

Running an experiment

Each iteration has three stages: curate, train, evaluate. You run each stage via the datacuration-bench CLI — one stage and one iter at a time; no overlapping stages within or across iters. The harness owns the run directory and all paths.

Activate the next iteration

"$BENCHMARK_ROOT_VENV/bin/datacuration-bench" next

Returns JSON with status, target_key, task_id, iteration, run_dir, task_goal, default_submission_path, dataset_path.

• status == "target_activated" and task_id == $BENCHMARK_CURRENT_TASK: proceed.
• status == "task_done": exit.
• Anything else: exit immediately.

Stage 1: Curate

Your working directory is /workspace, and curate.py is the one file you edit. Use the dataset_path from the JSON (/data/raw) as input, and write the curated HuggingFace dataset (save_to_disk, with images and texts columns, row count == target_rows) to default_submission_path (/workspace/output/curated). The reference script takes three positional args:

"$BENCHMARK_ROOT_VENV/bin/python" /workspace/curate.py <dataset_path> <default_submission_path> <target_rows>

Avoid row-by-row Python iteration on the dataset — e.g. for i in range(len(ds)), for row in ds:, etc.

On iter1 (before editing /workspace/curate.py), you MUST read the seeded pre-init baseline so your first strategy targets the base model's weaknesses: <run_dir>/../iter_000/eval/results/results.json and the per-question files under <run_dir>/../iter_000/eval/results/<model_name>/*.xlsx.

For iter2+ (before editing /workspace/curate.py), you MUST read in full:

• The full /workspace/results.tsv (every row, not just the tail) — your own running log.
• EVERY prior strategy snapshot: <run_dir>/../iter_<NNN>/workspace/curate.py for each prior strategy iteration NNN (zero-padded). The baseline iter_000 has no workspace snapshot — only eval results.
• EVERY prior aggregate score: <run_dir>/../iter_<NNN>/eval/results/results.json for each prior iteration NNN, including the baseline iter_000.
• (Optional, recommended) Per-question predictions: <run_dir>/../iter_<NNN>/eval/results/<model_name>/*.xlsx for the previous iteration (and iter_000 for the baseline).

Skipping any of the required reads above is a protocol violation. The benchmark snapshots your /workspace into <run_dir>/../iter_<NNN>/workspace/ at the end of each iteration, so prior curate.py is recoverable there.

If the task has strategy_timeout_seconds, check "$BENCHMARK_ROOT_VENV/bin/datacuration-bench" time-budget frequently — before any long op, after each major step, before submitting. If low, simplify. If strategy_timeout_seconds: null, ignore the timer.

Then audit and submit:

"$BENCHMARK_ROOT_VENV/bin/datacuration-bench" audit --path <default_submission_path>
"$BENCHMARK_ROOT_VENV/bin/datacuration-bench" submit --path <default_submission_path>

Stage 2: Train

Run as a blocking foreground Bash call. Do NOT use run_in_background, &, or nohup. See the Monitoring section.

/code/scripts/slurm/run_bench_stage_srun.sh finetune 2>&1

Success signal: "status": "completed" with model_path printed after training ends. Model files land under <run_dir>/finetune/.

Stage 3: Evaluate

Run as a blocking foreground Bash call. Do NOT use run_in_background, &, or nohup. See the Monitoring section.

/code/scripts/slurm/run_judge_eval_srun.sh 2>&1

Do not run datacuration-bench eval directly in this container. The suite is mounted read-only here, so evaluation must run on the compute node through the stage wrapper above.

VLMEvalKit writes per-benchmark scores to <run_dir>/eval/results/. On success, the target is automatically marked COMPLETED by the harness.

Scoring

After evaluation, extract raw scores from <run_dir>/eval/results/. VLMEvalKit writes per-benchmark result files under <run_dir>/eval/results/<model_name>/ (JSON, xlsx, or CSV). Look for keys like overall, score, accuracy, or avg. The aggregate <run_dir>/eval/results/results.json (keyed by model name) is also available.

For HallusionBench, the score is the average of aAcc, fAcc, and qAcc.

For MMMU_DEV_VAL, use the validation split score (not the dev split score). The eval outputs both; report the val score.

Normalization:

accuracy = mean of (MMVet/100, LLaVABench/100, OCRBench/1000, HallusionBench/100, MMMU_DEV_VAL/100, MathVista_MINI/100, MMStar/100, MMBench/100) across the task's target_evals.

Experimentation rules

What you CAN do:

• Edit /workspace/curate.py — the only file you change. Everything is fair game: selection criteria, filtering, subset balancing, deduplication, quality scoring, diversity sampling, metadata-based selection, etc. You may create helper files under /workspace too.

What you CANNOT do:

• Write anywhere outside /workspace. Everything under /code and /data is read-only (benchmark/train/eval source, task YAMLs, and the input dataset).
• Change training hyperparameters or evaluation benchmarks.
• Run uv sync, uv pip install, or uv add to modify the prepared vendor venvs.

Use discretion for new packages: try what's already available first.

The goal: highest accuracy (average normalized score across MMVet, LLaVABench, OCRBench, HallusionBench, MMMU_DEV_VAL, MathVista_MINI, MMStar, MMBench).

Data fraction: exactly target_rows samples. The submit step rejects any other row count.

Simplicity criterion: all else being equal, simpler is better. A small improvement that adds ugly complexity is not worth it. Removing something and getting equal or better results is a great outcome. Simplicity applies within a given skill-grounded approach (don't over-engineer the implementation), not across approaches.

Pre-init baseline: The baseline eval0 is normally pre-seeded — your first next returns iteration 1 and the baseline artifacts already exist under iter_000/eval/results/. Read them before your first curation; see the Pre-init baseline section. If you maintain a TSV row for the baseline, use commit 0000000 and description pre-init baseline (no finetune). Never re-run eval0: if next returns iteration 1, or eval0 is already completed, or results.tsv already has a pre-init row, move on. (Only when next returns the eval-only iteration 0 target — baseline_eval_activated — do you run it; that is the not-seeded fallback.)

First run (iter1): Always establish your starting reference first by running /workspace/curate.py unchanged. By default it is the random-selection baseline (select target_rows examples from the original dataset), but it may have been seeded with a preconfigured starting strategy at launch (or carry a non-baseline strategy from a previous session). You MUST NOT modify /workspace/curate.py for iter1 — run it exactly as-is, even if it looks non-baseline. If results.tsv already contains an iter1 row, skip it and move on — do not re-run it.

Logging results

When an iter is done, append a row to /workspace/results.tsv (tab-separated, NOT comma-separated). The header row is created in Setup; each iter (including the pre-init baseline) adds one data row.

Header and columns:

commit	accuracy	MMVet	LLaVABench	OCRBench	HallusionBench	MMMU_DEV_VAL	MathVista_MINI	MMStar	MMBench	status	description	skill_ref	failure_mode	next_skill_candidate

1. git commit hash (short, 7 chars) — use 0000000 for the pre-init row
2. accuracy (normalized average, e.g. 0.456700) — 0.000000 for crashes
3. MMVet raw score (e.g. 35.2) — 0.0 for crashes
4. LLaVABench raw score (e.g. 62.1) — 0.0 for crashes
5. OCRBench raw score (e.g. 310.0) — 0.0 for crashes
6. HallusionBench raw score (avg of aAcc, fAcc, qAcc; e.g. 42.5) — 0.0 for crashes
7. MMMU_DEV_VAL raw score (validation split; e.g. 34.0) — 0.0 for crashes
8. MathVista_MINI raw score (e.g. 28.5) — 0.0 for crashes
9. MMStar raw score (e.g. 35.0) — 0.0 for crashes
10. MMBench raw score (dev split Overall; e.g. 65.0) — 0.0 for crashes
11. status: keep (new best accuracy so far), discard (worse), or crash (run failed)
12. short text description of what this iteration did (the adaptation, in words)
13. skill_ref: the SKILL.md directory this iteration adapted (e.g. el2n-…-2107-07075v2). Use none for the baseline and pre-init rows only — every non-baseline iteration must cite a skill. A value of none on a non-baseline row is a rule violation.
14. failure_mode: diagnosis of what the eval output suggests is broken (what signal the model lacks, not which knob to tweak). Example: "model fails compositional reasoning on MMBench — high-variance examples filtered out."
15. next_skill_candidate: which SKILL.md dir you plan to draw from next, and a one-line reason. Must be from a category you haven't used in the last three iterations unless justified.

Example:

commit	accuracy	MMVet	LLaVABench	OCRBench	HallusionBench	MMMU_DEV_VAL	MathVista_MINI	MMStar	MMBench	status	description	skill_ref	failure_mode	next_skill_candidate
0000000	0.420000	32.0	58.0	295.0	40.0	32.5	27.0	33.0	62.0	keep	pre-init baseline (no finetune)	none	none (baseline)	el2n-…-2107-07075v2 — score via proxy forward pass
a1b2c3d	0.456700	35.2	62.1	310.0	42.5	34.0	28.5	35.0	65.0	keep	baseline random 10k	none	none (baseline)	el2n-…-2107-07075v2 — score via proxy forward pass
b2c3d4e	0.478200	37.1	64.3	320.0	44.0	35.5	30.0	36.5	67.0	keep	EL2N proxy-loss top-10k via LLaVA-base forward pass	el2n-…-2107-07075v2	hallucination unchanged — kept hard-but-wrong examples	dataset-cartography-…-2009-10795v2 — separate ambiguous from hard
c3d4e5f	0.412000	30.5	58.2	290.0	38.0	32.0	26.0	33.0	60.0	discard	Data Shapley approximation via leave-one-out loss deltas	data-shapley-…-1904-02868v6	LOO too noisy at 10k sample scale	influence-functions-…-2002-08484v3 — Hessian-free approx
d4e5f6g	0.000000	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	crash	dedup bug caused empty dataset	deita-…-2312-15685v2	crash, not a failure mode	retry with fix

Rows added to results.tsv must have a non-none skill_ref value unless they are the baseline or pre-init baseline. Rows violating this should be audited and rerun with a skill-grounded strategy.

You do not need to copy curate.py snapshots anywhere — the benchmark snapshots all of /workspace (including curate.py) into <run_dir>/../iter_<NNN>/workspace/ automatically when each iteration completes.

The experiment loop

LOOP UNTIL "$BENCHMARK_ROOT_VENV/bin/datacuration-bench" next RETURNS task_done:

1. Check git state (optional, if you use git).
2. "$BENCHMARK_ROOT_VENV/bin/datacuration-bench" next, check status and task_id.
3. Pick a skill. Diagnose the current failure mode from the last iteration's results. From /workspace/skills/, pick 1–3 SKILL.md files matching that failure mode and read them in full. If the last three iterations drew from the same category (data-acquisition / data-curation / data-selection / data-synthesis), pick from a different one. Skip for the pre-init baseline and iter1.
4. Write the adaptation. Name the concrete procedure you will adapt from the chosen skill, and what approximation you will use given the compute budget (e.g., "use base-model forward pass as a proxy since training a reference model is too expensive"). This becomes the commit message: <skill-dirname> — <one-line adaptation>.
5. Modify /workspace/curate.py to implement the adaptation (iter1: run unchanged — see First run). If the change is a subset-ratio/length/regex heuristic, it does not qualify — go back to step 3.
6. (Optional) commit in /workspace with message <skill-dirname> — <one-line adaptation> if you use git. No manual snapshot copying — the harness snapshots /workspace automatically.
7. Run the three stages in sequence (curate → train → evaluate) using the blocking commands above. Avoid dumping large logs into context.
8. Extract results from eval output files.
9. If any stage failed or results are missing: tail the matching stderr to diagnose — finetune failure: tail -n 50 <run_dir>/finetune/train_stderr.txt; eval failure: tail -n 50 <run_dir>/eval/eval_stderr.txt.
10. Record results in TSV. skill_ref must be the SKILL.md directory name (or none only for baseline/pre-init); failure_mode is the diagnosis of what signal the model lacks; next_skill_candidate is the next SKILL.md to draw from, ideally from a category not used in the last three iterations.
11. If accuracy improved (higher): keep the commit ("advance").
12. If accuracy equal or worse: revert /workspace/curate.py to the prior best (e.g. git reset --hard HEAD~1 if you use git). The iter's official eval score is already recorded by the harness; reverting only affects your curate.py starting point for the next iter.

Monitoring

Don't check anything excessively — every tool call re-reads your full context. Keep checks sparse and reasonable regardless of what file or mechanism you're using: tailing logs, listing directories (ls), grepping output files, watching step counts, counting benchmark completions, reading intermediate state, probing background task progress, etc., all count. Don't spawn Monitor watchers to poll progress.

ETA for curation: for data curation operations, always compute and print an ETA, and check proactively while it is running — do not wait on scripts that do not finish quickly without an ETA. If a curation script is running without producing progress/ETA output, kill it, add ETA logging, and re-run. Harness commands (finetune / eval / pre-init baseline) don't need progress monitoring — the harness owns the wall-clock.

TIMED YIELDS ARE NOT COMPLETION: finetune and eval run for many minutes, and your shell may hand control back with a "still running" / timed-out notice before the command has exited. That is not completion. Keep waiting on the same stage until the wrapper command actually returns AND the target's status.json records it done. Do NOT launch the next stage, write anything, or treat the run as finished on a timed yield. "Finetune launched but not yet evaluated and scored" is not done. The only signal the run is over is next returning task_done.

Crashes: if it's something dumb and easy to fix, fix and re-run. If fundamentally broken, log "crash" and move on.

NEVER STOP: once the experiment loop begins, do NOT pause to ask the human if you should continue. Do NOT ask "should I keep going?" or "is this a good stopping point?". The human might be asleep and expects you to continue working indefinitely until "$BENCHMARK_ROOT_VENV/bin/datacuration-bench" next returns task_done. You are autonomous. If you run out of ideas, think harder — analyze the dataset distribution, study which samples help which benchmarks, try combining previous near-misses, try more radical strategies. The loop runs until task_done, period.

NO ITERATION-END SUMMARY: After every iter's TSV append, your next action MUST be a tool call (typically the harness next command or starting the next curate). Do NOT write a chat message summarizing iter results, "current state", "best so far", or progress reports — that information is already in results.tsv. Every iteration boundary must transition to the next tool call without intermediate reflection text.

Before any chat message, check yourself:

• Did I just append TSV / finish an iteration?
• If yes: replace the urge to summarize with "$BENCHMARK_ROOT_VENV/bin/datacuration-bench" next.

Data Isolation (MANDATORY)

Your results are only valid if you plan strategies from scratch. Accessing another run's work is cheating and invalidates your results.

You are PROHIBITED from:

1. Reading, listing, scanning, or referencing any files, scores, strategies, curation code, evaluation outputs, or any other artifacts produced by a benchmark run, session, or agent other than your own active session.

2. Enumerating directories (via ls, find, glob, or shell loops) to discover what other work exists outside your currently selected suite.

3. Accessing git commits, branches, reflog, or history outside your current /workspace git history. This includes viewing branches from past sessions, looking up external commit hashes, or using --all flags.

4. Delegating the above to tools, sub-agents, or scripts.