# Slurm Apptainer > Curation-Bench Protocol (skill-grounded) - Skill: `jiachen-t-wang/slurm-apptainer` (Agent Skill, multi-file: 11 files) - Install (CLI): `npx skillmds add jiachen-t-wang/slurm-apptainer` - Raw SKILL.md: https://api.skillmd.com/api/skills/jiachen-t-wang/slurm-apptainer/raw - Safety review: pending (external: skill-scanner PASS, skillspector PASS) - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - Author: Jiachen-T-Wang (https://skillmd.com/u/jiachen-t-wang) - Updated: 2026-08-19 - Page: https://skillmd.com/skills/jiachen-t-wang/slurm-apptainer --- # Curation-Bench Protocol (skill-grounded) You are an AI agent working inside a dataset-curation benchmark. Your goal is to curate training data based on the task to maximize downstream evaluation performance after fine-tuning. The benchmark manages the suite lifecycle, training, and evaluation. You manage the data curation strategy. **Read only this file as protocol.** No other protocol files are available to you. ## Filesystem layout - `/workspace` — your working directory and the **only** place you may write. It holds `curate.py` (the one file you edit) and anything you choose to create (`output/curated/`, your own `results.tsv`, a `.git` repo, notes). - `/data/raw` — the curation **input** dataset (read-only). - `/workspace/skills///SKILL.md` — the paper-derived skill-card library (read-only); ground each strategy in these (see **Strategy must be skill-grounded**). - `/code/benchmark`, `/code/train`, `/code/eval`, `/code/scripts` — benchmark / training / eval / script source, read-only, for reference and for the train/eval stage scripts. - `/PROTOCOL.md` — this file. Run train/eval stages through `/code/scripts/slurm/*` exactly as shown below. Do not override `BENCH_REPO_SCRIPT_ROOT` or `BENCH_REPO_COMPUTE`; the launcher sets them so wrappers can read scripts from `/code` while Slurm executes from the compute-visible repository checkout. The benchmark owns the per-iteration run directory (`run_dir`, reported by `next`) where it writes `finetune/` and `eval/` outputs and, on each completed iteration, a snapshot of your `/workspace`. ## Session initialization **First of all, ground yourself in the research skills for this session:** - `ls /workspace/skills/*/` to see entries across all four categories (`data-acquisition`, `data-curation`, `data-selection`, `data-synthesis`). - Pick **at least 3 skills per category** whose titles suggest relevance to VLM instruction-tuning data selection. - **Read those SKILL.md files in full** before touching `/workspace/curate.py`. Spawn `Agent`/`Explore` tools in parallel to scan quickly. Listing titles is not enough — you must understand the procedures so you can adapt them on budget. Then start the workflow immediately. `BENCHMARK_CURRENT_TASK` tells you which task you own — only that task, not any other task in the suite. ```bash echo "My assigned task: $BENCHMARK_CURRENT_TASK" ``` If `"$BENCHMARK_ROOT_VENV/bin/datacuration-bench" next` ever returns a `task_id` different from `$BENCHMARK_CURRENT_TASK`, exit immediately — that's another agent session's responsibility. **At the start of every session**, your working directory is `/workspace`. If you want a clean git history for your own bookkeeping, you may (optionally) reset it: ```bash cd /workspace rm -rf .git git init git add curate.py git commit -m "fresh start: $BENCHMARK_CURRENT_TASK" ``` git is entirely your own choice — the benchmark snapshots all of `/workspace` each iteration regardless. If you use git, manage any ignore rules yourself and do not commit `output/`, `.venv/`, `*.log`, `runs/`, checkpoints, or other generated artifacts. ## Setup To set up a new experiment: 1. **Read the task**: `"$BENCHMARK_ROOT_VENV/bin/datacuration-bench" task` — task definition: `task_id`, `goal`, `target_rows`, `model_key`, `target_evals`, `strategy_timeout_seconds`, `dataset_path`, `default_submission_path`. 2. **Verify vendor venvs** (see "Venv setup" below). 3. **Initialize results.tsv**: create `/workspace/results.tsv` with the header row (see **Logging results**). 4. **Proceed**. After setup, start the workflow immediately. ## The Dataset The LLaVA-665K dataset is a multimodal instruction-following dataset with ~665K samples. Each sample has: - `images`: list of PIL images - `texts`: list of conversation turns (user/assistant pairs) Inspect `ds.column_names` before relying on optional metadata. The current Container mounts expose `images`, `texts`, and `original_id`; your submitted dataset must include at least `images` and `texts`. You are selecting a subset (of size `target_rows`) that maximizes downstream VLM benchmark performance after fine-tuning on the selected data. **IMPORTANT**: The dataset is on a read-only mount. Avoid operations that write cache files beside the input dataset. Prefer column-level access to build index lists, then use `ds.select(indices)`. ## Strategy must be skill-grounded Every iteration of `/workspace/curate.py` MUST be grounded in at least one SKILL.md under `/workspace/skills/`. Before editing `/workspace/curate.py`: 1. Name a **failure mode** from the last result (e.g., "model predicts wrong MC option — suggests low-signal examples dominate training", not "MMVet is low"). 2. Pick 1–3 SKILL.md files relevant to that failure mode. **Read them in full** (not just the title). Use `Agent`/`Explore` tools in parallel if you need to scan many. 3. State in the commit message: ` — `. Example: `el2n-…-2107-07075v2 — proxy-loss scoring via forward pass on LLaVA-base to keep high-error samples`. Skills fall into four buckets (see `skills/data-{acquisition,curation,selection,synthesis}/`). **Before repeating a bucket, attempt at least one iteration from each of the other three.** ### Forbidden strategies (CANNOT do) These require zero research and will be rejected regardless of accuracy: - Tweaking per-subset sample counts based on subset name (coco/gqa/ocr_vqa/textvqa/…) — "empirical adjustment on domain ratios." - Length / word-count / turn-count thresholds (e.g., "short answer < N chars", `SHORT_ANSWER_THRESHOLD`). - Regex keyword gates on question text (e.g., `MATH_WORDS`, `REASONING_WORDS`). - Rephrase prompt tuning without a skill justification. - Any strategy whose commit message cites a skill as "inspired by" vibes rather than an adapted procedure from a specific paper. If your next idea fits the list above, discard it and pick a skill instead. ## Venv setup (one-time) The cluster uses scratch-backed venvs prepared before launch. Do not run `uv sync`, `uv pip install`, or `uv add` unless the operator explicitly asks you to repair an environment. The normal agent workflow only verifies the existing venvs and runs commands through the scratch venv executables. ```bash PROJECT_ROOT=$(pwd) test -x "${BENCHMARK_ROOT_VENV:?}/bin/python" test -x "${BENCHMARK_TRAIN_VENV:?}/bin/python" test -x "${BENCHMARK_EVAL_VENV:?}/bin/python" "$BENCHMARK_ROOT_VENV/bin/datacuration-bench" --help >/dev/null ``` ## Pre-init baseline The pre-init baseline is the base model (no finetune) evaluated on iteration 0. It is deterministic per (base model, benchmark set) and is normally **pre-seeded**: your first `"$BENCHMARK_ROOT_VENV/bin/datacuration-bench" next` returns iteration 1, and the baseline artifacts already exist at `/../iter_000/eval/results/` (`results.json` + per-question `/*.xlsx`). **Before your first curation, read those baseline predictions** to see where the base model fails. Never run eval0 yourself when `next` returns iteration 1. Fallback (not seeded): only if `next` returns status `baseline_eval_activated` and stage `eval_base`, run eval as a blocking foreground call: ```bash /code/scripts/slurm/run_judge_eval_srun.sh 2>&1 ``` This evaluates the base model path from the selected profile, not a fine-tuned checkpoint. Do not run VLMEvalKit directly, do not run `datacuration-bench eval` directly in this container, and do not add inline API credentials. After the eval stage reports `{"status": "completed", ...}`, call `"$BENCHMARK_ROOT_VENV/bin/datacuration-bench" next` again to activate `iter1`. ## Running an experiment Each iteration has three stages: curate, train, evaluate. You run each stage via the `datacuration-bench` CLI — **one stage and one iter at a time; no overlapping stages within or across iters**. The harness owns the run directory and all paths. ### Activate the next iteration ```bash "$BENCHMARK_ROOT_VENV/bin/datacuration-bench" next ``` Returns JSON with `status`, `target_key`, `task_id`, `iteration`, `run_dir`, `task_goal`, `default_submission_path`, `dataset_path`. - `status == "target_activated"` and `task_id == $BENCHMARK_CURRENT_TASK`: proceed. - `status == "task_done"`: exit. - Anything else: exit immediately. ### Stage 1: Curate Your working directory is `/workspace`, and `curate.py` is the one file you edit. Use the `dataset_path` from the JSON (`/data/raw`) as input, and write the curated HuggingFace dataset (`save_to_disk`, with `images` and `texts` columns, row count == `target_rows`) to `default_submission_path` (`/workspace/output/curated`). The reference script takes three positional args: ```bash "$BENCHMARK_ROOT_VENV/bin/python" /workspace/curate.py ``` **Avoid row-by-row Python iteration on the dataset** — e.g. `for i in range(len(ds))`, `for row in ds:`, etc. On iter1 (before editing `/workspace/curate.py`), you **MUST** read the seeded pre-init baseline so your first strategy targets the base model's weaknesses: `/../iter_000/eval/results/results.json` and the per-question files under `/../iter_000/eval/results//*.xlsx`. For iter2+ (before editing `/workspace/curate.py`), you **MUST** read in full: - The full `/workspace/results.tsv` (every row, not just the tail) — your own running log. - **EVERY** prior strategy snapshot: `/../iter_/workspace/curate.py` for each prior strategy iteration `NNN` (zero-padded). The baseline `iter_000` has no workspace snapshot — only eval results. - **EVERY** prior aggregate score: `/../iter_/eval/results/results.json` for each prior iteration `NNN`, **including the baseline `iter_000`**. - (Optional, recommended) Per-question predictions: `/../iter_/eval/results//*.xlsx` for the previous iteration (and `iter_000` for the baseline). Skipping any of the required reads above is a protocol violation. The benchmark snapshots your `/workspace` into `/../iter_/workspace/` at the end of each iteration, so prior `curate.py` is recoverable there. If the task has `strategy_timeout_seconds`, check `"$BENCHMARK_ROOT_VENV/bin/datacuration-bench" time-budget` frequently — before any long op, after each major step, before submitting. If low, simplify. If `strategy_timeout_seconds: null`, ignore the timer. Then audit and submit: ```bash "$BENCHMARK_ROOT_VENV/bin/datacuration-bench" audit --path "$BENCHMARK_ROOT_VENV/bin/datacuration-bench" submit --path ``` ### Stage 2: Train Run as a **blocking foreground** Bash call. Do NOT use `run_in_background`, `&`, or `nohup`. See the **Monitoring** section. ```bash /code/scripts/slurm/run_bench_stage_srun.sh finetune 2>&1 ``` Success signal: `"status": "completed"` with `model_path` printed after training ends. Model files land under `/finetune/`. ### Stage 3: Evaluate Run as a **blocking foreground** Bash call. Do NOT use `run_in_background`, `&`, or `nohup`. See the **Monitoring** section. ```bash /code/scripts/slurm/run_judge_eval_srun.sh 2>&1 ``` Do not run `datacuration-bench eval` directly in this container. The suite is mounted read-only here, so evaluation must run on the compute node through the stage wrapper above. VLMEvalKit writes per-benchmark scores to `/eval/results/`. On success, the target is automatically marked COMPLETED by the harness. ## Scoring After evaluation, extract raw scores from `/eval/results/`. VLMEvalKit writes per-benchmark result files under `/eval/results//` (JSON, xlsx, or CSV). Look for keys like `overall`, `score`, `accuracy`, or `avg`. The aggregate `/eval/results/results.json` (keyed by model name) is also available. For HallusionBench, the score is the average of aAcc, fAcc, and qAcc. For MMMU_DEV_VAL, use the **validation** split score (not the dev split score). The eval outputs both; report the val score. Normalization: | Benchmark | Max score | |----------------|-----------| | MMVet | 100 | | LLaVABench | 100 | | OCRBench | 1000 | | HallusionBench | 100 | | MMMU_DEV_VAL | 100 | | MathVista_MINI | 100 | | MMStar | 100 | | MMBench | 100 | **accuracy** = mean of (MMVet/100, LLaVABench/100, OCRBench/1000, HallusionBench/100, MMMU_DEV_VAL/100, MathVista_MINI/100, MMStar/100, MMBench/100) across the task's `target_evals`. ## Experimentation rules **What you CAN do:** - Edit `/workspace/curate.py` — the only file you change. Everything is fair game: selection criteria, filtering, subset balancing, deduplication, quality scoring, diversity sampling, metadata-based selection, etc. You may create helper files under `/workspace` too. **What you CANNOT do:** - Write anywhere outside `/workspace`. Everything under `/code` and `/data` is read-only (benchmark/train/eval source, task YAMLs, and the input dataset). - Change training hyperparameters or evaluation benchmarks. - Run `uv sync`, `uv pip install`, or `uv add` to modify the prepared vendor venvs. **Use discretion for new packages**: try what's already available first. **The goal**: highest accuracy (average normalized score across MMVet, LLaVABench, OCRBench, HallusionBench, MMMU_DEV_VAL, MathVista_MINI, MMStar, MMBench). **Data fraction**: exactly `target_rows` samples. The submit step rejects any other row count. **Simplicity criterion**: all else being equal, simpler is better. A small improvement that adds ugly complexity is not worth it. Removing something and getting equal or better results is a great outcome. Simplicity applies within a given skill-grounded approach (don't over-engineer the implementation), not across approaches. **Pre-init baseline**: The baseline eval0 is normally pre-seeded — your first `next` returns iteration 1 and the baseline artifacts already exist under `iter_000/eval/results/`. Read them before your first curation; see the **Pre-init baseline** section. If you maintain a TSV row for the baseline, use commit `0000000` and description `pre-init baseline (no finetune)`. **Never re-run eval0**: if `next` returns iteration 1, or eval0 is already completed, or `results.tsv` already has a pre-init row, move on. (Only when `next` returns the eval-only iteration 0 target — `baseline_eval_activated` — do you run it; that is the not-seeded fallback.) **First run (iter1)**: Always establish your starting reference first by running `/workspace/curate.py` unchanged. By default it is the random-selection baseline (select `target_rows` examples from the original dataset), but it may have been seeded with a preconfigured starting strategy at launch (or carry a non-baseline strategy from a previous session). **You MUST NOT modify `/workspace/curate.py` for iter1 — run it exactly as-is, even if it looks non-baseline.** **If results.tsv already contains an iter1 row, skip it and move on — do not re-run it.** ## Logging results When an iter is done, append a row to `/workspace/results.tsv` (tab-separated, NOT comma-separated). The header row is created in Setup; each iter (including the pre-init baseline) adds one data row. Header and columns: ``` commit accuracy MMVet LLaVABench OCRBench HallusionBench MMMU_DEV_VAL MathVista_MINI MMStar MMBench status description skill_ref failure_mode next_skill_candidate ``` 1. git commit hash (short, 7 chars) — use `0000000` for the pre-init row 2. accuracy (normalized average, e.g. 0.456700) — 0.000000 for crashes 3. MMVet raw score (e.g. 35.2) — 0.0 for crashes 4. LLaVABench raw score (e.g. 62.1) — 0.0 for crashes 5. OCRBench raw score (e.g. 310.0) — 0.0 for crashes 6. HallusionBench raw score (avg of aAcc, fAcc, qAcc; e.g. 42.5) — 0.0 for crashes 7. MMMU_DEV_VAL raw score (validation split; e.g. 34.0) — 0.0 for crashes 8. MathVista_MINI raw score (e.g. 28.5) — 0.0 for crashes 9. MMStar raw score (e.g. 35.0) — 0.0 for crashes 10. MMBench raw score (dev split Overall; e.g. 65.0) — 0.0 for crashes 11. status: `keep` (new best accuracy so far), `discard` (worse), or `crash` (run failed) 12. short text description of what this iteration did (the adaptation, in words) 13. **skill_ref**: the SKILL.md directory this iteration adapted (e.g. `el2n-…-2107-07075v2`). Use `none` for the baseline and pre-init rows only — every non-baseline iteration must cite a skill. A value of `none` on a non-baseline row is a rule violation. 14. **failure_mode**: diagnosis of what the eval output suggests is broken (what signal the model lacks, not which knob to tweak). Example: "model fails compositional reasoning on MMBench — high-variance examples filtered out." 15. **next_skill_candidate**: which SKILL.md dir you plan to draw from next, and a one-line reason. Must be from a category you haven't used in the last three iterations unless justified. Example: ``` commit accuracy MMVet LLaVABench OCRBench HallusionBench MMMU_DEV_VAL MathVista_MINI MMStar MMBench status description skill_ref failure_mode next_skill_candidate 0000000 0.420000 32.0 58.0 295.0 40.0 32.5 27.0 33.0 62.0 keep pre-init baseline (no finetune) none none (baseline) el2n-…-2107-07075v2 — score via proxy forward pass a1b2c3d 0.456700 35.2 62.1 310.0 42.5 34.0 28.5 35.0 65.0 keep baseline random 10k none none (baseline) el2n-…-2107-07075v2 — score via proxy forward pass b2c3d4e 0.478200 37.1 64.3 320.0 44.0 35.5 30.0 36.5 67.0 keep EL2N proxy-loss top-10k via LLaVA-base forward pass el2n-…-2107-07075v2 hallucination unchanged — kept hard-but-wrong examples dataset-cartography-…-2009-10795v2 — separate ambiguous from hard c3d4e5f 0.412000 30.5 58.2 290.0 38.0 32.0 26.0 33.0 60.0 discard Data Shapley approximation via leave-one-out loss deltas data-shapley-…-1904-02868v6 LOO too noisy at 10k sample scale influence-functions-…-2002-08484v3 — Hessian-free approx d4e5f6g 0.000000 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 crash dedup bug caused empty dataset deita-…-2312-15685v2 crash, not a failure mode retry with fix ``` Rows added to `results.tsv` must have a non-`none` `skill_ref` value unless they are the baseline or pre-init baseline. Rows violating this should be audited and rerun with a skill-grounded strategy. You do **not** need to copy `curate.py` snapshots anywhere — the benchmark snapshots all of `/workspace` (including `curate.py`) into `/../iter_/workspace/` automatically when each iteration completes. ## The experiment loop LOOP UNTIL `"$BENCHMARK_ROOT_VENV/bin/datacuration-bench" next` RETURNS `task_done`: 1. Check git state (optional, if you use git). 2. `"$BENCHMARK_ROOT_VENV/bin/datacuration-bench" next`, check `status` and `task_id`. 3. **Pick a skill.** Diagnose the current failure mode from the last iteration's results. From `/workspace/skills/`, pick 1–3 SKILL.md files matching that failure mode and read them in full. If the last three iterations drew from the same category (`data-acquisition` / `data-curation` / `data-selection` / `data-synthesis`), pick from a different one. Skip for the pre-init baseline and iter1. 4. **Write the adaptation.** Name the concrete procedure you will adapt from the chosen skill, and what approximation you will use given the compute budget (e.g., "use base-model forward pass as a proxy since training a reference model is too expensive"). This becomes the commit message: ` — `. 5. Modify `/workspace/curate.py` to implement the adaptation (iter1: run unchanged — see **First run**). If the change is a subset-ratio/length/regex heuristic, it does not qualify — go back to step 3. 6. (Optional) commit in `/workspace` with message ` — ` if you use git. No manual snapshot copying — the harness snapshots `/workspace` automatically. 7. Run the three stages in sequence (curate → train → evaluate) using the blocking commands above. Avoid dumping large logs into context. 8. Extract results from eval output files. 9. If any stage failed or results are missing: tail the matching stderr to diagnose — finetune failure: `tail -n 50 /finetune/train_stderr.txt`; eval failure: `tail -n 50 /eval/eval_stderr.txt`. 10. Record results in TSV. `skill_ref` must be the SKILL.md directory name (or `none` only for baseline/pre-init); `failure_mode` is the diagnosis of what signal the model lacks; `next_skill_candidate` is the next SKILL.md to draw from, ideally from a category not used in the last three iterations. 11. If accuracy improved (higher): keep the commit ("advance"). 12. If accuracy equal or worse: revert `/workspace/curate.py` to the prior best (e.g. `git reset --hard HEAD~1` if you use git). The iter's official eval score is already recorded by the harness; reverting only affects your `curate.py` starting point for the next iter. ## Monitoring **Don't check anything excessively — every tool call re-reads your full context. Keep checks sparse and reasonable regardless of what file or mechanism you're using: tailing logs, listing directories (ls), grepping output files, watching step counts, counting benchmark completions, reading intermediate state, probing background task progress, etc., all count. Don't spawn Monitor watchers to poll progress.** **ETA for curation**: for data curation operations, always compute and print an ETA, and check proactively while it is running — do not wait on scripts that do not finish quickly without an ETA. If a curation script is running without producing progress/ETA output, kill it, add ETA logging, and re-run. Harness commands (`finetune` / `eval` / pre-init baseline) don't need progress monitoring — the harness owns the wall-clock. **TIMED YIELDS ARE NOT COMPLETION**: `finetune` and `eval` run for many minutes, and your shell may hand control back with a "still running" / timed-out notice **before the command has exited**. That is not completion. Keep waiting on the *same* stage until the wrapper command actually returns AND the target's `status.json` records it done. Do NOT launch the next stage, write anything, or treat the run as finished on a timed yield. "Finetune launched but not yet evaluated and scored" is **not done**. The only signal the run is over is `next` returning `task_done`. **Crashes**: if it's something dumb and easy to fix, fix and re-run. If fundamentally broken, log `"crash"` and move on. **NEVER STOP**: once the experiment loop begins, do NOT pause to ask the human if you should continue. Do NOT ask "should I keep going?" or "is this a good stopping point?". The human might be asleep and expects you to continue working *indefinitely* until `"$BENCHMARK_ROOT_VENV/bin/datacuration-bench" next` returns `task_done`. You are autonomous. If you run out of ideas, think harder — analyze the dataset distribution, study which samples help which benchmarks, try combining previous near-misses, try more radical strategies. The loop runs until `task_done`, period. **NO ITERATION-END SUMMARY**: After every iter's TSV append, your next action MUST be a tool call (typically the harness `next` command or starting the next curate). Do NOT write a chat message summarizing iter results, "current state", "best so far", or progress reports — that information is already in results.tsv. Every iteration boundary must transition to the next tool call without intermediate reflection text. Before any chat message, check yourself: - Did I just append TSV / finish an iteration? - If yes: replace the urge to summarize with `"$BENCHMARK_ROOT_VENV/bin/datacuration-bench" next`. ## Data Isolation (MANDATORY) Your results are only valid if you plan strategies from scratch. Accessing another run's work is cheating and invalidates your results. You are PROHIBITED from: 1. Reading, listing, scanning, or referencing any files, scores, strategies, curation code, evaluation outputs, or any other artifacts produced by a benchmark run, session, or agent other than your own active session. 2. Enumerating directories (via ls, find, glob, or shell loops) to discover what other work exists outside your currently selected suite. 3. Accessing git commits, branches, reflog, or history outside your current `/workspace` git history. This includes viewing branches from past sessions, looking up external commit hashes, or using --all flags. 4. Delegating the above to tools, sub-agents, or scripts.