# Experiment Scripts Skills

> Experiment Run Scripts (Fast execution + robust authoring)

- Skill: `steve2457/experiment-scripts-skills` (Agent Skill)
- Install (CLI): `npx skillmds@latest add steve2457/experiment-scripts-skills`
- Raw SKILL.md: https://api.skillmd.com/api/skills/steve2457/experiment-scripts-skills/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: steve2457 (https://skillmd.com/u/steve2457)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/steve2457/experiment-scripts-skills

---


# Experiment Run Scripts (Fast execution + robust authoring)

## Goals

- **Run efficiently**: launch parallel workers (often per-GPU) with clean logs and deterministic inputs.
- **Write maintainable scripts**: keep scripts as orchestrators (resource selection, sharding, logging, chaining), with the *experiment logic* living in Python modules/configs.

## Non-negotiable conventions

- **Run from repo root** (or make paths robust): treat configs/data/outputs as repo-relative by default.
- **Config-driven**: the script takes a single `CONFIG` path (YAML/JSON) and passes it through; avoid editing code to switch datasets/params.
- **Stable outputs**: results and logs always land in a predictable `outputs/<run-name>/...` folder (or a config-defined output dir).
- **Resume-first**: rerunning the same command should be safe. Prefer idempotent “skip if output exists” behavior in the underlying program.

## Operator quickstart (copy/paste)

### Environment

- **Python deps**: install with the project’s dependency manager (`requirements.txt`, `pyproject.toml`, conda env, etc.).
- **GPU deps (optional)**: if `nvidia-smi` is available, the script can shard across GPUs; otherwise it must **fallback to CPU single-shard**.

### Common pattern: batch parallelism over seeds/tasks

Typical invocation:

```bash
bash scripts/run_experiment.sh path/to/config.yaml
```

Behavioral requirements:
- **Parallel model**: launch N background jobs, then `wait` for the batch to finish.
- **Per-unit logs**: one log per seed/task/shard, named predictably.
- **Final aggregation**: after all workers finish, optionally run a merge/aggregate step.

### Common pattern: GPU sharding with CPU fallback

Run:

```bash
bash scripts/run_experiment.sh path/to/config.yaml
```

Logs (example):

```bash
outputs/<run-name>/logs/shard_*.log
```

Failure handling:
- **Any shard fails**: rerun the same script. If resume is implemented, completed work is skipped automatically.
- **Insufficient GPU memory**: the script should refuse to launch and print actionable diagnostics (free memory, processes).

### Post-processing chaining

Typical sequence (names are project-specific; keep the *pattern*):

```bash
python -m package.module --mode merge --config path/to/config.yaml
python -m package.module --mode postprocess --config path/to/config.yaml
python -m package.module --mode analyze --config path/to/config.yaml
```

## Quality checks & debugging (preferred: dedicated scripts)

### Spot-check outputs (human-auditable report)

Requirements for a good spotcheck tool:
- Accept `--input_dir` / `--out_dir`, `--n`, `--seed`, optional `--stratify`.
- Produce **CSV + HTML** (or Markdown) so humans can audit correctness quickly.

### Diagnose “feature extraction / peak detection / metrics not found”

Requirements for a good diagnostics tool:
- Never re-run expensive inference; scan existing artifacts.
- Save a summary JSON plus a sweep CSV for threshold sensitivity.

## How to author new runner scripts (the agent must follow)

### 0) Provide one stable entrypoint

- **Naming**: `scripts/run_<pipeline>.sh` (one script per pipeline).
- **Usage**: `bash scripts/run_<pipeline>.sh [config_path]` with a sensible default config.
- **Inputs**: accept *one primary argument* (`CONFIG`) and keep the rest as constants or environment-variable overrides.

### 1) Required Bash skeleton

The script must include:
- `set -euo pipefail`
- `CONFIG=${1:-"path/to/default.yaml"}`
- `LOG_DIR="outputs/<run-name>/logs"` and `mkdir -p "$LOG_DIR"`
- **GPU detection / fallback**:
  - If `nvidia-smi` is missing: run **single-shard** and `tee` logs to `shard_0.log`.
  - If `nvidia-smi` exists: select GPUs (by free memory or explicit allowlist) and set `TOTAL_SHARDS=${#AVAILABLE_GPUS[@]}`.
- **Shard launch**: for `shard_idx=0..TOTAL_SHARDS-1`, set `CUDA_VISIBLE_DEVICES=$gpu_id` and pass:
  - `--shard "$shard_idx"`
  - `--total_shards "$TOTAL_SHARDS"`
  - redirect logs to `"$LOG_DIR/shard_${shard_idx}.log"`
- **Wait + failure summary**: store `PIDS[]`, `wait` each PID, count failures, exit non-zero if any failed.
- **Post steps**: run merge/aggregate and optional analysis steps.

### 2) Logging & outputs (must be enforceable)

- **One log per shard**: `shard_${shard_idx}.log`.
- **Output dir is a parameter**: do not hardcode file names for downstream artifacts; print *where to find results*.
- **Print grep-friendly banners**: config path, start/end time, GPU list, per-shard PID, and failing shard log paths.

### 3) Minimum reproducibility bar

- **No code edits for runs**: dataset/size/output must be configurable.
- **Safe to rerun**: reruns should not corrupt completed artifacts; prefer atomic writes and “skip-if-exists”.

### 4) Common pitfalls to avoid

- **Confusing physical GPU id with shard id**: shard indices should be `0..TOTAL_SHARDS-1` even if physical GPU ids are `[2,5,7]`.
- **Brittle paths**: avoid assuming the current directory unless explicitly enforced.
- **Silent failures**: always surface exit codes and point to the exact failing log file.