Augmentation Benchmark Contract

Augmentation benchmark contract

Read docs/benchmark_execution_contract.md before changing code, launching, resuming, aggregating, plotting, or interpreting results. It is the source of truth; this skill is the short operational checklist.

Current production question

The only runnable matrix is RGB dataloader_disk: local ImageNet JPEG → library-native reader and decoder → recipe → collate → pinned H2D → model-ready CUDA batch. RGB uses one Standard g2-standard-16 VM with one L4 and the frozen values in configs/families/rgb.yaml.

One cell is exactly (family, implementation, recipe, seed). One pass records both primary outcomes:

1. end-to-end throughput; and
2. peak process GPU memory from pipeline construction through the final CUDA synchronization.

Never create a memory-only, micro, H2D-only, training, capacity, or smoke matrix unless the user explicitly adds that research question.

Non-negotiable execution rules

• Normalize never runs on CPU. CPU paths may decode, form a collatable shape, and apply their CPU recipe. The collated batch moves to CUDA first; conversion to float16 and Normalize then happen on GPU. DALI performs its native GPU normalization in the graph.
• The timed output is CUDA float16 BCHW B×3×224×224, materialized and synchronized. Resize is exactly Resize(224) → Normalize → ToTensor.
• Measure recipe execution, not Compose init or imports. Start the NVML process-memory monitor before pipeline construction.
• Keep batch, workers, prefetch policy, recipe parameters, dataset ordering, hardware, and timing boundary identical across rows. Pairwise summaries use the exact supported recipe intersection; coverage is a separate census.
• The VM runs one implementation sequentially through all of its recipes and seeds. It writes every valid cell to GCS immediately. Resume validates those immutable cells and runs only the missing ones. Do not split by duration, recipe, seed, or arbitrary shard.
• The current runner exposes only RGB. 9ch, video, and volume become runnable only after each has its own data format, config, recipes, GPU-only Normalize implementation, and one-batch L4 preflight per implementation-recipe pair: non-DALI uses one temporary DataLoader worker and DALI uses its native graph.

Before a production launch

1. Confirm a clean Git worktree and build the source archive from that commit.
2. Validate the RGB config (including archive identity and selection), catalog, lock, and matrix.
3. Verify output validation and the same-pass GPU-memory field in the result schema.
4. Use augbench launch-rgb; it searches eligible zones, protects an active labelled VM from duplication, and resumes only validated missing cells.
5. If the VM exits before a cell, inspect its runs/<run_id>/logs/ bootstrap log; do not retry blindly.
6. A new run may reclaim only terminal or suspended augbench VMs; never delete active VMs or GCS result artifacts.

If a required measurement is absent, repair the execution path and generate the smallest missing part of the production matrix. Never substitute prose for data.