Add Dataset

Usage

/add-dataset <name> <source> — e.g. /add-dataset soneva https://example.com/soneva-dataset, or /add-dataset soneva /mnt/data/soneva for a local path.

<source> maps to one of these download labels: website · hugging-face · google-drive · local · api · other (anything else — ask the user) — canonical definitions in Datasets/extra-files/closed_lists.md#download-type-download. Step 1's "From <source>" resolution rule (item 2 below) classifies <source> against these for real — not restated here.

Both <name> and <source> are required. Parse them from $ARGUMENTS (or from the request, if invoked without the slash command). If either is missing or ambiguous, stop and ask — don't guess a name or search for a source yourself.

What "done" means

Adding a dataset = a DatasetVSLAMLAB subclass + settings YAML, registered in Datasets/get_dataset.py, plus a smoke-test config/experiment pair that has actually been run end to end, then committed.

Not done until step 9 (commit) has run. Steps 1–7 produce files that look correct by inspection; step 8 is what proves the dataset works. In past runs step 8/9 got skipped because step 7 (the README row) feels like a natural stopping point — it isn't.

File scope (hard constraint)

The only files this skill may create or modify:

• Everything else in the repo — base classes, other datasets' files, other configs/*.yaml, the templates themselves, the rest of README.md — is read-only reference material, even when editing it would be convenient. If working through this skill turns up a bug, inconsistency, or improvement outside this scope, don't edit it — find a way to finish the dataset without that change, and file it instead (see the issue exception below).
• Outside the repo, the only path this skill may touch is VSLAM-LAB-Benchmark/<DATASET_FOLDER>/ (this dataset's downloaded benchmark data) — freely create/remove/recreate anything inside it, that's what steps 6/8's test runs are for. Nothing else outside the repo.
• Git exception: step 9 stages and commits exactly the files listed above, as one local commit. Never push, amend, rewrite, or touch branches.
• Issue exception: an out-of-scope finding gets filed as a GitHub issue, not silently worked around and not left for the user to notice on their own — gh issue create --label <label> ... (see CLAUDE.md's Issue Labels for which one fits). Include what was found, where, and how this run worked around it. This is the only other out-of-scope write this skill may perform, alongside step 9's commit — the file itself still never gets edited.

Workflow

Step 0 — Refresh the dataset table

Run python3 Datasets/extra-files/generate_dataset_table.py to regenerate Datasets/extra-files/dataset_table.md from the current Datasets/dataset_files/*.yaml/.py. Steps 1 and 2 both read from it, so it must reflect current repo state first.

@../../../Datasets/extra-files/dataset_table.md

Step 1 — Gather the required fields, then report them

@../../../Datasets/extra-files/closed_lists.md

Resolve each field in this order — don't skip ahead:

1. From the prompt — whatever the user already gave beyond <name>/<source> (modes, sequence names, etc.).
2. From <source> — inspect it to fill whatever the prompt didn't; this also reveals which of the five download patterns fits (see the closed-list definitions above, or Datasets/extra-files/closed_lists.md#download-type-download directly). A dataset can mix patterns per sequence. Always pin down a real pattern — other isn't one to implement against.
3. Ask the user — for anything still unresolved. Don't fill gaps with a guess or plausible default.

modes applies dataset-wide, not per-sequence — DatasetVSLAMLAB.check_sequence_integrity() requires every sequence to satisfy every listed mode (e.g. an rgb_1 folder for every sequence if stereo/stereo-vi is listed), regardless of whether that sequence's own source data supports it. If sequences have genuinely different capabilities, split into separate dataset_names by capability instead of forcing the richest modes list on all of them — repeat steps 3–9 per split. Model: dataset_rover.py's rover-t265/rover-d435i/rover-picam (by sensor type); dataset_pamir.py/dataset_pamir_rig.py's pamir/pamir-rig (2024 rig dive vs. 2025 single-camera dives) — the split there was only caught in step 8, after a stereo sequence had already broken a mono-only dataset's check_sequence_availability. Catch it here instead.

resize — true if source images are bigger than 640×480 by pixel area, else false. Only decides the YAML's initial target_resolution value; the field is safely removable later with no code change (create_rgb_folder's uniform runtime branch on self.target_resolution — step 4c below). Full detail: dataset_template.yaml's target_resolution comment.

If the prompt/source/user names a mode, camera model, or raw format outside the current closed lists, don't add it as new — flag it in step 2's Notes and ask the user how to proceed.

Before moving to step 2, print this table plus a Notes line for anything worth flagging (most importantly any inconsistency between what the user said and what <source> shows — e.g. user said mono-only but the source also has depth frames):

If a note flags an unresolved inconsistency, stop and confirm with the user before proceeding.

Step 2 — Cross-check against similar datasets, confirm with the user

Using Datasets/extra-files/dataset_table.md, find datasets sharing at least one label with the step-1 table — every field except dataset_name/sequence_names. Five of them (cam_models, raw_formats, modes, download, download_issues) are columns in the table, so scan it directly. The other three (resize, groundtruth_available, calibration_type) aren't table columns — check those only among datasets already surfaced by the five table-based labels, by reading that candidate's .yaml/.py directly: resize ↔ its .yaml sets target_resolution; groundtruth_available ↔ its create_groundtruth_csv does real parsing rather than writing a header-only file; calibration_type ↔ its create_calibration_yaml uses a global constant vs. parses a per-sequence file (Datasets/extra-files/closed_lists.md#calibration-type-calibration_type). Show the user a comparison table of just the overlaps:

Ask the user if this list looks right before going further.

Once confirmed, read the matched datasets' .py/.yaml — which function(s) to study depends on which label matched:

A dataset matching on more than one label is worth studying more closely.

When several matches are candidates for the same label, trust them in this order (check vslamlab_maintainer/about.authors in each .yaml):

1. vslamlab_maintainer.name is Alejandro Fontan.
2. vslamlab_maintainer.name also appears in that dataset's own about.authors (integrator was an original creator).
3. Any other dataset.

Prefer the highest-priority candidate available.

Step 3 — Write the dataset YAML

Copy Datasets/extra-files/dataset_template.yaml → Datasets/dataset_files/dataset_<name>.yaml and populate it directly from the step-1 table — this is the real, final YAML, not a placeholder. The template's own inline comments are the canonical reference for each field's exact shape and gotchas (the index below just points into them — don't re-derive or re-explain a pattern that's already documented there):

@../../../Datasets/extra-files/dataset_template.yaml

• dataset_name, sequence_names (confirmed list).

• rgb_hz — RGB capture rate in Hz; required by the base class (cfg["rgb_hz"], no default), not a step-1 field — get it from <source> or ask.

• cam_models, raw_formats, modes — as YAML lists, e.g. ['pinhole'], ['ros1', 'zip'], ['mono', 'stereo']. The template above lists the closed-list values; the full mapping for all three is canonical in closed_lists.md (already included in step 1) — gotchas for cam_models are in step 4's template's create_calibration_yaml WARNINGS.

• The download field for the step-1 download pattern — the template above lists the exact field name/shape; the fetch mechanism is canonical in closed_lists.md (already included in step 1).

• If resize is true, add target_resolution: [640, 480]; omit entirely if false — see step 1's resize field above for why this is just an initial value, safely changed later with no code change.

• Any mode-specific fields a sibling YAML of the same modes/source carries (e.g. depth_factor for rgbd, url_download_root_gt for a separate groundtruth archive like dataset_kitti.yaml).

• An about: block (license, summary, homepage, authors) and vslamlab_maintainer: block, matching the shape in dataset_eth.yaml.

calibration_type and download_issues aren't YAML fields — they inform create_calibration_yaml/get_download_issues in step 4.

Step 4 — Implement the Python class

Copy Datasets/extra-files/dataset_template.py → Datasets/dataset_files/dataset_<name>.py, subclass DatasetVSLAMLAB, name it <Name>Dataset in PEP 8 CapWords — capitalize each underscore-separated token of dataset_name, no acronym exceptions (soneva → SonevaDataset, eiffel_tower → EiffelTowerDataset, hilti2022 → Hilti2022Dataset). Study the source-pattern model from step 1 and a same-mode sibling rather than writing from scratch.

@../../../Datasets/extra-files/dataset_template.py

Before any real logic:

• Module header (Author/Assisted by/Version: 1.0/Created/License) must match the YAML's vslamlab_maintainer: block exactly (Author = .name, Assisted by = .assisted_by or None, Created = .date).
• Class docstring: """<Display Name> dataset helper for VSLAM-LAB benchmark.""" — <Display Name> is the proper/brand name from about.summary/about.homepage (natural Title Case), never the class name or the lowercase slug. A few disambiguating words are fine ("""MADMAX Mars rover navigation dataset helper...""").
• Import grouping: from __future__ import annotations first, then up to three blank-line-separated groups (stdlib, third-party, project-local), plain import x before from x import y within each, alphabetical case-insensitive. No linter enforces this — see dataset_soneva.py. Manual discipline only.
• Scaffold first: add every template hook as a placeholder (pass or print("TODO") + pass) so the class is importable/instantiable immediately, before any real logic exists to hide import/signature/ABC errors behind.
• Prefer utilities.py over hand-rolled logic (path helpers, CSV read/write, downloadFile/decompressFile, HF/COLMAP helpers, make_printers, ...) — but utilities.py is out of scope for this skill. If something reusable is genuinely missing, tell the user and suggest it as a follow-up; don't add it yourself.
• Delete a hook entirely (not a hollow stub) if it doesn't apply: create_imu_csv for non--vi, get_download_issues if download_issues is blank.

Then implement each hook for real, in order:

Gotchas, per hook. The template included above already carries the full explanation for each one — this table is only an index of what to double-check and where it's demonstrated, not a restatement:

remove_unused_files — retention tiers. BENCHMARK_RETENTION (path_constants.py, default Retention.STANDARD) closed list — the three tiers' exact meanings and the if BENCHMARK_RETENTION != Retention.FULL / == Retention.MINIMAL code shape are canonical in Datasets/extra-files/closed_lists.md#benchmark-retention-benchmark_retention. Every implementation gotcha (unlink-path mismatches, shutil.rmtree vs. unlink(), symlinked raw folders, and the four shared-archive scopes — whole-dataset-only, scene/group-scoped, dataset-wide indefinitely reused, exact-file share) is canonical in dataset_template.py's remove_unused_files comment (included above). Not restated here.

Step 5 — Register it

In Datasets/get_dataset.py:

• Add from Datasets.dataset_files.dataset_<name> import <Name>Dataset under the correct mode section comment (Monocular / RGBD / Stereo / Stereo-VI / Development).
• Add to the switcher dict in get_dataset(): "<name>": lambda: <Name>Dataset(),.

Step 6 — Smoke-test config + experiment pair

Reference: test_config_eth.yaml/test_exp_eth.yaml (eth is mono+rgbd, 97 sequences) — demonstrates both rules below. Follow it, or a closer step-2 sibling, rather than inventing the shape.

configs/test_config_<name>.yaml — a small, representative subsample, never the whole dataset (even for a small dataset):

<dataset_name>:
  - sequence_01
  - sequence_02

Representative = sequences exercising different sizes/conditions if heterogeneous, not just the first N alphabetically. See test_config_eth.yaml (2 of 97, different scene categories) and test_config_sweetcorals.yaml (4 of 13, one per site group, including the one sequence with real calibration/groundtruth).

configs/test_exp_<name>.yaml — one block per mode this dataset supports (step-1 modes), not just one total:

test_exp_<name>_<baseline>:
  Config: test_config_<name>.yaml
  NumRuns: 1
  Parameters: {verbose: 1, mode: <one of this dataset's modes>, rgb_idx: [0,2000]}
  Module: <baseline>

• <baseline> (Module) must be a pixi environment name from pixi.toml's [environments] table — match the closest sibling's choice where possible (droidslam/dpvo are common lightweight picks for mono/rgbd); different modes can use different baselines (test_exp_eth.yaml does).
• A second baseline for the same mode is worth adding when convenient, not just when a second mode forces it (test_exp_soneva.yaml/test_exp_sweetcorals.yaml, both mono-only, each run two).
• rgb_idx: [0,2000] caps the smoke test to the first ~2000 frames — omit only if the matched sibling's convention doesn't use it (test_exp_videos.yaml/test_exp_strayscanner.yaml). Some siblings (eth/soneva/sweetcorals) instead use max_rgb/step_size to spread a fixed frame count across the whole sequence — a deliberate choice when late-sequence content (e.g. loop closures) matters, not the default to copy elsewhere.

Step 7 — Add the dataset to README.md

Pick the table first. README.md has two tables for this mechanism, updated two different ways:

• Datasets — a fixed, published benchmark with its own sequences (groundtruth typically shipped or derivable). This is the generated <table>...</table> HTML block — regenerated from every dataset_*.yaml's about: block, never hand-edited.
• Tools — a data-capture app/format whose sequences the user brings themselves (e.g. strayscanner; no fixed published sequence set). This is a plain markdown pipe table, hand-edited.

Both register through the identical DatasetVSLAMLAB subclass + YAML + get_dataset.py mechanism — the table choice is a README categorization only. If it's unclear which fits, ask the user rather than guessing.

If Datasets table: confirm step 3's YAML has about.features populated — closed list, canonical in Datasets/extra-files/closed_lists.md#dataset-features-aboutfeatures. Ask the user if unclear, don't guess. Then run:

python3 Datasets/extra-files/generate_readme_datasets_table.py

This rewrites the entire <table> block in place from about.features/about.summary/about.homepage/modes/cam_models across all datasets — it is not additive by hand; don't insert a row yourself.

If Tools table: append one row as the last real entry — immediately above that table's commented-out placeholder rows for not-yet-implemented entries:

| [**<Display Name>**](<homepage URL>) | <feature emoji(s)> | `<dataset_name>` | <modes> | <cam_models> |

• <Display Name>/<homepage URL> — from the YAML's about: block.
• <feature emoji(s)> — closed list, same as the Datasets table (Datasets/extra-files/closed_lists.md#dataset-features-aboutfeatures). Ask the user if unclear, don't guess.
• <modes> — step-1 list; a mode + its -vi variant collapse to one entry (`mono(-vi)`), backticked, space-separated.
• <cam_models> — step-1 list, backticked, space-separated.

Step 8 — Simulate the download, function by function (required — do not skip)

Even when steps 1–7 look correct on inspection, only running the code catches a wrong URL, a path typo, or a malformed calibration field. Using the first sequence in sequence_names, drive download_process's hooks yourself, one at a time (dataset = get_dataset(dataset_name), call each method directly) instead of one opaque CLI call:

1. download_sequence_data
2. create_rgb_folder
3. create_rgb_csv
4. create_calibration_yaml
5. create_imu_csv (skip if deleted in step 4)
6. create_groundtruth_csv
7. remove_unused_files

Track state as you go — this example illustrates a non--vi dataset (create_imu_csv already deleted in step 4, so it's skipped and everything renumbers accordingly); include it as its own row, between create_calibration_yaml and create_groundtruth_csv, if your dataset has it:

After each function, report in detail:

• Inputs: sequence_name + state/files it consumed.
• Output: exact paths created, file/image counts, folder sizes; row counts + first rows for create_rgb_csv/create_groundtruth_csv; actual focal_length/principal_point/image_dimension values for create_calibration_yaml; paths actually deleted for remove_unused_files.

This is slower than letting the pipeline run silently, but it pinpoints which stage produced bad output instead of only learning after the fact.

Only move to step 9 once every row reads processed. If a hook fails, fix dataset_<name>.py/.yaml and re-run from that hook (or from the top if the fix touches download_sequence_data/__init__) — don't commit an unproven dataset.

Step 9 — Commit

Before staging, check whether this run surfaced anything new — a download sub-pattern, a mode/camera-model nuance, a gotcha, a stale or missing Model: citation — that isn't already covered by SKILL.md, CLAUDE.md, dataset_template.py, or dataset_template.yaml. All four are out of this skill's file scope (per the hard constraint above), so don't edit them directly — use the Issue exception to file it (improvement label) so a future doc pass can fold it in. This is the same final-sweep habit Datasets/extra-files/dataset_cleanup_log.md (checklist item 16) codifies for cleanup passes, applied here so it isn't only ad hoc.

Stage exactly the files this skill created/modified, by name — never git add -A/git add .:

• Datasets/dataset_files/dataset_<name>.py
• Datasets/dataset_files/dataset_<name>.yaml
• Datasets/get_dataset.py
• configs/test_config_<name>.yaml
• configs/test_exp_<name>.yaml
• README.md
• Datasets/extra-files/dataset_table.md

Run git status first and confirm the staged set matches this list exactly. Commit with a concise message like Add <name> dataset (check git log --oneline -10 for style). Local commit only — never push, force-push, or amend.

Full reference docs live on the project's GitHub Wiki if more detail is needed.