# Update Datasets

> Build and publish Bactopia's version-pinned datasets to Cloudflare R2. Currently implements the AMRFinder+ database. It verifies the amrfinderplus module is at the latest bioconda version, rebuilds amrfinderplus.tar.gz inside the module's pinned container, and (after confirmation) uploads it via rclone to datasets/v<version>/amrfinderplus.tar.gz. Use when asked to update datasets, rebuild the amrfinderplus database, publish a dataset bundle, refresh the version-pinned datasets, or prepare datasets for a release.

- Skill: `bactopia/update-datasets` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add bactopia/update-datasets`
- Raw SKILL.md: https://api.skillmd.com/api/skills/bactopia/update-datasets/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: bactopia (https://skillmd.com/u/bactopia)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/bactopia/update-datasets

---


# Update Datasets

Build and publish Bactopia's **version-pinned** datasets to Cloudflare R2.

`bactopia datasets` downloads bundles from `https://datasets.bactopia.com/datasets/`.
Most bundles are shared across releases, but a few are pinned to the pipeline
version. The only version-pinned dataset today is **AMRFinder+**
(`conf/params.config`: `amrfinderplus_url = ".../datasets/v${params.bactopia_version}/amrfinderplus.tar.gz"`),
so this skill implements that path concretely; the name stays generic for future
version-pinned datasets.

The publish version is `versions.yml`'s `bactopia` value, and the R2 layout is
`datasets/v<version>/amrfinderplus.tar.gz`. The tool version that **builds** the
database must equal the version the pipeline **pins** (an outdated binary can
produce a database the pinned binary cannot load), so the skill hard-gates on
the `amrfinderplus` module being current before building. This skill is the sole
owner of the build recipe — the old `modules/amrfinderplus/update/` module that
carried it was removed.

## Steps

1. **Currency gate (hard stop).** Confirm the module tool is the latest bioconda
   release before building:
   ```
   bash .agents/skills/update-module/scripts/run-bactopia-update.sh \
       --bactopia-path /home/rpetit3/repos/bactopia/bactopia --module amrfinderplus --json --silent
   ```
   Parse the entry whose `tool == "ncbi-amrfinderplus"`.
   - `needs_update == true` -> **STOP**. Tell the user to run `/update-module`
     (bumps `modules/amrfinderplus/run/module.config`), then re-run this skill.
     Do not build against a stale version.
   - `latest_version == null` (API failure) -> **STOP** and report the failure.
   - `needs_update == false` -> proceed.

2. **Build.** Rebuild the database in the module's pinned container:
   ```
   bash .agents/skills/update-datasets/scripts/build-amrfinderplus-db.sh \
       --bactopia-path /home/rpetit3/repos/bactopia/bactopia
   ```
   Add `--runtime singularity` if the user asks or docker is unavailable. This
   downloads the latest NCBI database and produces the tarball; expect it to take
   several minutes and hundreds of MB. The script reads the container image from
   `modules/amrfinderplus/run/module.config`, so version bumps flow automatically.

3. **Report the build.** Show the script's summary fields (tarball path, sha256,
   tool version, database version). Read the publish version:
   ```
   awk '/^bactopia:/{print $2}' /home/rpetit3/repos/bactopia/bactopia/versions.yml
   ```
   The R2 key is `datasets/v<version>/amrfinderplus.tar.gz`.

4. **Upload (gated behind explicit confirmation).** The rclone destination base
   is `bactopia-r2:bactopia` (remote `bactopia-r2`, bucket `bactopia`);
   `$BACTOPIA_R2_DEST` overrides it if set. The public URL
   `https://datasets.bactopia.com/datasets/...` maps to
   `bactopia-r2:bactopia/datasets/...`, so the full key is
   `<base>/datasets/v<version>/amrfinderplus.tar.gz`.
   - Verify reachability: `rclone lsd bactopia-r2:bactopia`. On failure, report
     it and fall back to printing the manual command below — do not upload.
   - Show the exact command and **ask for confirmation** before running it:
     ```
     rclone copyto "<tarball>" "bactopia-r2:bactopia/datasets/v<version>/amrfinderplus.tar.gz" --s3-no-check-bucket --progress
     ```
   - `--s3-no-check-bucket` is **required**: R2 API tokens cannot `CreateBucket`,
     which rclone otherwise attempts before the first upload and fails with a
     `403 AccessDenied`. The bucket already exists, so skip the check.
   - Only on an explicit yes, run it. `copyto` overwrites an existing key, so
     re-publishing a version is idempotent.

5. **Verify publication.** After upload, confirm the public URL resolves:
   ```
   curl -s -o /dev/null -w '%{http_code}' -I "https://datasets.bactopia.com/datasets/v<version>/amrfinderplus.tar.gz"
   ```
   `200` -> published (this is exactly `/release-checklist` check 15's sentinel).
   Report the HTTP code.

## Notes

- The currency gate reuses `/update-module`'s `run-bactopia-update.sh`; the build
  image is read from `modules/amrfinderplus/run/module.config`, so a version bump
  there is picked up without editing this skill.
- **Generic by design:** to add a future version-pinned dataset, add a sibling
  build script and a step block — the R2 layout, publish-version logic, and
  upload gating are shared.

### Sibling skills

- `/update-module` — the currency gate's remediation (step 1).
- `/release-checklist` — check 15 probes this exact bundle and recommends this
  skill when it 404s.