# Earth2studio Create Datasource

> Create and validate Earth2Studio data source wrappers (DataSource, ForecastSource, DataFrameSource, ForecastFrameSource) from remote stores. May add Python dependencies to pyproject.toml as part of development. Do NOT use for fetching data with existing sources, model inference, or Earth2Studio installation/setup tasks.

- Skill: `nvidia/earth2studio-create-datasource-3` (Agent Skill, multi-file: 18 files)
- Install (CLI): `npx skillmds@latest add nvidia/earth2studio-create-datasource-3`
- Raw SKILL.md: https://api.skillmd.com/api/skills/nvidia/earth2studio-create-datasource-3/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- License: Apache-2.0
- Author: NVIDIA (https://skillmd.com/u/nvidia)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/nvidia/earth2studio-create-datasource-3

---


# Create and Validate Data Source

## Purpose

End-to-end workflow for implementing a new Earth2Studio data source wrapper
that connects a remote data store (S3, GCS, Azure, HTTP, HuggingFace) to
Earth2Studio's async data fetching infrastructure — from analysis through
implementation, testing, validation, and PR submission.

## Prerequisites

- Earth2Studio dev environment with `uv` (`uv run python` must work)
- Git configured with fork (`origin`) and upstream (`upstream`) remotes
- Access to the target remote data store (credentials if private)
- Python 3.10+

> **⚠️ Credential Safety:** If the remote store requires authentication,
> credentials must be passed via environment variables only. Never hard-code
> secrets in source files. Before any git commit, verify no credentials are
> staged (check `git diff --cached` for API keys, tokens, or passwords).

## Workspace

Use the directory containing `pyproject.toml`. For Harbor evals, write to
`/workspace/output/` preserving paths. Never read `evals/targets/`.

## Instructions

> **Python Environment:** Always use `uv run python` or the local `.venv`.
> Never use the system Python directly.

Follow every step in order.

> **[CONFIRM] gates:** Step 1 (Source Type), Step 12 (Sanity-Check Plots),
> and Step 13 (Ready to Submit) require explicit user approval before
> proceeding. All other `[CONFIRM]` markers are advisory — present
> decisions inline and proceed without blocking.
>
> **Deliverables first:** Write the source file and test file (Steps 6–7)
> before extended exploration, documentation, registration, CHANGELOG, or PR
> work. Skip Steps 8–14 when the user asks for implementation only.
>
> **Before you finish:** Run verification commands in the repo root so results
> appear in the session log:
>
> ```bash
> uv run pytest test/data/test_<source>.py -x
> make format && make lint
> ```
>
> **Be concise:** Avoid long architecture reports; summarize decisions in a
> few sentences and move on to file writes.
>
> **Hangs or User Feedback** If agent becomes stuck or user provides a
> correction during this skills use, conservatively review relevant part of
> the skill and improve. Be concise.
>
> **One source type per invocation.** Invoke again for companion types.

### Reference Files

Load these on demand during the relevant steps:

| File | Content | Load at |
|---|---|---|
| `references/reference-implementation.py` | Obstore-backed source skeleton with FILL comments | Steps 6–10 |
| `references/reference-lexicon.py` | Lexicon skeleton with FILL comments | Step 4 |
| `references/testing-guide.py` | Test skeleton with FILL comments | Step 11 |
| `references/validation-guide.md` | Plot templates, PR body template, Greptile handling | Steps 12–14 (optional, for templates) |

---

### Workflow Overview

```text
Step 0: Obtain reference → Step 1: Determine type → Step 2: Dependencies
→ Step 3: Add deps → Step 4: Create lexicon → Step 5: Update vocab/schema
→ Step 6: Create skeleton → Step 7: Implement source → Step 8: Register
→ Step 9: Documentation → Step 10: CHANGELOG → Step 11: Tests
→ Step 12: Validate & plots (user confirms) → Step 13: PR + sanity comment
→ Step 14: Greptile review
```

---

### Step 0 — Obtain Remote Data Store Reference

If `$ARGUMENTS` is provided, use it (URL → WebFetch; file path → read).

If empty, ask:

> Please provide a URL, API documentation link, or description of the
> remote data store. This will be used to understand storage format,
> access pattern, variable inventory, temporal/spatial resolution.

---

### Step 1 — Determine Source Type

| Protocol | Returns | Has `lead_time`? | Use |
|---|---|---|---|
| **DataSource** | `xr.DataArray` | No | Gridded analysis/reanalysis |
| **ForecastSource** | `xr.DataArray` | Yes | Gridded forecast |
| **DataFrameSource** | `pd.DataFrame` | No | Sparse/station obs |
| **ForecastFrameSource** | `pd.DataFrame` | Yes | Sparse forecast obs |

Key factors: gridded vs sparse → DataArray vs DataFrame; analysis vs forecast → Source vs ForecastSource.

#### [CONFIRM — Source Type]

Present recommended type with justification. Ask for confirmation.

---

### Step 2 — Examine Remote Store & Propose Dependencies

**Analyze:** storage backend, file format, authentication, access pattern,
temporal/spatial resolution, variable inventory.

**Prefer Earth2Studio obstore helpers:**

| Backend / format | Preferred | Avoid |
|---|---|---|
| S3/GCS/HTTP object files | `obstore_store_from_url` + `obstore_fetch_to_cache` | raw SDK clients |
| Cloud Zarr | `obstore_zarr_store` | hand-rolled fsspec/zarr stores |
| Azure/object auth | `obstore.store.from_url` via shared helpers | provider SDKs directly |
| HuggingFace | `huggingface_hub` (core dep) | custom scripts |

Only fall back to dedicated libraries when obstore/shared helpers cannot access the store.

Check `pyproject.toml` — only propose packages not already present.
Core deps include: `obstore`, `zarr`, `netCDF4`, `h5py`, `pygrib`,
`huggingface-hub`, `pandas`, `pyarrow`.

#### [CONFIRM — Dependencies & Access Pattern]

Present: backend, obstore helper/store URL, new packages (with license), auth method.

---

### Step 3 — Add Dependencies

> **Load `references/reference-implementation.py` from Step 6 through Step 10.**
> Load `references/reference-lexicon.py` during Step 4.

If new packages needed:

1. Present the package name(s), version constraints, and licenses to the user
2. Confirm the user approves adding them before proceeding
3. `uv add --extra data <package>`
4. `uv lock`
5. Add optional dependency imports using `OptionalDependencyFailure` pattern

---

### Step 4 — Create Lexicon Class

Create `earth2studio/lexicon/<source_name>.py` with:

- `metaclass=LexiconType`
- `VOCAB: dict[str, str]` mapping E2S names → remote keys
- `get_item(cls, val)` returning `tuple[str, Callable]`
- Use `::` separator for structured keys
- Use `references/reference-lexicon.py` as the starting template

Map remote variables against `E2STUDIO_VOCAB` (282 entries in
`earth2studio/lexicon/base.py`).

#### [CONFIRM — Lexicon & Variable Mapping]

Present: class name, key format, full mapping table, modifiers, reference URL.

---

### Step 5 — Update E2STUDIO_VOCAB / SCHEMA (if needed)

- New vocab: surface = descriptive abbrev; pressure = `{name}{level}`
- New schema fields: DataFrame sources only, check `E2STUDIO_SCHEMA`

#### [CONFIRM — Vocabulary & Schema Updates]

Skip if no updates needed.

---

### Step 6 — Create Skeleton Data Source File

Use `references/reference-implementation.py` as the starting template.

Follow canonical method ordering:

1. Class constants
2. SCHEMA
3. `__init__`
4. `_async_init`
5. `__call__`
6. `fetch`
7. `_create_tasks`
8. `fetch_wrapper`
9. `fetch_array`
10. `_validate_time`
11. Helpers
12. `cache` property
13. `available` classmethod

Use async task dataclass pattern for parallel execution.

#### [CONFIRM — Skeleton]

Present: class name, file path, skeleton code, task dataclass.

---

### Step 7 — Implement the Data Source & Tests

> **The test file is a co-equal deliverable.** Create `test/data/test_<filename>.py`
> alongside the source. Add `test_<source>_call_mock` for async sources.

**Sync sources:** Use `prep_data_inputs`/`prep_forecast_inputs`, direct `__call__`.

**Async sources:** See `references/reference-implementation.py` for required patterns:
`_sync_async`, `gather_with_concurrency`, `async_retry`, `obstore_fetch_to_cache`,
`obstore_zarr_store` for Zarr, `try/finally` cleanup. Constructor params:
`cache=True`, `verbose=True`, `async_timeout=600`, `async_workers=None`,
`retries=3`. DataFrame sources add `time_tolerance`.

---

### Step 8 — Register the Source

- `earth2studio/data/__init__.py` — alphabetical import
- `earth2studio/lexicon/__init__.py` — alphabetical import
- Verify `pyproject.toml` deps

---

### Step 9 — Update Documentation

- Add the class alphabetically to the appropriate autosummary block in
  `docs/modules/datasources_analysis.md`, `datasources_forecast.md`, or
  `datasources_dataframe.md`
- Class docstring: Parameters, Warning (download size), Note (reference URLs),
  Badges (last)
- Keep the existing `region`, `dataclass`, and `product` badges accurate. Add
  `provider:<name>` when the service provider is known, and add
  `dataset:<family>` only when the source belongs to a named dataset family;
  do not infer a family from the provider. Use `dataset:gfs` for GEFS sources.
- Reuse lowercase badge keys already defined in `mkdocs.yml`. If a new
  provider or dataset family is required, add its badge definition there and
  use `hide_in: [autosummary, filter]`; the catalog reads these badges even
  though data-source API tables do not display or filter on them.
- Example: `region:global dataclass:reanalysis dataset:era5 product:atmos
  provider:google`
- All public methods: NumPy-style docstrings

---

### Step 10 — Update CHANGELOG.md

Add entry under the current unreleased version. See
`references/reference-implementation.py` REGISTRATION CHECKLIST for the format.

One line per source. Do NOT add separate lexicon entries.

---

### Step 11 — Verify Style & Expand Tests

Run `make format && make lint && make license`. Load `references/testing-guide.py`
for test skeletons. Required tests: `test_<source>_fetch` (slow), `_cache` (slow),
`_call_mock`, `_exceptions`, `_available`. Target 90%+ coverage with `--slow`.

#### [CONFIRM — Tests]

Present test file, functions, coverage.

---

### Step 12 — Validate Variables & Sanity-Check

1. Validate all lexicon vars against real data (run script, do NOT commit)
2. Remove variables with < 10% valid data
3. Create sanity-check plot (gridded or sparse template)
4. Tell user the plot path and ask for visual confirmation

#### [CONFIRM — Sanity-Check Plots]

User MUST visually inspect plots. Do not proceed without confirmation.

---

### Step 13 — Branch, Commit & Open PR

> **⚠️ WARNING: This step performs irreversible git operations (push, PR
> creation). Do NOT proceed without explicit user confirmation. Present all
> planned git commands to the user and wait for approval.**

1. Create branch `feat/data-source-<name>`
2. Commit (do NOT add sanity-check script/images)
3. **Present full command list to user for review**
4. Push to fork (only after user approves)
5. `gh pr create --repo NVIDIA/earth2studio` (only after user approves)
6. **Immediately post sanity-check validation as PR comment** with:
   - Variable coverage table (name, count, range, unit)
   - Data validation summary (regions, storms/stations, time range)
   - Key findings (physically reasonable values, conversions verified)
   - Full validation script in `<details>` block
   - Image placeholder: `<!-- Drag and drop sanity-check image here -->`

#### [CONFIRM — Ready to Submit] (MANDATORY)

**This gate is MANDATORY and blocks all git push/PR operations.**
Present the following to the user and wait for explicit approval:

1. List of all files to be committed
2. Full git commands that will be executed (push, PR create)
3. Target repository and branch
4. Warning: "These operations are irreversible and will be visible publicly"

Do NOT push or create PRs until the user explicitly confirms.

---

### Step 14 — Automated Code Review

1. Poll for Greptile review (5 min timeout)
2. Categorize feedback (bug/style/perf/docs/suggestion/false-positive)
3. Present triage table to user
4. **Wait for user to approve which comments to address and which to dismiss**
5. Implement accepted fixes
6. Respond to PR comments (only with user-approved actions)
7. Push (with user confirmation)

#### [CONFIRM — Review Triage]

User approves which comments to address and which to dismiss. Never
dismiss code review feedback autonomously.

---

## Examples

```text
User: Add a data source for the NOAA GFS analysis on S3
Agent: [loads skill, proceeds through Steps 0–14]
```

---

## Limitations

- One source type per invocation
- Requires network access for validation (Step 12)
- SPDX license headers required in all files

---

## Reminders

**DO:** `uv run python`, `loguru.logger`, alphabetical order in `__init__.py`/RST/CHANGELOG,
canonical method ordering, obstore helpers, `gather_with_concurrency`, `async_retry`,
reference URLs in docstrings, `try/finally` cleanup.

**AVOID:** hand-rolled fsspec stores, `asyncio.to_thread`, bare `tqdm.gather`,
xarray for loading, full file downloads.

**NEVER:** `loop.set_default_executor()`, commit secrets, commit sanity-check scripts/images.