Download Script Dev

Download Script Development Skill

Overview

Assist in developing, refining, and debugging data download scripts for GAIA hydroclimatological data sources. This skill provides templates, configuration schemas, and troubleshooting guidance for building reproducible data pipelines across 10+ environmental data sources.

Requirements

• Python 3.9+ with xarray, geopandas, rioxarray
• Source-specific libraries: herbie-data (HRRR), pyPRISMClimate (PRISM), obspy (IRIS), boto3 (WRF/S3), elevation (DEM), s3fs (CONUS404)
• System dependencies: wgrib2 for HRRR (install via conda-forge, not pip)

When to Use

• Developing a new download script for a GAIA data source
• Debugging an existing download script (timeouts, auth errors, CRS mismatches)
• Adapting a notebook pattern to a new use case or study area
• Validating download configuration parameters
• Understanding which access pattern or library to use for a data source

Script Structure Pattern

All download scripts follow a CONFIG-at-top pattern separating parameters from logic:

import xarray as xr
import geopandas as gpd
from concurrent.futures import ThreadPoolExecutor

# ============================================================
# Configuration — modify these parameters before running
# ============================================================
CONFIG = {
    "source": "SOURCE_NAME",
    "date_range": ("2024-01-01", "2024-01-31"),
    "variables": ["var1", "var2"],
    "aoi_path": "../data/GIS/boundary.json",
    "output_path": "../data/output.zarr",
    "output_format": "zarr",
    "max_workers": 8,
}
# ============================================================
# Download logic — generally no need to modify below this line
# ============================================================

def main():
    # 1. Load AOI
    aoi = gpd.read_file(CONFIG["aoi_path"])

    # 2. Download data (parallel)
    # 3. Combine datasets
    # 4. Spatial subset
    # 5. Derive variables (if needed)
    # 6. Save to output format
    # 7. Print QC summary
    pass

if __name__ == "__main__":
    main()

Four Data Access Patterns

1. Direct HTTP Download (PRISM, Stage IV, DEM)

Simple URL-based fetching with requests. Use ThreadPoolExecutor for parallel downloads. Handle retries for network failures.

2. REST API Query (USGS, Synoptic)

Parameterized endpoints returning JSON or RDB format. Build URL query strings from CONFIG parameters. Parse response formats appropriately (RDB requires custom parsing).

3. Cloud Object Storage / S3 (CONUS404, HRRR, WRF-CMIP6)

Access via s3fs, boto3, or library wrappers. Supports partial reads and lazy loading with xarray.open_zarr(). Use anonymous/unsigned credentials for public buckets.

4. Specialized Libraries (Herbie for HRRR, pyPRISMClimate, obspy for IRIS)

Domain-specific wrappers that handle authentication, URL construction, and data parsing internally. Consult library documentation for parameter conventions.

Spatial Subsetting Methods

Choose based on grid type:

Ensure the AOI CRS matches the data CRS before subsetting. Model grids often use Lambert Conformal Conic — reproject the AOI with aoi.to_crs(ds.rio.crs).

Parallel Download Pattern

from concurrent.futures import ThreadPoolExecutor, as_completed

def download_item(item):
    """Download a single item. Return path or dataset."""
    # ... download logic
    return result

with ThreadPoolExecutor(max_workers=CONFIG["max_workers"]) as executor:
    futures = {executor.submit(download_item, item): item for item in items}
    for i, future in enumerate(as_completed(futures), 1):
        result = future.result()
        print(f"Downloaded {i}/{len(items)}")

Worker count guidance: 4-8 for HTTP downloads, 8-16 for S3 reads, 2-4 for API endpoints with rate limits.

Output Formats

Common Issues and Debugging

wgrib2 Not Found (HRRR)

wgrib2 is a C binary, not pip-installable. Install via conda-forge: conda install -c conda-forge wgrib2 or pixi add wgrib2. Verify with shutil.which("wgrib2"). Add a runtime guard in scripts:

import shutil
assert shutil.which("wgrib2"), "wgrib2 not found. Install: conda install -c conda-forge wgrib2"

S3 Authentication Errors (CONUS404, WRF)

Public buckets require anonymous access. For CONUS404: set anon=True in s3fs.S3FileSystem(). For WRF: use botocore.UNSIGNED config in boto3.

# CONUS404
fs = s3fs.S3FileSystem(anon=True, client_kwargs={"endpoint_url": "https://usgs.osn.mghpcc.org"})

# WRF-CMIP6
from botocore import UNSIGNED
from botocore.config import Config
s3 = boto3.client("s3", config=Config(signature_version=UNSIGNED))

CRS Mismatch During Spatial Subsetting

If rio.clip() raises a CRS error, reproject the AOI to match the dataset:

aoi = aoi.to_crs(ds.rio.crs)  # reproject AOI to dataset CRS
ds_clipped = ds.rio.clip(aoi.geometry)

For datasets without CRS metadata, set it explicitly: ds.rio.write_crs("EPSG:4326", inplace=True).

Memory Issues with Large Datasets

Use chunked loading to avoid loading entire datasets into RAM:

ds = xr.open_dataset(path, chunks={"time": 100})  # lazy loading
print(f"Dataset size: {ds.nbytes / 1e9:.1f} GB")   # check before computing

Process in temporal batches rather than loading the full dataset. Use .compute() only on subsets.

Network Timeouts and Retries

Wrap downloads in retry logic with exponential backoff:

from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

session = requests.Session()
session.mount("https://", HTTPAdapter(max_retries=Retry(total=3, backoff_factor=2)))
response = session.get(url, timeout=60)

For S3, boto3 has built-in retry configuration via botocore.config.Config(retries={"max_attempts": 3}).

USGS RDB Format Parsing

USGS returns tab-separated RDB format with comment headers (#) and a data-type row below the column headers. Parse by skipping both:

lines = [l for l in response.text.splitlines() if not l.startswith("#")]
df = pd.read_csv(io.StringIO("\n".join([lines[0]] + lines[2:])), sep="\t")

Note: USGS returns data in local time zones. Convert to UTC using station timezone metadata.

Additional Resources

Reference Files

For detailed data source documentation, code templates, and configuration schemas, consult:

• references/sources/ — Per-source documentation files (e.g., sources/hrrr.md, sources/conus404.md): endpoints, response formats, authentication setup, and example API calls. Load only the source relevant to the current task.
• references/DOWNLOAD_PATTERNS.md — Complete code templates for each access pattern with full pipeline examples for HRRR, CONUS404, and USGS
• references/CONFIGURATION.md — Per-source parameter tables with types, defaults, and validation rules; size estimation formulas