Steadfastasart Geoscience Skills Pooch

Pooch - Data File Fetching

Quick Reference

import pooch

# Download single file
file_path = pooch.retrieve(
    url="https://example.com/data.csv",
    known_hash="sha256:abc123...",  # None to skip verification
    fname="data.csv",
    path=pooch.os_cache("myproject")
)

# Create registry for multiple files
REGISTRY = pooch.create(
    path=pooch.os_cache("myproject"),
    base_url="https://example.com/data/",
    registry={"data.csv": "sha256:abc123...", "model.nc": "sha256:def456..."}
)
data_file = REGISTRY.fetch("data.csv")

# Generate hash for local file
file_hash = pooch.file_hash("/path/to/file.csv")

Key Functions

Essential Operations

Download Files

# With hash verification
file_path = pooch.retrieve(
    url="https://example.com/data.nc",
    known_hash="sha256:abc123..."
)

# Without verification (development only)
file_path = pooch.retrieve(url="https://example.com/data.nc", known_hash=None)

# From Zenodo DOI
file_path = pooch.retrieve(
    url="doi:10.5281/zenodo.1234567/data.zip",
    known_hash="sha256:abc123..."
)

Extract Archives

# ZIP archive
files = pooch.retrieve(
    url="https://example.com/data.zip",
    known_hash="sha256:abc123...",
    processor=pooch.Unzip()
)

# Decompress single gzip file
file_path = pooch.retrieve(
    url="https://example.com/data.csv.gz",
    known_hash="sha256:abc123...",
    processor=pooch.Decompress(name="data.csv")
)

Additional Options

# Progress bar for large downloads
file_path = pooch.retrieve(url=url, known_hash=hash, progressbar=True)

# HTTP authentication
file_path = pooch.retrieve(
    url="https://example.com/protected/data.csv",
    known_hash=None,
    downloader=pooch.HTTPDownloader(auth=("user", "pass"))
)

Processor Options

Cache Locations

Error Handling

try:
    file_path = pooch.retrieve(url=url, known_hash=hash)
except pooch.exceptions.HTTPDownloadError:
    print("Download failed - check URL")
except pooch.exceptions.DownloadError:
    print("Network issue")

When to Use vs Alternatives

Use pooch when you need reproducible data downloads with automatic caching and integrity verification, especially for scientific data registries.

Consider alternatives when you need full data version control with git integration (use DVC), simple one-off downloads without caching needs (use requests), or command-line batch downloads (use wget).

Common Workflows

Set up reproducible data download with registry

• Identify all required data files and their URLs
• Generate SHA256 hashes with pooch.file_hash() for each file
• Create registry with pooch.create() specifying base URL and file hashes
• Fetch files with REGISTRY.fetch() in analysis scripts
• Add processors for compressed files (Unzip(), Untar(), Decompress())
• Test registry by clearing cache and re-fetching
• Document registry in project README for collaborators

References

• Registry Configuration - Create and manage file registries

Scripts

• scripts/create_registry.py - Generate registry from local files

Converted and distributed by TomeVault — claim your Tome and manage your conversions.