Osdu Data Load

OSDU Data Load

Execute directly. Do NOT delegate to an agent or sub-agent.

Step 1: Parse the user's intent

Step 2: Verify prerequisites

Before running any command, check:

1. uv is installed: uv --version
2. Data repos are cloned (the script will tell you if they're missing, but you can check proactively):
   • data-definitions — needed for reference-data, schemas, activity-templates
   • open-test-data — needed for tno, volve, nopims

If repos are missing, clone them using the clone skill or the commands shown in the script output.

For CIMPL environments, kubectl must be configured and pointing at the cluster. The script auto-detects everything else.

Step 3: Run the command

The script path (relative to plugin root):

LOAD_SCRIPT="skills/osdu-data-load/scripts/load.py"

List Available Datasets

uv run "$LOAD_SCRIPT" datasets

Check What's Loaded

uv run "$LOAD_SCRIPT" check --dataset reference-data
uv run "$LOAD_SCRIPT" check --dataset all

Load Data

Two modes:

• --direct (recommended): Uses Storage API directly. Fast, synchronous.
• Default (Workflow): Submits to Airflow DAG pipeline. Slower but validates schema/integrity.

# Load reference data (fast direct mode)
uv run "$LOAD_SCRIPT" load --dataset reference-data --direct

# Load TNO test dataset
uv run "$LOAD_SCRIPT" load --dataset tno --direct

# Load everything
uv run "$LOAD_SCRIPT" load --dataset all --direct

# Preview without submitting
uv run "$LOAD_SCRIPT" load --dataset reference-data --dry-run

# Filter by filename
uv run "$LOAD_SCRIPT" load --dataset reference-data --direct --filter seismic

Dataset Registry

Environment Configuration

CIMPL Environments (Zero Config)

If kubectl is configured and pointing at a CIMPL cluster, the script auto-detects everything:

• Reads datafier-secret from the osdu namespace
• Derives Keycloak token URL from the platform namespace
• Manages port-forwards automatically (Keycloak 18082, Storage 18081, Search 18084, Workflow 18083)
• Defaults to osdu-demo-legaltag legal tag

Azure ADME / Helm Environments

export OSDU_URL="https://platform5552.energy.azure.com"
export OSDU_DATA_PARTITION="opendes"
export OSDU_CLIENT_ID="<client-id>"
export OSDU_CLIENT_SECRET="<client-secret>"
export OSDU_TENANT_ID="<tenant-id>"
export OSDU_LEGAL_TAG="opendes-legal-tag-load"

Keycloak Environments (Manual)

export OSDU_URL="https://osdu.myenv.example.com"
export OSDU_DATA_PARTITION="osdu"
export OSDU_CLIENT_ID="datafier"
export OSDU_CLIENT_SECRET="<secret>"
export OSDU_TOKEN_URL="https://keycloak.myenv.example.com/realms/osdu/protocol/openid-connect/token"
export OSDU_LEGAL_TAG="osdu-demo-legaltag"

Required Data Repos

git clone https://community.opengroup.org/osdu/data/data-definitions.git ~/workspace/data-definitions
git clone https://community.opengroup.org/osdu/data/open-test-data.git ~/workspace/open-test-data

Environment Variables

Error Handling