Watsonx Data Fabric

IBM watsonx Data Fabric Skill

Purpose: stand up and operate any component of the IBM Data Fabric (the combined Cloud Pak for Data + watsonx stack) from the command line. The skill is opinionated toward watsonx as a Service on IBM Cloud (the ibmcloud login --apikey "$IBM_API_KEY" -r us-south path) but every recipe also works on Cloud Pak for Data Software (cpdctl against a CP4D URL).

What IBM calls "Data Fabric"

When to load which sub-doc

Mental model

                ┌───────────────── IBM Cloud / Cloud Pak for Data ─────────────────┐
                │                                                                  │
   ibmcloud  ──▶│  IAM (users, service IDs, access groups, trusted profiles)       │
   login        │     │                                                            │
                │     ▼                                                            │
                │  ┌────────────────────────────────────────────────────────────┐  │
                │  │ IBM Knowledge Catalog (IKC) ── governance plane            │  │
                │  │  catalogs · glossary · rules · masking · lineage           │  │
                │  └─────────────┬──────────────────────────┬───────────────────┘  │
                │                │ policies                 │ assets/lineage       │
                │  ┌─────────────▼─────────┐  ┌─────────────▼────────────────────┐ │
                │  │ watsonx.data (lake-   │  │ DataStage / Data Virtualization  │ │
                │  │ house): Presto C++/   │  │ Match 360 / MDM                  │ │
                │  │ Java + Spark + Milvus │  │ (movement, federation, MDM)      │ │
                │  │ on Iceberg in COS     │  └──────────────────────────────────┘ │
                │  └─────────────┬─────────┘                                       │
                │                │ governed SQL / vectors                          │
                │  ┌─────────────▼────────────────────────────────────────────┐    │
                │  │ watsonx.ai  (foundation models, prompt lab, tuning, RAG) │    │
                │  │     + watsonx.governance (factsheets, model risk)        │    │
                │  └──────────────────────────────────────────────────────────┘    │
                └──────────────────────────────────────────────────────────────────┘

Read that as: IKC governs, watsonx.data stores + queries, DataStage moves, watsonx.ai reasons, watsonx.governance audits. COS is the underlying object store; IAM is the front door.

Required environment

# Required for every recipe in this skill
export IBM_API_KEY=...                       # IBM Cloud user or service-ID API key
export IBM_REGION=us-south                   # us-south, us-east, eu-de, eu-gb, au-syd, ca-tor, jp-tok
export IBM_RESOURCE_GROUP=Default            # or a dedicated rg, e.g. rg-datafabric
export IBM_ACCOUNT_ID=$(ibmcloud account show --output json 2>/dev/null | jq -r .account_id)
# watsonx.data + watsonx.ai use the same IAM identity but separate service CRNs.
export WXD_INSTANCE_NAME=wxd-prod
export WXAI_INSTANCE_NAME=wxai-prod
export COS_INSTANCE_NAME=cos-datafabric

Bootstrapping the CLI (full detail in auth/auth.md)

# 1. Log in (the auth step the user asked for)
ibmcloud login --apikey "$IBM_API_KEY" -r "$IBM_REGION"
ibmcloud target -g "$IBM_RESOURCE_GROUP"

# 2. Plugins for every part of the fabric
ibmcloud plugin install -f cloud-object-storage
ibmcloud plugin install -f cloud-pak-for-data       # provisioning + watsonx services
ibmcloud plugin install -f code-engine              # runtime for FastAPI/agents
ibmcloud plugin install -f kubernetes-service       # if you use IKS/ROKS
ibmcloud plugin install -f schematics               # for Terraform-style automation

# 3. cpdctl (the actual data-fabric CLI: assets, projects, watsonx.data, watsonx.ai)
platform=$(uname -s | tr '[A-Z]' '[a-z]'); arch=$(uname -m | sed 's/x86_64/amd64/')
curl -fsSLO "https://github.com/IBM/cpdctl/releases/latest/download/cpdctl_${platform}_${arch}.tar.gz"
tar -xzf "cpdctl_${platform}_${arch}.tar.gz" && sudo mv cpdctl /usr/local/bin/

# 4. Link cpdctl to your ibmcloud session (no second login needed)
cpdctl config profile set wx --ibmcloud
cpdctl config profile use wx
cpdctl version && cpdctl wx-data --help

Defaults & conventions used in this skill

• One COS instance per environment (cos-<env>), one bucket per catalog: wxd-<env>-<domain> (e.g. wxd-prod-finance). Region-local buckets only — never cross-region for the lakehouse.
• Catalog naming: iceberg_<domain> for Iceberg, hive_<domain> for Hive-format, mv_<purpose> for Milvus vector collections.
• Engines: separate presto_bi (Java, for governed SQL/BI), presto_fast (C++, for high-throughput analytics), spark_etl (for ingestion + Iceberg maintenance), spark_ai (for ML/feature workloads). Bills are per-engine, so isolate.
• Projects vs catalogs vs spaces (IKC/watsonx):
  • Catalog = governed, shared, asset-level access control.
  • Project = workbench (notebooks, jobs, flows, models) for a team.
  • Space = deployment target for ML/AI assets.
• Governance first: register every connection in IKC before pointing watsonx.data or DataStage at it. That is what makes lineage and masking flow end-to-end — the JD-style "data fabric" promise.
• Wrap every Cortex-equivalent (watsonx.ai) call in a deployment so prompts, models, and guardrails live in source control, not ad-hoc HTTP.

60-second smoke test (after auth)

ibmcloud resource service-instances --service-name lakehouse  # watsonx.data
ibmcloud resource service-instances --service-name pm-20      # watsonx.ai (Watson ML)
ibmcloud resource service-instances --service-name cloud-object-storage
cpdctl project list
cpdctl wx-data engine list   # requires WX_DATA_INSTANCE_ID set
cpdctl wx-ai deployment list

If any of those fail, jump to auth/auth.md → "Troubleshooting".

Quick recipes (load full sub-doc for details)

# (1) Provision a watsonx.data Lite instance and capture its ID
ibmcloud resource service-instance-create "$WXD_INSTANCE_NAME" \
  lakehouse lite "$IBM_REGION" \
  -p '{"use_case":"generative_ai"}'
export WX_DATA_INSTANCE_ID=$(ibmcloud resource service-instance \
  "$WXD_INSTANCE_NAME" --output json | jq -r '.[0].guid')

# (2) Register an existing COS bucket as an Iceberg catalog
cpdctl wx-data bucket register \
  --bucket-name wxd-prod-finance \
  --bucket-type ibm_cos \
  --endpoint "s3.us-south.cloud-object-storage.appdomain.cloud" \
  --access-key "$COS_HMAC_AK" --secret-key "$COS_HMAC_SK" \
  --catalog-name iceberg_finance --catalog-type iceberg

# (3) Provision a Presto (Java) engine and associate the catalog
cpdctl wx-data engine create --engine-name presto_bi \
  --engine-type presto --engine-version 0.294 --engine-size small
cpdctl wx-data catalog associate \
  --catalog-name iceberg_finance --engine-name presto_bi

# (4) Create an IKC project, register a connection, create a data asset
cpdctl project create --name fin-analytics --output json \
  --storage '{"type":"assetfiles","guid":"'"$(uuidgen)"'"}'
# then: cpdctl asset data-asset create  (see knowledge-catalog/knowledge-catalog.md)

# (5) Deploy a watsonx.ai foundation model and call it
cpdctl wx-ai deployment create --name granite-13b-prod \
  --asset-id <foundation_model_asset_id> \
  --space-id "$WX_SPACE_ID" \
  --hardware-spec-name S

Cross-skill notes

• For Snowflake parallels (Cortex Search ≈ watsonx.data + Milvus, Cortex Analyst ≈ watsonx.ai text-to-SQL on a semantic layer), see the snowflake-cortex skill.
• For browser-driven UI verification of provisioned consoles, see bowser.