# Watsonx Data Fabric

> Build anything in the IBM watsonx / Cloud Pak for Data 'Data Fabric' stack from the CLI: provision watsonx.data lakehouse (Presto/Spark/Milvus, Iceberg catalogs, COS buckets), IBM Knowledge Catalog (catalogs, projects, data assets, glossaries, policies, masking, lineage), DataStage flows, Data Virtualization, Watson Query, Master Data Management / Match 360, watsonx.ai foundation-model deployments, watsonx.governance factsheets, COS buckets, IAM access groups, trusted profiles, service IDs, API keys. USE FOR: ibmcloud login, ibmcloud resource service-instance create, ibmcloud plugin install cloud-pak-for-data / cloud-object-storage / code-engine, cpdctl wx-data engine create, cpdctl wx-data bucket register, cpdctl asset data-asset create, cpdctl project create, cpdctl wx-ai deployment create, cpdctl datastage flow create, Presto/Iceberg/Spark/Milvus on watsonx.data, IKC catalog + glossary + governance rules, watsonx.ai prompt + RAG deploy, watsonx.governance factsheet, design IBM data fabric reference archite

- Skill: `slysik/watsonx-data-fabric` (Agent Skill, multi-file: 23 files)
- Install (CLI): `npx skillmds@latest add slysik/watsonx-data-fabric`
- Raw SKILL.md: https://api.skillmd.com/api/skills/slysik/watsonx-data-fabric/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- License: MIT
- Author: slysik (https://skillmd.com/u/slysik)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/slysik/watsonx-data-fabric

---


# IBM watsonx Data Fabric Skill

Purpose: stand up and operate any component of the IBM **Data Fabric** (the
combined **Cloud Pak for Data + watsonx** stack) from the command line. The
skill is opinionated toward **watsonx as a Service on IBM Cloud** (the
`ibmcloud login --apikey "$IBM_API_KEY" -r us-south` path) but every recipe
also works on Cloud Pak for Data Software (`cpdctl` against a CP4D URL).

## What IBM calls "Data Fabric"

| Capability | IBM product | Primary CLI |
|---|---|---|
| Lakehouse (Iceberg + Presto + Spark + Milvus) | **watsonx.data** | `cpdctl wx-data ...` (+ `ibmcloud resource service-instance`) |
| Catalog, glossary, lineage, masking | **IBM Knowledge Catalog (IKC)** | `cpdctl asset ...`, `cpdctl project ...` |
| ETL / data integration | **DataStage** (on CP4D / as a Service) | `cpdctl datastage ...` |
| Federated query | **Data Virtualization / Watson Query** | `cpdctl connection ...` + SQL |
| Master data / 360-degree view | **Match 360 / MDM** | REST + `cpdctl` |
| Foundation-model platform | **watsonx.ai** | `cpdctl wx-ai ...` |
| AI governance, factsheets | **watsonx.governance** | `cpdctl` + REST |
| Object storage backing lakehouse | **IBM Cloud Object Storage (COS)** | `ibmcloud cos ...` |
| Identity, RBAC | **IBM Cloud IAM** | `ibmcloud iam ...` |
| Container runtime for services/agents | **Code Engine** | `ibmcloud ce ...` |

## When to load which sub-doc

| Sub-doc | Use when |
|---|---|
| [auth/auth.md](auth/auth.md) | First-time setup, switching accounts, service IDs, API keys, `ibmcloud` plugins, `cpdctl` install + profiles, trusted profiles, MFA. |
| [watsonx-data/watsonx-data.md](watsonx-data/watsonx-data.md) | Provision a watsonx.data instance, register COS buckets, create Iceberg/Hive catalogs, provision Presto (Java/C++)/Spark/Milvus engines, run SQL, ingest, submit Spark jobs. |
| [knowledge-catalog/knowledge-catalog.md](knowledge-catalog/knowledge-catalog.md) | Create IKC catalogs and projects, register connections, create data assets, build glossary/categories/terms, attach governance rules, configure masking and row-level rules, browse lineage. |
| [watsonx-ai/watsonx-ai.md](watsonx-ai/watsonx-ai.md) | Deploy foundation models (provided, deploy-on-demand, custom), Prompt Lab + REST `text/generation`, RAG with Milvus, tuning, factsheets. |
| [datastage-dv/datastage-dv.md](datastage-dv/datastage-dv.md) | DataStage flows, Data Virtualization / Watson Query connections, Match 360 / MDM stubs. |
| [recipes.md](recipes.md) | End-to-end recipes: greenfield data fabric in one script, lakehouse + IKC governance + Prompt Lab RAG, dbt-on-Presto, lineage publishing. |
| [lessons-learned.md](lessons-learned.md) | **Read this first when a recipe fails.** Dated log of every gotcha we've actually hit (wrong plan name, secret-vs-name confusion, ADLS-only fields rejected for COS, async polling, CRN vs GUID, frozen-account behavior, etc.) plus reusable patterns (idempotent provisioning, secret hygiene, discover-before-guess). |
| [v3-api/v3-api.md](v3-api/v3-api.md) | **Canonical v3 REST reference**, verified against the official OpenAPI spec snapshot in [`v3-api/watsonxdata-v3.openapi.json`](v3-api/watsonxdata-v3.openapi.json). Use when cpdctl's stricter client-side validation gets in your way, when you need a curl-only path for CI, or when the Python SDK (still v2) is missing an operation. |
| [sales-enablement/README.md](sales-enablement/README.md) | **Data Fabric SME sales enablement** — 4-pillar product expertise (watsonx.data, watsonx.data intelligence, watsonx.data integration, Confluent), competitive battlecards (Databricks, Snowflake, AWS, Informatica, Collibra, Redpanda), IBM value assessment methodology, persona talk tracks (CDO/CFO/CTO/Eng/Architect/LOB), and the cross-portfolio integrated story. Use when prepping for sales calls, value assessments, RFPs, or the IBM Level-3 product certification. |

## Mental model

```
                ┌───────────────── IBM Cloud / Cloud Pak for Data ─────────────────┐
                │                                                                  │
   ibmcloud  ──▶│  IAM (users, service IDs, access groups, trusted profiles)       │
   login        │     │                                                            │
                │     ▼                                                            │
                │  ┌────────────────────────────────────────────────────────────┐  │
                │  │ IBM Knowledge Catalog (IKC) ── governance plane            │  │
                │  │  catalogs · glossary · rules · masking · lineage           │  │
                │  └─────────────┬──────────────────────────┬───────────────────┘  │
                │                │ policies                 │ assets/lineage       │
                │  ┌─────────────▼─────────┐  ┌─────────────▼────────────────────┐ │
                │  │ watsonx.data (lake-   │  │ DataStage / Data Virtualization  │ │
                │  │ house): Presto C++/   │  │ Match 360 / MDM                  │ │
                │  │ Java + Spark + Milvus │  │ (movement, federation, MDM)      │ │
                │  │ on Iceberg in COS     │  └──────────────────────────────────┘ │
                │  └─────────────┬─────────┘                                       │
                │                │ governed SQL / vectors                          │
                │  ┌─────────────▼────────────────────────────────────────────┐    │
                │  │ watsonx.ai  (foundation models, prompt lab, tuning, RAG) │    │
                │  │     + watsonx.governance (factsheets, model risk)        │    │
                │  └──────────────────────────────────────────────────────────┘    │
                └──────────────────────────────────────────────────────────────────┘
```

Read that as: **IKC governs**, **watsonx.data stores + queries**, **DataStage
moves**, **watsonx.ai reasons**, **watsonx.governance audits**. COS is the
underlying object store; IAM is the front door.

## Required environment

```bash
# Required for every recipe in this skill
export IBM_API_KEY=...                       # IBM Cloud user or service-ID API key
export IBM_REGION=us-south                   # us-south, us-east, eu-de, eu-gb, au-syd, ca-tor, jp-tok
export IBM_RESOURCE_GROUP=Default            # or a dedicated rg, e.g. rg-datafabric
export IBM_ACCOUNT_ID=$(ibmcloud account show --output json 2>/dev/null | jq -r .account_id)
# watsonx.data + watsonx.ai use the same IAM identity but separate service CRNs.
export WXD_INSTANCE_NAME=wxd-prod
export WXAI_INSTANCE_NAME=wxai-prod
export COS_INSTANCE_NAME=cos-datafabric
```

## Bootstrapping the CLI (full detail in `auth/auth.md`)

```bash
# 1. Log in (the auth step the user asked for)
ibmcloud login --apikey "$IBM_API_KEY" -r "$IBM_REGION"
ibmcloud target -g "$IBM_RESOURCE_GROUP"

# 2. Plugins for every part of the fabric
ibmcloud plugin install -f cloud-object-storage
ibmcloud plugin install -f cloud-pak-for-data       # provisioning + watsonx services
ibmcloud plugin install -f code-engine              # runtime for FastAPI/agents
ibmcloud plugin install -f kubernetes-service       # if you use IKS/ROKS
ibmcloud plugin install -f schematics               # for Terraform-style automation

# 3. cpdctl (the actual data-fabric CLI: assets, projects, watsonx.data, watsonx.ai)
platform=$(uname -s | tr '[A-Z]' '[a-z]'); arch=$(uname -m | sed 's/x86_64/amd64/')
curl -fsSLO "https://github.com/IBM/cpdctl/releases/latest/download/cpdctl_${platform}_${arch}.tar.gz"
tar -xzf "cpdctl_${platform}_${arch}.tar.gz" && sudo mv cpdctl /usr/local/bin/

# 4. Link cpdctl to your ibmcloud session (no second login needed)
cpdctl config profile set wx --ibmcloud
cpdctl config profile use wx
cpdctl version && cpdctl wx-data --help
```

## Defaults & conventions used in this skill

- **One COS instance per environment** (`cos-<env>`), one **bucket per
  catalog**: `wxd-<env>-<domain>` (e.g. `wxd-prod-finance`). Region-local
  buckets only — never cross-region for the lakehouse.
- **Catalog naming**: `iceberg_<domain>` for Iceberg, `hive_<domain>` for
  Hive-format, `mv_<purpose>` for Milvus vector collections.
- **Engines**: separate `presto_bi` (Java, for governed SQL/BI), `presto_fast`
  (C++, for high-throughput analytics), `spark_etl` (for ingestion + Iceberg
  maintenance), `spark_ai` (for ML/feature workloads). Bills are per-engine,
  so isolate.
- **Projects vs catalogs vs spaces** (IKC/watsonx):
  - **Catalog** = governed, shared, asset-level access control.
  - **Project** = workbench (notebooks, jobs, flows, models) for a team.
  - **Space** = deployment target for ML/AI assets.
- **Governance first**: register every connection in IKC before pointing
  watsonx.data or DataStage at it. That is what makes lineage and masking
  flow end-to-end — the JD-style "data fabric" promise.
- **Wrap every Cortex-equivalent (watsonx.ai) call in a deployment** so
  prompts, models, and guardrails live in source control, not ad-hoc HTTP.

## 60-second smoke test (after auth)

```bash
ibmcloud resource service-instances --service-name lakehouse  # watsonx.data
ibmcloud resource service-instances --service-name pm-20      # watsonx.ai (Watson ML)
ibmcloud resource service-instances --service-name cloud-object-storage
cpdctl project list
cpdctl wx-data engine list   # requires WX_DATA_INSTANCE_ID set
cpdctl wx-ai deployment list
```

If any of those fail, jump to `auth/auth.md` → "Troubleshooting".

## Quick recipes (load full sub-doc for details)

```bash
# (1) Provision a watsonx.data Lite instance and capture its ID
ibmcloud resource service-instance-create "$WXD_INSTANCE_NAME" \
  lakehouse lite "$IBM_REGION" \
  -p '{"use_case":"generative_ai"}'
export WX_DATA_INSTANCE_ID=$(ibmcloud resource service-instance \
  "$WXD_INSTANCE_NAME" --output json | jq -r '.[0].guid')

# (2) Register an existing COS bucket as an Iceberg catalog
cpdctl wx-data bucket register \
  --bucket-name wxd-prod-finance \
  --bucket-type ibm_cos \
  --endpoint "s3.us-south.cloud-object-storage.appdomain.cloud" \
  --access-key "$COS_HMAC_AK" --secret-key "$COS_HMAC_SK" \
  --catalog-name iceberg_finance --catalog-type iceberg

# (3) Provision a Presto (Java) engine and associate the catalog
cpdctl wx-data engine create --engine-name presto_bi \
  --engine-type presto --engine-version 0.294 --engine-size small
cpdctl wx-data catalog associate \
  --catalog-name iceberg_finance --engine-name presto_bi

# (4) Create an IKC project, register a connection, create a data asset
cpdctl project create --name fin-analytics --output json \
  --storage '{"type":"assetfiles","guid":"'"$(uuidgen)"'"}'
# then: cpdctl asset data-asset create  (see knowledge-catalog/knowledge-catalog.md)

# (5) Deploy a watsonx.ai foundation model and call it
cpdctl wx-ai deployment create --name granite-13b-prod \
  --asset-id <foundation_model_asset_id> \
  --space-id "$WX_SPACE_ID" \
  --hardware-spec-name S
```

## Cross-skill notes

- For Snowflake parallels (Cortex Search ≈ watsonx.data + Milvus, Cortex
  Analyst ≈ watsonx.ai text-to-SQL on a semantic layer), see the
  `snowflake-cortex` skill.
- For browser-driven UI verification of provisioned consoles, see `bowser`.