Data Validation Setup

Cloud Data Validation Setup

One-time infrastructure setup for validating migrated data between a source database and Snowflake using the Cloud Data Validation feature via the scai CLI.

Supported sources: SQL Server, Redshift, Oracle, Teradata, PostgreSQL, Snowflake Supported target: Snowflake

Prerequisite

Load ../../data-infrastructure/SKILL.md first. It handles shared prerequisites, compute pool registration, and worker config (source host/port/credentials, source database, source schema). Return here after it completes.

For migration-source projects, validation requires target tables to already be deployed to Snowflake. For Snowflake-to-Snowflake validation, the source and target objects must already exist; do not run deployment or data migration first.

Step 1: Verify Orchestrator Service

After the shared infrastructure step has registered the compute pool, confirm the orchestrator service started:

SELECT SYSTEM$GET_SERVICE_STATUS('SNOWCONVERT_AI.DATA_MIGRATION.DATA_MIGRATION_SERVICE');

If it returns [] (suspended/not started), resume it manually:

ALTER SERVICE SNOWCONVERT_AI.DATA_MIGRATION.DATA_MIGRATION_SERVICE RESUME;

Wait 30-60s and re-check until status shows READY.

This completes infrastructure setup. The actual validation is started later by the validate-objects skill via:

1. progress_setup(mode="data_validation") — choose full vs incremental (+ sync strategy).
2. validate_data(mode="setup", where=..., validation_type=..., sync_strategy=...) — generates artifacts/data_validation/workflows/<hash>.yaml and patches known toggles / sync strategy.
3. data_infrastructure(mode="up") — bring the shared orchestrator + worker up once. Snowflake-source projects bring up only the orchestrator (start_worker=false).
4. Agent edits (e.g. watermarkColumn, partition columns) then validate_data(mode="run", workflow_path=...) — pure dispatch (scai data validate create-workflow) against the already-running infrastructure.

Do not run scai data validate create-workflow directly — use validate_data for dispatch and data_infrastructure for the orchestrator/worker lifecycle. Do not write ad-hoc comparison scripts — use the validation workflow YAML and validate_data instead.

When editing validation workflows, load Workflow Config Reference for camelCase field names (sourceWhereClause, targetWhereClause, indexColumnList, …) and accepted snake_case/legacy aliases. See common edit scenarios there (including excluding drift-prone timestamp columns from row compare before validate_data(mode="run")). For task-level debugging, see Task model reference.

Checklist

Shared infrastructure checklist is owned by ../../data-infrastructure/SKILL.md. Validation-specific items:

- [ ] Compute pool passed to data_infrastructure(mode="up", compute_pool=...) — SPCS only
- [ ] Worker config has no remaining <placeholder> values — unless pure Iceberg or Snowflake-to-Snowflake validation
- [ ] Level 1 scai data doctor — no Fail checks (see data-infrastructure skill)
- [ ] Data Validation Service running (READY)

Return control to the parent skill.

Reference

• Advanced operations reference — rate limiting, incremental DV, revalidate (when the customer asks)
• Workflow Config Reference — field-by-field reference for the per-call validation workflow YAML (validationConfiguration, per-table overrides, mappings, advanced). Load this when editing the file generated by validate_data(mode="setup").
• Task Model Reference
• Data Doctor reference
• Teardown (cost-saving suspend)