Add Warehouse Adapter

Add a Warehouse Adapter

Step-by-step checklist for adding a new warehouse connection to Lightdash. Follow the Athena adapter as the canonical example (PRs #19751/#19752). The MotherDuck/DuckDB adapter is the most recent implementation.

Layer 1: Common Types (packages/common/)

• Add entry to WarehouseTypes enum in src/types/projects.ts
• Define CreateXxxCredentials type with type: WarehouseTypes.XXX discriminant
  • Sensitive fields (tokens, passwords, keys) should be optional (token?: string) — not required with empty-string sentinel
• Define XxxCredentials = Omit<CreateXxxCredentials, SensitiveCredentialsFieldNames>
• Add sensitive fields to sensitiveCredentialsFieldNames array if not already present (tokens, passwords, keys)
• Add CreateXxxCredentials to CreateWarehouseCredentials union
• Add XxxCredentials to WarehouseCredentials union
• Update UserWarehouseCredentials union in src/types/userWarehouseCredentials.ts
• Update UserWarehouseCredentialsWithSecrets union
• Add case to getFieldQuoteChar() in src/utils/warehouse.ts
• Verify getAggregatedField() in src/utils/warehouse.ts handles the new adapter (may fall into existing case)
• Check src/utils/timeFrames.ts adapter config map has the new adapter
• Check src/compiler/translator.ts convertTimezone() has the new adapter
• Run: pnpm -F common typecheck && pnpm -F common lint

Layer 2: Backend (packages/backend/)

Database Migration

• Create migration: pnpm -F backend create-migration add_xxx_warehouse_type
• up: Insert { warehouse_type: 'xxx' } into warehouse_types
• down: Delete xxx credentials and type

Entity & Profile

• Add 'xxx' to warehouseTypes array in src/database/entities/warehouseCredentials.ts
• Add case to credentialsTarget() in src/dbt/profiles.ts (before default)
  • Pass secrets via envVarReference()/envVar() pattern (not raw env var names or inline values)
  • Follow ClickHouse pattern: password: envVarReference('password') in target, [envVar('password')]: credentials.password in environment
• Verify quoteChars in src/dbt/DbtMetadataApiClient.ts has the adapter

Service Updates (exhaustive switch statements)

• src/services/ProjectService/ProjectService.ts - clearSecretsFromCredentials()
  • Blank sensitive fields (token: undefined) instead of destructuring + unsafe as cast
  • Follow the same pattern as other warehouses ({ ...credentials, password: '' })
• src/services/ProjectService/ProjectService.ts - warehouse client creation switch
• src/services/ProjectService/ProjectService.ts - user credentials creation switch
• src/services/ProjectService/ProjectService.ts - getDatabaseFromWarehouseCredentials()
• src/models/UserWarehouseCredentials/UserWarehouseCredentialsModel.ts
• Run: pnpm -F backend typecheck && pnpm -F backend lint

Layer 3: Warehouse Client (packages/warehouses/)

• Create src/warehouseClients/XxxWarehouseClient.ts
  • Extend WarehouseBaseClient<CreateXxxCredentials>
  • Implement XxxSqlBuilder extends WarehouseBaseSqlBuilder
  • Implement: streamQuery(), getCatalog(), getAllTables(), getFields() (inherited test(), runQuery(), executeAsyncQuery() work via base class — don't add no-op overrides)
  • Pass credentials directly to client library (like ClickHouse/Databricks), never via process.env
  • Use constructor overrides param for alternate construction (e.g. pre-aggregate) instead of AnyType casts on readonly fields
• Add case in src/warehouseClientFromCredentials.ts factory
• Add case in src/ssh/sshTunnel.ts (typically a no-op break for cloud warehouses)
• Export from src/index.ts
• Write tests in src/warehouseClients/XxxWarehouseClient.test.ts
• Run: pnpm -F @lightdash/warehouses typecheck && pnpm -F @lightdash/warehouses test

Layer 4: Frontend (packages/frontend/)

Warehouse Form

• Create src/components/ProjectConnection/WarehouseForms/XxxForm.tsx
  • Follow AthenaForm.tsx pattern with Mantine v8 components
  • Export XxxSchemaInput for DbtSettingsForm
• Add SVG logo to src/components/ProjectConnection/ProjectConnectFlow/Assets/

Registration

• Add defaults to WarehouseForms/defaultValues.ts (XxxDefaultValues + add to warehouseDefaultValues)
• Add validators to WarehouseForms/validators.ts
• Register in WarehouseSettingsForm.tsx (labels + forms maps)
• Register in DbtSettingsForm.tsx (schema input switch)
• Register in ProjectConnectFlow/utils.tsx (WarehouseTypeLabels array)
• Register in UserSettings/MyWarehouseConnectionsPanel/CreateCredentialsModal.tsx (defaultCredentials)
• Register in UserSettings/MyWarehouseConnectionsPanel/EditCredentialsModal.tsx (getCredentialsWithPlaceholders)
• Register in UserSettings/MyWarehouseConnectionsPanel/WarehouseFormInputs.tsx
• Run: pnpm -F frontend typecheck && pnpm -F frontend lint

Layer 5: CLI (packages/cli/)

• Create src/dbt/targets/xxx.ts
  • Define XxxTarget type matching dbt profile structure
  • Define xxxSchema: JSONSchemaType<XxxTarget> for ajv validation
  • Export convertXxxSchema() function
• Add case 'xxx': in src/dbt/profile.ts warehouseCredentialsFromDbtTarget()
• Add mock credentials in src/handlers/dbt/getWarehouseClient.ts getMockCredentials()

Layer 6: Docker (Dockerfile)

• Add the dbt adapter pip package (dbt-xxx) to dbt venvs 1.8+ in the Dockerfile
  • The adapter package name is typically dbt-xxx (e.g., dbt-duckdb, dbt-athena)
  • Use unversioned "dbt-xxx" if the adapter has its own version scheme, or "dbt-xxx~=X.Y.0" if it tracks dbt-core versions
  • Check PyPI for the package name and version compatibility
  • Only add to dbt versions that support the adapter (1.8+, not older 1.4-1.7)
  • Reference: dbt-athena was added from 1.9+, dbt-duckdb from 1.8+

Layer 7: Demo Project Compatibility (examples/full-jaffle-shop-demo/)

• Update dbt/data/seeds.yml column types for the new adapter (DuckDB doesn't support jsonb, TIME, etc.)
• Update SQL macros in dbt/macros/ that have warehouse-specific branches (e.g. quarter_end_date.sql, casts.sql)
• Add a profile target in profiles/profiles.yml for testing
• Test: dbt seed --target xxx && dbt run --target xxx

Post-Implementation

• pnpm generate-api (regenerates routes.ts and swagger.json)
• pnpm generate:chart-as-code-schema
• pnpm check:chart-as-code-schema
• Run full typecheck: pnpm -F common typecheck && pnpm -F backend typecheck && pnpm -F frontend typecheck

Reference Files

Canonical Examples

• Athena: PRs #19751/#19752 (first adapter added with this pattern)
• MotherDuck/DuckDB: Branch 02-23-feat_add_duckdb_warehouse_client_with_s3_support