# Salesforce Data Pipeline Etl

> Export large Salesforce datasets to a lakehouse via Bulk API 2.0, CDC streams, or Salesforce Data Pipelines. NOT for ad-hoc exports — use data/etl-vs-api-data-patterns.

- Skill: `pranavnagrecha/salesforce-data-pipeline-etl` (Agent Skill, multi-file: 7 files)
- Install (CLI): `npx skillmds add pranavnagrecha/salesforce-data-pipeline-etl`
- Raw SKILL.md: https://api.skillmd.com/api/skills/pranavnagrecha/salesforce-data-pipeline-etl/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: PranavNagrecha (https://skillmd.com/u/pranavnagrecha)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/pranavnagrecha/salesforce-data-pipeline-etl

---


# Salesforce Data Pipeline / ETL

Production Salesforce → lake pipelines combine a one-time Bulk API 2.0 snapshot with an ongoing CDC or Platform Event stream. Naive incremental loads on LastModifiedDate lose updates during the query window; CDC guarantees ordered delta capture.

## Adoption Signals

Analytics workloads that need <1h freshness on Salesforce data in a warehouse.

- Bulk API 2.0 export when the source is full-table and the warehouse handles deduplication.
- Change Data Capture (CDC) export when downstream consumers need ordered events with primary-key delta semantics.

## Recommended Workflow

1. Initial snapshot: Bulk API 2.0 query job per object; store in staging.
2. Subscribe to CDC for each object via Pub/Sub API (or Change Data Capture Stream).
3. Apply deltas onto snapshot using event ChangeEventHeader.commitTimestamp + changeType (CREATE/UPDATE/DELETE/GAP_FILL).
4. Handle GAP_FILL events with a re-query of affected Ids (CDC may gap on heavy load).
5. Monitor replay lag; auto-refresh from Bulk snapshot if lag > threshold.

## Key Considerations

- CDC event retention is 3 days — downtime >3 days requires full re-snapshot.
- Field-level deletions of custom fields trigger schema migrations downstream.
- Big Object data cannot be streamed via CDC.
- Avoid SOQL polling at scale — you hit API limits.

## Worked Examples (see `references/examples.md`)

- *Snowflake mirror* — Finance analytics
- *Gap-fill handler* — CDC gap after outage

## Common Gotchas (see `references/gotchas.md`)

- **CDC retention exceeded** — Consumer offline >3 days; events lost.
- **Missing GAP_FILL** — Silently lose records.
- **SOQL polling fallback** — Eats API allocation.

## Top LLM Anti-Patterns (full list in `references/llm-anti-patterns.md`)

- LastModifiedDate polling as primary path
- Ignoring GAP_FILL events
- No replay-id checkpointing

## Official Sources Used

- Apex REST & Callouts — https://developer.salesforce.com/docs/atlas.en-us.apexcode.meta/apexcode/apex_callouts.htm
- Named Credentials — https://help.salesforce.com/s/articleView?id=sf.named_credentials_about.htm
- Connect REST API — https://developer.salesforce.com/docs/atlas.en-us.chatterapi.meta/chatterapi/
- Private Connect — https://help.salesforce.com/s/articleView?id=sf.private_connect_overview.htm
- Bulk API 2.0 — https://developer.salesforce.com/docs/atlas.en-us.api_asynch.meta/api_asynch/
- Pub/Sub API — https://developer.salesforce.com/docs/platform/pub-sub-api/guide/intro.html