Bulk API And Large Data Loads

Bulk API and Large Data Loads

This skill activates when a practitioner needs to move more than a few thousand records into or out of Salesforce programmatically, choose between Bulk API 2.0 and synchronous REST/SOAP, configure serial vs. parallel processing, or recover from failed bulk jobs. It does not cover Data Loader UI workflows or real-time single-record callouts.

Before Starting

Gather this context before working on anything in this domain:

• Record volume and object type. The threshold for Bulk API 2.0 is >2,000 records per operation. Below that, use bulkified REST (Composite) or SOAP. Confirm the object — some objects with complex sharing models or heavy triggers benefit from serial mode regardless of volume.
• Most common wrong assumption. Practitioners assume parallel mode is always safe and that a Bulk API 2.0 job either succeeds or fails atomically. Neither is true. Parallel mode causes lock contention on objects with complex triggers or sharing recalculation. And a job in JobComplete state may still contain failed and unprocessed records — the job itself succeeded but individual records did not.
• Key limits in play. Bulk API 2.0 automatically creates one internal batch per 10,000 records, up to a daily cap of 150,000,000 records per org. Each upload request must not exceed 150 MB (after base64 encoding). A batch that cannot process within 5 minutes is retried up to 20 times before the job moves to Failed. There is no SLA on Bulk API 2.0 processing time since it is fully asynchronous.

Core Concepts

Bulk API 2.0 vs. REST/SOAP

Any data operation involving more than 2,000 records is a good candidate for Bulk API 2.0. Operations under 2,000 records should use bulkified synchronous calls (REST Composite or SOAP). Bulk API 2.0 is the recommended choice over legacy Bulk API (v1) because it has a simpler workflow, automatic batching, a single results endpoint, and automatic PK chunking for query jobs.

Bulk API 2.0 supports two job types:

• Ingest jobs — insert, update, upsert, delete, hardDelete operations using CSV data.
• Query jobs — asynchronous SOQL queries that return large result sets, with automatic PK chunking.

Bulk API 2.0 uses standard OAuth 2.0 bearer token authentication, the same as all other Salesforce REST APIs.

Job Lifecycle (Ingest)

Every Bulk API 2.0 ingest job moves through these states:

1. Open — Job created, accepting data uploads.
2. UploadComplete — Caller signals upload is done; Salesforce begins processing. This PATCH is mandatory — omitting it means the job never starts.
3. InProgress — Salesforce auto-batches records (one batch per 10,000 records) and processes them.
4. JobComplete — All batches processed. Individual records may still have failed.
5. Failed — Salesforce could not process the job (e.g., repeated batch timeouts after 20 retries).
6. Aborted — Job canceled by the creator or a user with Manage Data Integrations permission.

Once a job reaches JobComplete or Failed, the caller must retrieve results from three endpoints:

• GET .../successfulResults/ — records that were processed successfully.
• GET .../failedResults/ — records that failed with error messages.
• GET .../unprocessedRecords/ — records that were never attempted (only present for Failed or Aborted jobs).

Serial vs. Parallel Mode

Bulk API 2.0 processes jobs in parallel mode by default, creating internal batches that run concurrently. This is faster but can cause record lock contention when:

• The object has complex Apex triggers that update shared parent records.
• Sharing recalculation is triggered by ownership changes.
• Records in different batches reference the same parent (causing parent-level locks).

Serial mode (legacy Bulk API v1 concept; in Bulk API 2.0 the equivalent is one upload chunk at a time with careful batching) processes batches one at a time, minimizing lock contention at the cost of slower throughput. For objects prone to locking, group child records by parent ID within the same batch and consider using serial mode in Bulk API (v1) if lock errors persist.

The official guidance: avoid serial mode unless parallel mode results in lock timeouts and you cannot reorganize batches to avoid the locks.

Failed Record Handling

Bulk API 2.0 does not roll back successfully processed records when other records in the same job fail. Each record is processed independently. After a job completes:

1. Retrieve failedResults — each row contains sf__Error describing the failure reason.
2. Retrieve unprocessedRecords — records that were never attempted (batch-level failure, not record-level).
3. Fix the root cause (validation rule violation, missing required field, duplicate, etc.).
4. Submit a new job with only the failed and unprocessed records.

Do not delete your local source CSV until all three result endpoints confirm complete coverage.

Common Patterns

Pattern 1: Standard Bulk Ingest Pipeline

When to use: Any insert, update, upsert, or delete of >2,000 records from an external system or migration pipeline.

How it works:

1. Authenticate with OAuth 2.0 to get a bearer token.
2. POST /services/data/vXX.X/jobs/ingest/ with {"object": "Account", "operation": "insert", "contentType": "CSV"} — note the id in the response.
3. PUT /services/data/vXX.X/jobs/ingest/{jobId}/batches with CSV body (up to 100 MB raw / 150 MB base64). Repeat for large datasets.
4. PATCH /services/data/vXX.X/jobs/ingest/{jobId} with {"state": "UploadComplete"} — this starts processing.
5. Poll GET /services/data/vXX.X/jobs/ingest/{jobId} until state is JobComplete or Failed.
6. Retrieve successfulResults, failedResults, and unprocessedRecords.
7. Resubmit a new job for any records in failed or unprocessed.

Why not synchronous REST: Synchronous REST calls are subject to governor limits per transaction and cannot handle millions of records in a single call. Bulk API 2.0 processes records asynchronously in the background without tying up a transaction thread.

Pattern 2: Bulk Query for Large Extracts

When to use: Extracting more than 1 million records, or any query that would time out via synchronous SOQL.

How it works:

1. POST /services/data/vXX.X/jobs/query with {"operation": "query", "query": "SELECT Id, Name FROM Account"}.
2. Poll the job status until JobComplete.
3. GET .../results with pagination using the Sforce-Locator response header to iterate through result pages.

Bulk API 2.0 query jobs automatically apply PK chunking to split large queries into manageable segments.

Decision Guidance

Recommended Workflow

Step-by-step instructions for an AI agent or practitioner activating this skill:

1. Gather context — confirm the org edition, relevant objects, and current configuration state
2. Review official sources — check the references in this skill's well-architected.md before making changes
3. Implement or advise — apply the patterns from Core Concepts and Common Patterns sections above
4. Validate — run the skill's checker script and verify against the Review Checklist below
5. Document — record any deviations from standard patterns and update the template if needed

Review Checklist

Run through these before marking work in this area complete:

• Volume confirmed — >2,000 records justifies Bulk API 2.0; <2,000 should use REST Composite
• Operation type confirmed — insert, update, upsert, delete, or hardDelete
• Parallel vs. serial mode decision documented and rationale recorded
• Upload chunk size confirmed — raw CSV ≤100 MB per PUT request (150 MB after base64)
• UploadComplete PATCH is included in the implementation (job never starts without it)
• Poll loop implemented with exponential backoff, terminates on JobComplete or Failed
• All three result endpoints are retrieved: successfulResults, failedResults, unprocessedRecords
• Failed and unprocessed records are resubmitted in a new job (not the same job)
• Source CSV retained locally until all records confirmed processed
• Validation rules and triggers verified — bulk jobs do not bypass them

Salesforce-Specific Gotchas

Non-obvious platform behaviors that cause real production problems:

1. Parallel mode lock contention on trigger-heavy objects — Parallel batches running concurrently can simultaneously try to update the same parent record or trigger shared resource updates, causing UNABLE_TO_LOCK_ROW errors. This surfaces in failedResults and silently leaves records unprocessed. Use serial mode (Bulk API v1) or group records by parent ID to minimize collisions.

2. Unprocessed records are not in failedResults — Practitioners check failedResults and assume any record not there was successful. Unprocessed records from a batch-level failure appear only in unprocessedRecords. If you do not call that endpoint, you silently lose records. Always call all three result endpoints after every job.

3. UploadComplete is mandatory — omitting it means nothing happens — A job created with state: Open will never start processing until the caller sends PATCH {"state": "UploadComplete"}. There is no timeout that auto-starts the job. Forgetting this PATCH is a common silent failure in new implementations.

4. Job state machine gotcha: JobComplete does not mean all records succeeded — JobComplete means Salesforce finished processing attempts. Records inside the job may still have failed with validation errors, duplicate errors, or permission errors. Treating JobComplete as "everything worked" without checking failedResults is a data integrity risk.

Output Artifacts

Related Skills

• data-import-and-management — for Data Loader UI steps, declarative import tools, and non-programmatic data operations
• apex/callouts-and-http-integrations — for real-time single-record or low-volume REST integrations that should not use Bulk API
• data-model-design-patterns — for object design choices that affect bulk load performance (e.g., avoiding lock-prone parent structures)