grzmol Data Management
When to use
- Seeding a scratch org or a developer sandbox so the feature under test has records to work on.
- Backfilling a new field, migrating records between orgs, or extracting a sample of real data.
- Planning a schema change that moves, reshapes or drops data.
- A deploy is blocked because a field cannot be deleted while data still references it.
This skill covers moving records. Moving metadata is skill sf-deployment-strategies;
sfdx-project.json and package directories are skill sf-project-structure.
Pick the mechanism by volume
| Records | Mechanism | Command |
|---|---|---|
| A handful, hand-written, relationships by reference | sObject tree | sf data import tree |
| 1 to a few hundred, ad hoc | Single-record commands | sf data create record, sf data update record, sf data delete record |
| Under 2,000 | Bulkified synchronous REST (Composite) or SOAP | Outside the CLI, or sf data import bulk anyway |
| Over 2,000 | Bulk API 2.0 | sf data import bulk, sf data upsert bulk, sf data update bulk, sf data delete bulk |
| Reading over 10,000 | Bulk API 2.0 query | sf data export bulk |
| Reading under 10,000 | Synchronous query | sf data query |
The 2,000 threshold is the guide's own: "Any data operation that includes more than 2,000 records is a good candidate for Bulk API 2.0. Jobs with fewer than 2,000 records should involve bulkified synchronous calls in REST (for example, Composite) or SOAP."
The 10,000 threshold is the CLI's own advice: if sf data query would return more than 10,000
records, use sf data export bulk instead.
Core patterns
1. Seed a scratch org from a tree plan
Export a shaped sample once, commit it, replay it forever:
sf data export tree \
--query "SELECT Id, Name, Industry, (SELECT FirstName, LastName, Email FROM Contacts) FROM Account WHERE Industry != null LIMIT 20" \
--plan --prefix seed --output-dir data/seed \
--target-org acme-uat
--plan writes one JSON file per sObject plus a plan definition file. Import it into a fresh org:
sf data import tree --plan data/seed/seed-Account-Contact-plan.json --target-org my-scratch
Two facts that decide whether you use --plan or --files:
- Files listed in a plan may contain more than 200 records; the CLI batches them to respect
the API's 200-record limit. Files passed with
--filesmay not. - Order matters. A file whose records have lookups into another file must be listed after it. Accounts before Contacts.
Plan schema: items (one entry per sObject type), each with sobject (the type name; child file
references must have roots of this type) and files (an array of file paths, in load order).
2. Load volume with Bulk API 2.0
sf data import bulk \
--file data/accounts.csv \
--sobject Account \
--wait 10 \
--target-org acme-uat
Without --wait the command starts the job, prints the id and hands the terminal back. Resume with:
sf data import resume --use-most-recent --wait 10 --target-org acme-uat
sf data bulk results --job-id <id> --target-org acme-uat
--wait 0 is the default. In a script, either wait or record the job id - a fire-and-forget load
that nobody checks is a load that silently half-failed.
3. Upsert on an external id, never on a guessed match
sf data upsert bulk \
--file data/accounts.csv \
--sobject Account \
--external-id Legacy_Id__c \
--wait 10 \
--target-org acme-uat
--external-id is required and accepts either a custom external id field or Id. An upsert
inserts when the key does not match and updates when it does. That is the only safe way to make a
load re-runnable.
Design the external id field with External ID and Unique set. Without unique, a duplicate key makes the upsert ambiguous and the row fails.
4. Relate records without knowing Ids
In an sObject tree file, referenceId names a record and @ref points at it:
{
"records": [
{
"attributes": { "type": "Account", "referenceId": "acme" },
"Name": "Acme",
"Industry": "Manufacturing"
},
{
"attributes": { "type": "Contact", "referenceId": "ada" },
"LastName": "Lovelace",
"AccountId": "@acme"
}
]
}
In a CSV load, the equivalent is a relationship column keyed on the parent's external id:
LastName,Account.Legacy_Id__c
Lovelace,LEG-1001
That avoids a two-pass load entirely: no query for parent ids between the two files.
5. Extract volume out
sf data export bulk \
--query "SELECT Id, Name, Account.Name FROM Contact" \
--output-file out/contacts.csv \
--result-format csv \
--wait 10 \
--target-org acme-uat
--all-rows includes soft-deleted records. --query-file reads the SOQL from a file, which keeps a
long query out of shell quoting. Resume a timed-out export with sf data export resume.
6. Delete, and decide about the Recycle Bin
sf data delete bulk --sobject Account --file data/delete.csv --wait 10 --target-org acme-uat
The CSV must have one column, Id, and one record id per line.
--hard-delete marks the records immediately eligible for deletion instead of sending them to the
Recycle Bin. Hard delete is not reversible. The vibe-force Bash guard blocks destructive data
operations against a production alias unless VF_ALLOW_PROD=1 is set - that guard exists because
this flag exists.
7. Query in a script
sf data query \
--query "SELECT COUNT(Id) total FROM Account WHERE Legacy_Id__c != null" \
--target-org acme-uat --json | jq '.result.records[0].total'
--result-format takes human (default), csv or json; --json overrides it. --use-tooling-api
switches to Tooling objects. Post-deploy verification queries live in skill
sf-post-deploy-verification.
Limits that shape the plan
| Item | Bulk API 2.0 |
|---|---|
| Batch allocation | 15,000 batches per rolling 24 hours, shared with Bulk API 1.0. Only ingest jobs consume batches; query jobs do not |
| Records uploaded per 24 hours | 150,000,000 |
| Maximum file size per job | 150 MB of base64-encoded content. Upload data stays under 100 MB to leave room for the roughly 50% base64 increase |
| Maximum characters in a field | 131,072 |
| Maximum fields in a record | 5,000 |
| Maximum characters in a record | 400,000 |
| Job lifespan | Terminal-state jobs older than 7 days are deleted; non-terminal jobs are periodically cleaned up |
| Maximum time a job can stay open | 24 hours (ingest jobs only) |
| Results lifespan | Retrievable within 7 days of job completion unless the job is deleted |
| Retries | Handled automatically. "Retried more than 20 times" on ingest means use a smaller file; "more than 15 times" on query means add filter criteria |
| Query results timeout | 20 minutes |
Batches are created for you in Bulk API 2.0. In Bulk API 1.0 you create them yourself, at up to 10,000 records and 10 MB each - which is why 1.0 is the one that runs out of the shared 15,000-batch allocation first.
Full tables including Bulk API 1.0 and query-job specifics: references/bulk-api-limits.md.
Data-loss safety
No destructive metadata change without a backout path. The rules that matter:
| Change | Risk | Required handling |
|---|---|---|
| Deleting a field | Data in it is gone | Stop writing it in one release, delete it in a later one. Export the column first |
| Changing a field type | Silent truncation or conversion failure | Export, change, re-import, verify counts |
| Converting a master-detail relationship | Cannot be converted once data exists | Rebuild the relationship as a controlled migration |
| Changing an org-wide default to more restrictive | Records disappear from users' views | Verify with a query as an affected user before and after |
| Hard delete | Not reversible | --hard-delete never runs against production without VF_ALLOW_PROD=1 |
Always take a count before and after:
sf data query --query "SELECT COUNT(Id) c FROM Account" --target-org acme-uat --json | jq '.result.records[0].c'
A load that reports success and changes no counts did not do what you think.
Anti-patterns
Loading with Id as the upsert key from another org
Record ids are not portable between orgs. An upsert on Id against a target org where those ids do
not exist inserts duplicates. Use a real external id field.
A CSV with no external id at all
The load works once. The re-run creates a second copy of everything. Every repeatable load needs a unique, indexed external id.
Children before parents
sf data import tree --files Contact.json,Account.json # wrong order
The Contact lookups have nothing to resolve against. List Accounts first, or use a plan file with the items in dependency order.
sf data query for a migration extract
Over 10,000 records it is the wrong tool; the CLI says so itself. Use sf data export bulk.
Fire-and-forget bulk jobs in CI
sf data import bulk --file big.csv --sobject Account --target-org acme-uat
echo "loaded" # nothing was verified; --wait defaults to 0
Either --wait and check the exit code, or capture the job id and poll sf data bulk results.
Using Apex to load volume
A migration inside Apex spends the 10,000-DML-row per-transaction limit and the CPU budget for
nothing. Bulk API 2.0 exists so the load happens outside a transaction. See skill
sf-governor-limits.
Seeding with production data unfiltered
Real records carry real personal data. Extract a shaped sample with a LIMIT and a WHERE, scrub
the columns that matter, and commit that. Record-level access rules for the data once loaded are
in skill sf-security-model.
Verification
# The load itself
sf data import bulk --file data/accounts.csv --sobject Account --wait 10 --target-org <alias>
sf data bulk results --job-id <id> --target-org <alias>
# Counts before and after
sf data query --query "SELECT COUNT(Id) c FROM Account" --target-org <alias> --json | jq '.result.records[0].c'
# Rows that failed to match their parent
sf data query --query "SELECT COUNT(Id) c FROM Contact WHERE AccountId = null" --target-org <alias> --json
# Org storage and API capacity before a big job
sf org list limits --target-org <alias>
# The vibe-force post-deploy gate, which runs verification queries
node "${CLAUDE_PLUGIN_ROOT}/scripts/checks/vf-check.mjs" smoke --target-org <alias>
References
- references/bulk-api-limits.md - Bulk API 2.0 and 1.0 limits and allocations
- references/cli-data-commands.md - every
sf datacommand with its required flags - references/load-recipes.md - seeding, migration, backfill and extract recipes end to end
Sibling skills: sf-cli-operations (auth, config, JSON output conventions),
sf-scratch-orgs-sandboxes (org lifecycle and what seeding runs against),
sf-project-structure (sfdx-project.json, where seed data lives in the repo),
sf-soql-sosl-optimization (selective queries for extracts),
sf-governor-limits (why volume does not belong in Apex),
sf-deployment-strategies (destructive metadata changes and backout),
sf-security-model (access to the data once it is loaded),
sf-post-deploy-verification (verification queries after a release).
Sources: Salesforce Developer Limits and Allocations Quick Reference, "Bulk API and Bulk API 2.0
Limits and Allocations"; Bulk API 2.0 and Bulk API Developer Guide; Salesforce CLI command
reference for sf data. All Summer '26 / API version 67.0.