REST API Pagination Patterns

REST API Pagination Patterns

Activate when integrating with a paginated REST API — either consuming an external API from Apex or serving a paginated endpoint from Salesforce. Pagination mistakes cause three classic failures: infinite loops, missed records, and duplicate records due to concurrent writes between page fetches.

Before Starting

• Identify pagination style. Cursor / Next-link / Offset / Link-header / Token-based — all have different termination semantics.
• Plan termination. Every pagination loop needs a safety cap (max iterations) plus the documented termination signal.
• Respect rate limits. Back off on 429; implement exponential backoff.

Core Concepts

Salesforce QueryMore (outbound to external consumer)

/services/data/v60.0/query?q=SELECT+Id+FROM+Account
→ { "done": true, "nextRecordsUrl": "/services/data/v60.0/query/01g...", "records": [...] }

Loop: fetch nextRecordsUrl until done == true.

Cursor-based (opaque token)

Response includes a cursor string; pass back as ?cursor=<value> until the server returns an empty cursor or explicit end-of-stream flag.

Offset / limit

Classic ?offset=0&limit=100. Simple but vulnerable to drift: records inserted/deleted between pages cause missed or duplicate rows.

Link header (RFC 5988)

Link: <https://api/x?page=2>; rel="next", <https://api/x?page=10>; rel="last"

Parse headers; loop until no rel="next" present.

Rate limit headers

X-RateLimit-Remaining, Retry-After — consult before issuing the next page.

Common Patterns

Pattern: QueryMore loop in Apex

HttpResponse resp = http.send(req);
Map<String,Object> body = (Map<String,Object>) JSON.deserializeUntyped(resp.getBody());
List<Object> all = new List<Object>();
all.addAll((List<Object>) body.get('records'));
while (body.get('done') == false) {
    String nextUrl = (String) body.get('nextRecordsUrl');
    HttpRequest r2 = new HttpRequest();
    r2.setEndpoint(base + nextUrl);
    // auth ...
    resp = http.send(r2);
    body = (Map<String,Object>) JSON.deserializeUntyped(resp.getBody());
    all.addAll((List<Object>) body.get('records'));
}

Pattern: Cursor loop with safety cap

String cursor = null;
Integer safetyCap = 1000; // max pages
for (Integer i = 0; i < safetyCap; i++) {
    // fetch with cursor
    if (cursor == null || cursor == '') break;
}
if (i == safetyCap) throw new IntegrationException('Pagination safety cap hit');

Pattern: Rate-limit-aware pagination

Check X-RateLimit-Remaining; if < threshold, sleep per Retry-After before next page. In Apex: use Queueable chaining for long-running paginations (cannot Thread.sleep).

Decision Guidance

Recommended Workflow

1. Identify the source API's pagination style from documentation.
2. Implement the loop with an explicit termination signal AND a safety cap.
3. For offset/limit: either confirm dataset is stable or add resume tokens.
4. Parse rate-limit headers; back off on 429.
5. For long paginations in Apex, chain Queueables — each handles one page batch.
6. Persist intermediate results in case of mid-stream failure.
7. Write integration test that mocks 3+ pages including empty-tail.

Review Checklist

• Termination signal matches API spec
• Safety cap on page count (avoid infinite loops)
• Rate-limit headers honored
• Apex: long paginations chained via Queueable (callout limit: 100/transaction)
• Resume/checkpoint on mid-stream failure
• Test mocks multi-page and empty-tail
• No blind offset pagination on mutable data

Salesforce-Specific Gotchas

1. Apex has a 100-callout-per-transaction limit. Paginating over thousands of pages requires Queueable chaining.
2. QueryMore cursor expires after ~15 minutes. Long paginations need restart logic if they straddle the expiry.
3. Test.setMock supports only one response per request. Multi-page tests need a mock class tracking call count.

Output Artifacts

Related Skills

• integration/bulk-api-patterns — for large-volume data operations
• integration/rate-limit-handling — backoff strategies
• apex/apex-queueable-patterns — chaining for long-running callouts