Large Payload Completeness

Verify Large Payload Fetches Are COMPLETE — Anti Silent-Truncation

When fetching a large payload (database queries, API responses, log dumps, HAR files, transcript extracts, or any source exceeding ~50k characters), you must explicitly verify completeness before claiming you have read everything.

Why This Matters

Silent truncation is a particularly dangerous failure mode because coherent output masks incomplete data. An AI can produce a plausible, well-reasoned response based on the first chunk of a paginated fetch without realizing 60–70% of the source is missing. The response sounds complete and internally consistent, which fools both the AI and the human reader into believing the analysis is based on the full dataset.

Real-world example: An agent was asked to build a comprehensive report from a large database across multiple rounds of data collection. The agent fetched in chunks and implicitly assumed completeness. The response was coherent. Only later, when the human questioned a specific detail, did re-inspection reveal that a substantial middle section (containing critical data) and the final portion were silently skipped. The first chunk had been sufficient to construct a seemingly complete narrative, masking the gap.

The cost: rework, missed details in the deliverable, and erosion of trust.

Mandatory Gate — Before Operating on Fetched Content

Apply this gate every time you fetch a large payload:

1. Cross-check size returned vs. expected.

   • If the source is known to contain 337k characters and the fetch returned 90k, the gap of 247k must be explained. An unexplained gap means the fetch is incomplete.
   • Do not assume "the important bits came through." Do not assume "rest is probably duplicate or low-value." Declare the gap explicitly.

2. Follow all pagination cursors to completion.

   • For paginated APIs: follow next_cursor, next_page, has_more: false, or equivalent end-of-list signal provided by the source.
   • Confirm explicitly that the source has signaled "end of data" — do not stop because you got enough results to construct an answer.
   • Common sources: Notion databases and pages, GitHub API, Stripe API, any REST API with page or cursor parameters, HAR file entries array, JSON Lines log files.

3. Declare coverage explicitly before claiming completeness.

   • Example: "I fetched characters 0–90k of an expected 337k total (27% coverage). The remaining 73% is unread."
   • Do not just say "I read the source" without quantifying what fraction you actually covered.
   • If the expected size is unknown, estimate or declare the estimate ("this file appears to be ~500 lines; I read 156").

4. Cross-check with a different tool or metric when viable.

   • If one fetch method returns N items, verify with a count or length check from the same source (e.g., log.entries.length vs. payload size estimate).
   • If two tools should return the same data, compare their sizes.

5. Be suspicious of coherent output.

   • An answer that "makes sense" and "reads well" is not evidence of complete data. Coherence is cheap; verification is costly.
   • If you constructed a summary from the first page of results and didn't check pagination, the summary may sound plausible while being systematically biased toward early results.

Symptoms and Detection

Watch for this pattern during your own work:

• You have fetched data and are about to declare findings or close the task.
• You feel confident in your answer.
• But you have not explicitly verified that you read the entire source.

This is the moment to apply the gate. Stop. Verify coverage before proceeding.

When to Apply This Skill

• Fetching from any paginated API (REST, GraphQL, custom) — verify all pages/cursors are consumed.
• Reading large files (>~50k characters): log dumps, HAR files, JSONL transcripts, CSV exports, database dumps.
• Querying databases where result count may exceed a page limit.
• Building any deliverable (report, analysis, audit, summary, recommendation) from a large source — the cost of incompleteness is high, and the buyer will likely notice missing data.
• Extracting data from multi-page documents or sources that logically span multiple requests.

Checklist

Before you claim "I have read everything from source X":

• Expected size of source is known or estimated (e.g., "API returns up to 1000 results per page; I expect ~20k items total")
• Actual size fetched is known (e.g., "I fetched 45 items across 3 pages" or "the response body is 234k chars")
• Size reconciliation is explicit (e.g., "I have 45 items from 3 pages; the API returned has_more: false, so 45 is the complete set")
• Pagination cursors (if applicable) were exhausted (next_cursor is null, has_more is false, or no 4th page exists)
• Coverage as a percentage is stated when feasible (e.g., "100% of API results" or "characters 0–234k of ~450k expected")

Applies to: all AI agents, all projects. This is a specialization of the broader "evidence-based operation protocol" that focuses on a specific failure mode: silent truncation from incomplete pagination or chunked fetches. It does not replace that broader protocol; it reinforces one mode of failure that is easy to miss when the partial output is coherent.