# Large Payload Completeness > Verify large payload fetches are COMPLETE — anti silent-truncation. Cross-check size vs expected, follow all pagination cursors. Use when fetching from paginated APIs, large databases, HAR files, log dumps, transcripts, or any payload >50k chars. - Skill: `avizmarlon/large-payload-completeness` (Agent Skill) - Install (CLI): `npx skillmds@latest add avizmarlon/large-payload-completeness` - Raw SKILL.md: https://api.skillmd.com/api/skills/avizmarlon/large-payload-completeness/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - Author: avizmarlon (https://skillmd.com/u/avizmarlon) - Updated: 2026-09-22 - Page: https://skillmd.com/skills/avizmarlon/large-payload-completeness --- ## Verify Large Payload Fetches Are COMPLETE — Anti Silent-Truncation When fetching a large payload (database queries, API responses, log dumps, HAR files, transcript extracts, or any source exceeding ~50k characters), you must **explicitly verify completeness before claiming you have read everything**. ### Why This Matters Silent truncation is a particularly dangerous failure mode because **coherent output masks incomplete data**. An AI can produce a plausible, well-reasoned response based on the first chunk of a paginated fetch without realizing 60–70% of the source is missing. The response sounds complete and internally consistent, which fools both the AI and the human reader into believing the analysis is based on the full dataset. **Real-world example:** An agent was asked to build a comprehensive report from a large database across multiple rounds of data collection. The agent fetched in chunks and implicitly assumed completeness. The response was coherent. Only later, when the human questioned a specific detail, did re-inspection reveal that a substantial middle section (containing critical data) and the final portion were silently skipped. The first chunk had been sufficient to construct a seemingly complete narrative, masking the gap. The cost: rework, missed details in the deliverable, and erosion of trust. ### Mandatory Gate — Before Operating on Fetched Content Apply this gate every time you fetch a large payload: 1. **Cross-check size returned vs. expected.** - If the source is known to contain 337k characters and the fetch returned 90k, the gap of 247k **must be explained**. An unexplained gap means the fetch is incomplete. - Do not assume "the important bits came through." Do not assume "rest is probably duplicate or low-value." Declare the gap explicitly. 2. **Follow all pagination cursors to completion.** - For paginated APIs: follow `next_cursor`, `next_page`, `has_more: false`, or equivalent end-of-list signal provided by the source. - Confirm explicitly that the source has signaled "end of data" — do not stop because you got enough results to construct an answer. - Common sources: Notion databases and pages, GitHub API, Stripe API, any REST API with `page` or `cursor` parameters, HAR file `entries` array, JSON Lines log files. 3. **Declare coverage explicitly before claiming completeness.** - Example: *"I fetched characters 0–90k of an expected 337k total (27% coverage). The remaining 73% is unread."* - Do not just say "I read the source" without quantifying what fraction you actually covered. - If the expected size is unknown, estimate or declare the estimate ("this file appears to be ~500 lines; I read 156"). 4. **Cross-check with a different tool or metric when viable.** - If one fetch method returns N items, verify with a count or length check from the same source (e.g., `log.entries.length` vs. payload size estimate). - If two tools should return the same data, compare their sizes. 5. **Be suspicious of coherent output.** - An answer that "makes sense" and "reads well" is not evidence of complete data. Coherence is cheap; verification is costly. - If you constructed a summary from the first page of results and didn't check pagination, the summary may sound plausible while being systematically biased toward early results. ### Symptoms and Detection **Watch for this pattern during your own work:** - You have fetched data and are about to declare findings or close the task. - You feel confident in your answer. - But you have not explicitly verified that you read the entire source. **This is the moment to apply the gate.** Stop. Verify coverage before proceeding. ### When to Apply This Skill - Fetching from any **paginated API** (REST, GraphQL, custom) — verify all pages/cursors are consumed. - Reading **large files** (>~50k characters): log dumps, HAR files, JSONL transcripts, CSV exports, database dumps. - Querying **databases** where result count may exceed a page limit. - Building any **deliverable** (report, analysis, audit, summary, recommendation) from a large source — the cost of incompleteness is high, and the buyer will likely notice missing data. - Extracting data from **multi-page documents** or sources that logically span multiple requests. ### Checklist Before you claim "I have read everything from source X": - [ ] **Expected size of source is known or estimated** (e.g., "API returns up to 1000 results per page; I expect ~20k items total") - [ ] **Actual size fetched is known** (e.g., "I fetched 45 items across 3 pages" or "the response body is 234k chars") - [ ] **Size reconciliation is explicit** (e.g., "I have 45 items from 3 pages; the API returned `has_more: false`, so 45 is the complete set") - [ ] **Pagination cursors (if applicable) were exhausted** (`next_cursor` is null, `has_more` is false, or no 4th page exists) - [ ] **Coverage as a percentage is stated** when feasible (e.g., "100% of API results" or "characters 0–234k of ~450k expected") --- **Applies to:** all AI agents, all projects. This is a specialization of the broader "evidence-based operation protocol" that focuses on a specific failure mode: silent truncation from incomplete pagination or chunked fetches. It does not replace that broader protocol; it reinforces one mode of failure that is easy to miss when the partial output is coherent.