File And Partner Feed Ingestion
Overview
Use this skill when ingestion depends on externally delivered files rather than APIs or CDC streams. It helps agents design landing, validation, late-file handling, checksum or manifest controls, replay safety, and publish boundaries for partner-managed feeds.
When to Use
- onboarding
SFTP, MFT, or shared-bucket file feeds
- ingesting recurring
CSV, JSON, XML, or columnar extracts from external partners
- validating manifests, checksums, control totals, or arrival windows
- handling late, duplicate, partial, or corrected file deliveries
- defining replay-safe landing and publish behavior for externally supplied data
Do not assume file ingestion is simple just because the payload arrives in a batch.
Workflow
Define the feed contract.
Clarify:
- file format and expected schema
- naming convention
- arrival schedule or SLA
- manifest, checksum, or control-total expectations
- ownership and contact path for the partner or source team
Design the landing boundary.
Decide:
- where raw files land
- what metadata is captured
- how duplicates are detected
- how partial or corrupt deliveries are quarantined
Validate before publish.
Include:
- schema and required-field checks
- checksum or manifest validation
- row-count or control-total validation
- late-file and missing-file detection
Make replay behavior explicit.
Decide how:
- corrected files are handled
- duplicate files are ignored or reconciled
- replay windows are bounded
- downstream publishes stay protected until validation passes
Define failure and escalation behavior.
Document:
- who gets alerted
- what blocks publish
- how the partner is contacted
- how recovery evidence is recorded
Common Rationalizations
| Rationalization |
Reality |
| "It is just a file drop." |
File-based feeds often fail through late arrival, partial delivery, duplicate upload, or silent schema drift. |
| "We can validate after loading." |
Weak landing controls let corrupt or incomplete partner data flow too far downstream. |
| "If the partner resends the file, we are fine." |
Resends without replay-safe handling can create duplicate or conflicting publishes. |
Red Flags
- file naming and arrival expectations are undocumented
- manifest, checksum, or control totals are ignored
- duplicate and corrected file behavior is undefined
- late or missing files do not create actionable alerts
- downstream publish opens before the landing validations complete
Verification
1---2name: file-and-partner-feed-ingestion3description: Guides agents through file-based and partner-feed ingestion workflows. Use when landing data from SFTP, managed file transfer, shared buckets, recurring flat files, manifests, or externally supplied feeds that need validation, replay safety, and publish discipline.4---56# File And Partner Feed Ingestion78## Overview910Use this skill when ingestion depends on externally delivered files rather than APIs or CDC streams. It helps agents design landing, validation, late-file handling, checksum or manifest controls, replay safety, and publish boundaries for partner-managed feeds.1112## When to Use1314- onboarding `SFTP`, MFT, or shared-bucket file feeds15- ingesting recurring `CSV`, `JSON`, `XML`, or columnar extracts from external partners16- validating manifests, checksums, control totals, or arrival windows17- handling late, duplicate, partial, or corrected file deliveries18- defining replay-safe landing and publish behavior for externally supplied data1920Do not assume file ingestion is simple just because the payload arrives in a batch.2122## Workflow23241. Define the feed contract.25 Clarify:26 - file format and expected schema27 - naming convention28 - arrival schedule or SLA29 - manifest, checksum, or control-total expectations30 - ownership and contact path for the partner or source team31322. Design the landing boundary.33 Decide:34 - where raw files land35 - what metadata is captured36 - how duplicates are detected37 - how partial or corrupt deliveries are quarantined38393. Validate before publish.40 Include:41 - schema and required-field checks42 - checksum or manifest validation43 - row-count or control-total validation44 - late-file and missing-file detection45464. Make replay behavior explicit.47 Decide how:48 - corrected files are handled49 - duplicate files are ignored or reconciled50 - replay windows are bounded51 - downstream publishes stay protected until validation passes52535. Define failure and escalation behavior.54 Document:55 - who gets alerted56 - what blocks publish57 - how the partner is contacted58 - how recovery evidence is recorded5960## Common Rationalizations6162| Rationalization | Reality |63| --- | --- |64| "It is just a file drop." | File-based feeds often fail through late arrival, partial delivery, duplicate upload, or silent schema drift. |65| "We can validate after loading." | Weak landing controls let corrupt or incomplete partner data flow too far downstream. |66| "If the partner resends the file, we are fine." | Resends without replay-safe handling can create duplicate or conflicting publishes. |6768## Red Flags6970- file naming and arrival expectations are undocumented71- manifest, checksum, or control totals are ignored72- duplicate and corrected file behavior is undefined73- late or missing files do not create actionable alerts74- downstream publish opens before the landing validations complete7576## Verification7778- [ ] The feed contract defines format, arrival, and ownership expectations79- [ ] Landing, quarantine, and duplicate-handling rules are explicit80- [ ] Validation covers schema, completeness, and control totals where relevant81- [ ] Replay and corrected-file behavior are documented before go-live82- [ ] Failure handling and partner escalation paths are operationally clear
vaquarkhan