Amey-Thakur Agent map reduce
When work is too large for one pass, split it. The pattern is straightforward and fails in specific ways: slices that overlap or leave gaps, and a merge step that concatenates rather than reconciles.
Method
- Slice along a natural boundary. Per file, per document, per customer. Arbitrary chunking splits meaning across workers and produces findings neither can complete.
- Give each worker the same instructions and output shape. Divergent formats make the merge a parsing problem instead of a reasoning one (see agent-handoff-protocol).
- Include the shared context each slice needs. Enough to interpret its slice, without the whole corpus, which is the entire point of the split (see agent-context-isolation).
- Make the reduce step reason, not concatenate. Deduplicate, reconcile contradictions, and rank. Ten appended summaries are not a summary.
- Handle disagreement between slices explicitly. Two workers reaching opposite conclusions is a finding for the reducer to surface rather than average.
- Bound parallelism to what the system tolerates. Rate limits and cost ceilings are real constraints, and queueing is better than failing halfway through.
- Make partial failure survivable. One failed slice should not lose the run; record it, continue, and report the gap honestly rather than silently producing a partial answer.
Boundaries
Map reduce handles volume; it does not help when the task needs whole context at once, such as judging a document's overall coherence. Merged output is only as consistent as the instructions each worker received. Costs scale with slices, so the split should be as coarse as the work allows.