Deduplicate Work Across Agents
Duplicate assignment wastes budget and produces two versions of one artifact that then conflict. Normalise the roster before launch and reconcile overlaps at fan-in.
Procedure
- Normalise every task key before comparing: lowercase, trim, collapse whitespace, strip trailing punctuation.
- Treat near-identical keys as duplicates: two slugs differing only by hyphen placement are the same job.
- Build the roster keyed by the normalised task id; a second assignment of an existing key is rejected, not appended.
- Detect residual overlap with a sorted check:
sort -k2 notes/tasks.tsv | uniq -d -f1lists repeated task ids. - Where two children must touch the same file, that is a partition problem, not a dedupe one — split by path.
- At fan-in, dedupe outputs on the child key:
jq -s 'unique_by(.slug)' out/all.jsonl. - When two files describe the same artifact with different content, do not keep both — pick by evidence and note the loser.
- Count assignments before and after dedupe and reconcile the drop in
notes/waves.log. - Never let dedupe silently drop a distinct task that only looked like a duplicate; confirm intent when unsure.
- Report the dedupe count to the requester so a later "why did this run fewer tasks than I listed" question has an answer.
Pitfalls
- Comparing raw slugs so
foo-barandfoobarboth run and produce competing artifacts. - Dropping a duplicate that was actually a re-run of a failed task, losing the intended retry.
- Deduping outputs by file content when two children legitimately produce identical bytes for different inputs.
- Merging duplicates at fan-in only, after paying for both children in full.
- Treating "same directory" as "same task" and collapsing two genuinely different files.
Verification
sort notes/tasks.tsv | uniq -d; jq -s 'length, (unique_by(.slug)|length)' out/all.jsonl
# passes when no duplicate keys print and merged length equals deduped length
Report to the user: the assignments before and after dedupe, and any artifact produced twice.