Dual Write Migration Pattern
Dual write keeps two data stores in sync by writing to both the old and new system on every mutation. This enables gradual migration with rollback safety.
When to Use This Skill
| Use this skill when... |
Use shadow-mode instead when... |
| Migrating between databases or schemas |
Validating read-path behavior under real traffic |
| Switching storage backends (SQL to NoSQL, etc.) |
Testing a new service without writing to it |
| Need both systems to stay authoritative during transition |
Only need to compare responses, not persist data |
| Planning zero-downtime data migrations |
Mirroring traffic to a staging environment |
| Reviewing code that writes to multiple data stores |
Evaluating performance of a replacement system |
Core Concepts
Migration Phases
| Phase |
Primary reads |
Primary writes |
Secondary writes |
Duration |
| 1. Prepare |
Old |
Old |
None |
Setup |
| 2. Dual write |
Old |
Old + New |
New (async or sync) |
Migration window |
| 3. Backfill |
Old |
Old + New |
New |
Until parity |
| 4. Shadow read |
Old + New (compare) |
Old + New |
New |
Validation |
| 5. Cutover |
New |
New |
Old (optional) |
Transition |
| 6. Cleanup |
New |
New |
None |
Final |
Write Strategies
| Strategy |
Consistency |
Latency impact |
Failure mode |
| Synchronous |
Strong |
Higher (2x write) |
Fail if either store fails |
| Async secondary |
Eventual |
Minimal |
Secondary may lag |
| Outbox pattern |
Eventual |
Minimal |
Requires message broker |
| Change data capture |
Eventual |
None (DB-level) |
Requires CDC infrastructure |
Implementation Architecture
Synchronous Dual Write
Client Request
│
▼
┌─────────────┐
│ Application │
│ Layer │
└──────┬──────┘
│ write(data)
▼
┌─────────────┐
│ Dual Write │
│ Adapter │
├──────┬──────┤
│ │ │
▼ │ ▼
Old DB │ New DB
│
Compare on
read (optional)
Key Components
| Component |
Responsibility |
| Write adapter |
Routes writes to both stores, handles failures |
| Read comparator |
Reads from both, logs discrepancies, returns primary |
| Backfill job |
Copies historical data from old to new store |
| Reconciliation |
Detects and resolves drift between stores |
| Feature flags |
Controls which phase is active per entity/tenant |
Implementation Patterns
Write Adapter Pattern
The write adapter wraps both stores behind a single interface:
- Accept the write request
- Write to the primary (old) store first
- Write to the secondary (new) store
- If secondary fails: log the failure, enqueue for retry, do not fail the request
- Return the primary store's result to the caller
Read Comparison Pattern
During the shadow read phase:
- Read from the primary (old) store — this is the authoritative response
- Read from the secondary (new) store asynchronously
- Compare results field by field
- Log discrepancies with context (entity ID, field, old value, new value)
- Return the primary store's result
- Track comparison metrics (match rate, common divergence fields)
Backfill Strategy
- Snapshot the old store at a known point in time
- Begin dual writes for all new mutations
- Copy historical records in batches (oldest first or by priority)
- Track backfill progress per entity type
- Reconcile records modified during backfill (dual write wins over backfill)
Failure Handling
| Failure scenario |
Response |
Recovery |
| Secondary write fails |
Log, continue, enqueue retry |
Async retry with backoff |
| Primary write fails |
Fail the request (do not write to secondary) |
Standard error handling |
| Both fail |
Fail the request |
Standard error handling |
| Secondary write timeout |
Log, continue |
Async verification and repair |
| Inconsistency detected |
Log with full context |
Manual or automated reconciliation |
Consistency Guarantees
- Primary store is always the source of truth until cutover
- Secondary store may lag during async dual write
- Reconciliation jobs detect and repair drift
- Cutover only happens when match rate reaches threshold (e.g., 99.9%)
Cutover Decision Criteria
| Metric |
Threshold |
How to measure |
| Read comparison match rate |
> 99.9% |
Shadow read comparison logs |
| Backfill completion |
100% |
Backfill progress tracker |
| Secondary write success rate |
> 99.95% |
Write adapter metrics |
| P99 latency impact |
< 20% increase |
Application metrics |
| Reconciliation gap |
0 unresolved |
Reconciliation job output |
Common Pitfalls
| Pitfall |
Mitigation |
| Ordering issues between stores |
Use idempotent writes, include version/timestamp |
| Transaction boundaries differ |
Design writes to be independently valid |
| Schema mismatch between stores |
Map fields explicitly, handle nullability differences |
| Backfill conflicts with live writes |
Live dual writes take precedence over backfill |
| Performance degradation |
Start with async secondary writes |
| Partial failures leave inconsistency |
Reconciliation job as safety net |
| Forgetting to dual-write in all code paths |
Centralize through write adapter, audit call sites |
Rollback Plan
| Phase |
Rollback action |
Data impact |
| Dual write |
Stop writing to new store |
No data loss |
| Shadow read |
Stop comparing reads |
No data loss |
| Cutover (reads) |
Switch reads back to old |
No data loss if still dual-writing |
| Cutover (writes) |
Reverse write order |
May need reconciliation |
| Cleanup |
Cannot rollback |
Old store decommissioned |
Monitoring Checklist
Agentic Optimizations
| Context |
Approach |
| Code review |
Check that all write paths go through the dual-write adapter |
| Architecture review |
Verify failure handling, rollback plan, and cutover criteria |
| Implementation |
Start with write adapter + async secondary, add comparison later |
| Testing |
Simulate secondary failures, verify primary is unaffected |
Quick Reference
| Term |
Definition |
| Primary store |
The authoritative data store (old system during migration) |
| Secondary store |
The new data store being migrated to |
| Backfill |
Copying historical data from primary to secondary |
| Reconciliation |
Detecting and repairing differences between stores |
| Cutover |
Switching the primary designation from old to new |
| Match rate |
Percentage of shadow reads that return identical results |
| Write adapter |
Abstraction layer that routes writes to both stores |
1---
2name: dual-write
3description: Dual-write pattern for safe data store transitions. Use when planning DB migrations, switching storage backends, or reviewing code writing to multiple systems simultaneously.
4user-invocable: false
5allowed-tools: Read, Grep, Glob, Edit, Write, Bash, TodoWrite
6created: 2026-02-18
7modified: 2026-05-09
8reviewed: 2026-02-18
9---
10
11# Dual Write Migration Pattern
12
13Dual write keeps two data stores in sync by writing to both the old and new system on every mutation. This enables gradual migration with rollback safety.
14
15## When to Use This Skill
16
17| Use this skill when... | Use shadow-mode instead when... |
18|------------------------|--------------------------------|
19| Migrating between databases or schemas | Validating read-path behavior under real traffic |
20| Switching storage backends (SQL to NoSQL, etc.) | Testing a new service without writing to it |
21| Need both systems to stay authoritative during transition | Only need to compare responses, not persist data |
22| Planning zero-downtime data migrations | Mirroring traffic to a staging environment |
23| Reviewing code that writes to multiple data stores | Evaluating performance of a replacement system |
24
25## Core Concepts
26
27### Migration Phases
28
29| Phase | Primary reads | Primary writes | Secondary writes | Duration |
30|-------|--------------|----------------|------------------|----------|
31| 1. Prepare | Old | Old | None | Setup |
32| 2. Dual write | Old | Old + New | New (async or sync) | Migration window |
33| 3. Backfill | Old | Old + New | New | Until parity |
34| 4. Shadow read | Old + New (compare) | Old + New | New | Validation |
35| 5. Cutover | New | New | Old (optional) | Transition |
36| 6. Cleanup | New | New | None | Final |
37
38### Write Strategies
39
40| Strategy | Consistency | Latency impact | Failure mode |
41|----------|------------|----------------|--------------|
42| Synchronous | Strong | Higher (2x write) | Fail if either store fails |
43| Async secondary | Eventual | Minimal | Secondary may lag |
44| Outbox pattern | Eventual | Minimal | Requires message broker |
45| Change data capture | Eventual | None (DB-level) | Requires CDC infrastructure |
46
47## Implementation Architecture
48
49### Synchronous Dual Write
50
51```
52Client Request
53 │
54 ▼
55┌─────────────┐
56│ Application │
57│ Layer │
58└──────┬──────┘
59 │ write(data)
60 ▼
61┌─────────────┐
62│ Dual Write │
63│ Adapter │
64├──────┬──────┤
65│ │ │
66▼ │ ▼
67Old DB │ New DB
68 │
69 Compare on
70 read (optional)
71```
72
73### Key Components
74
75| Component | Responsibility |
76|-----------|---------------|
77| Write adapter | Routes writes to both stores, handles failures |
78| Read comparator | Reads from both, logs discrepancies, returns primary |
79| Backfill job | Copies historical data from old to new store |
80| Reconciliation | Detects and resolves drift between stores |
81| Feature flags | Controls which phase is active per entity/tenant |
82
83## Implementation Patterns
84
85### Write Adapter Pattern
86
87The write adapter wraps both stores behind a single interface:
88
891. Accept the write request
902. Write to the primary (old) store first
913. Write to the secondary (new) store
924. If secondary fails: log the failure, enqueue for retry, do not fail the request
935. Return the primary store's result to the caller
94
95### Read Comparison Pattern
96
97During the shadow read phase:
98
991. Read from the primary (old) store — this is the authoritative response
1002. Read from the secondary (new) store asynchronously
1013. Compare results field by field
1024. Log discrepancies with context (entity ID, field, old value, new value)
1035. Return the primary store's result
1046. Track comparison metrics (match rate, common divergence fields)
105
106### Backfill Strategy
107
1081. Snapshot the old store at a known point in time
1092. Begin dual writes for all new mutations
1103. Copy historical records in batches (oldest first or by priority)
1114. Track backfill progress per entity type
1125. Reconcile records modified during backfill (dual write wins over backfill)
113
114## Failure Handling
115
116| Failure scenario | Response | Recovery |
117|-----------------|----------|----------|
118| Secondary write fails | Log, continue, enqueue retry | Async retry with backoff |
119| Primary write fails | Fail the request (do not write to secondary) | Standard error handling |
120| Both fail | Fail the request | Standard error handling |
121| Secondary write timeout | Log, continue | Async verification and repair |
122| Inconsistency detected | Log with full context | Manual or automated reconciliation |
123
124### Consistency Guarantees
125
126- Primary store is always the source of truth until cutover
127- Secondary store may lag during async dual write
128- Reconciliation jobs detect and repair drift
129- Cutover only happens when match rate reaches threshold (e.g., 99.9%)
130
131## Cutover Decision Criteria
132
133| Metric | Threshold | How to measure |
134|--------|-----------|----------------|
135| Read comparison match rate | > 99.9% | Shadow read comparison logs |
136| Backfill completion | 100% | Backfill progress tracker |
137| Secondary write success rate | > 99.95% | Write adapter metrics |
138| P99 latency impact | < 20% increase | Application metrics |
139| Reconciliation gap | 0 unresolved | Reconciliation job output |
140
141## Common Pitfalls
142
143| Pitfall | Mitigation |
144|---------|-----------|
145| Ordering issues between stores | Use idempotent writes, include version/timestamp |
146| Transaction boundaries differ | Design writes to be independently valid |
147| Schema mismatch between stores | Map fields explicitly, handle nullability differences |
148| Backfill conflicts with live writes | Live dual writes take precedence over backfill |
149| Performance degradation | Start with async secondary writes |
150| Partial failures leave inconsistency | Reconciliation job as safety net |
151| Forgetting to dual-write in all code paths | Centralize through write adapter, audit call sites |
152
153## Rollback Plan
154
155| Phase | Rollback action | Data impact |
156|-------|----------------|-------------|
157| Dual write | Stop writing to new store | No data loss |
158| Shadow read | Stop comparing reads | No data loss |
159| Cutover (reads) | Switch reads back to old | No data loss if still dual-writing |
160| Cutover (writes) | Reverse write order | May need reconciliation |
161| Cleanup | Cannot rollback | Old store decommissioned |
162
163## Monitoring Checklist
164
165- [ ] Write success rate per store
166- [ ] Write latency per store (P50, P95, P99)
167- [ ] Read comparison match rate
168- [ ] Backfill progress percentage
169- [ ] Reconciliation queue depth
170- [ ] Error rate by failure type
171- [ ] Feature flag state per tenant/entity
172
173## Agentic Optimizations
174
175| Context | Approach |
176|---------|----------|
177| Code review | Check that all write paths go through the dual-write adapter |
178| Architecture review | Verify failure handling, rollback plan, and cutover criteria |
179| Implementation | Start with write adapter + async secondary, add comparison later |
180| Testing | Simulate secondary failures, verify primary is unaffected |
181
182## Quick Reference
183
184| Term | Definition |
185|------|-----------|
186| Primary store | The authoritative data store (old system during migration) |
187| Secondary store | The new data store being migrated to |
188| Backfill | Copying historical data from primary to secondary |
189| Reconciliation | Detecting and repairing differences between stores |
190| Cutover | Switching the primary designation from old to new |
191| Match rate | Percentage of shadow reads that return identical results |
192| Write adapter | Abstraction layer that routes writes to both stores |
laurigates