Shadow Mode Migration Pattern
Shadow mode mirrors production traffic to a new system without affecting users. The shadow system's responses are discarded — only the production response reaches the user — but both responses are logged and compared to validate correctness.
When to Use This Skill
| Use this skill when... |
Use dual-write instead when... |
| Validating read behavior of a replacement service |
Both systems need to persist writes |
| Testing performance under real production load |
You need the new store to be authoritative |
| Comparing response correctness before cutover |
Migrating data stores that must stay in sync |
| Evaluating a new service version safely |
The new system needs to receive and store mutations |
| Load testing a new deployment with real traffic |
You need strong consistency between systems |
Core Concepts
Traffic Flow
Client Request
│
▼
┌─────────────┐
│ Router / │
│ Proxy │
├──────┬──────┤
│ │ │
▼ │ ▼
Prod │ Shadow
System │ System
│ │ │
▼ │ ▼
Prod │ Shadow
Response│ Response
│ │ │
▼ │ (discard)
Client │ │
│ ▼
│ Compare &
│ Log
▼
Shadow Modes
| Mode |
Description |
Use case |
| Full mirror |
100% of traffic duplicated |
Final validation before cutover |
| Sampled mirror |
Percentage of traffic (e.g., 10%) |
Early validation, capacity-constrained shadow |
| Selective mirror |
Specific request types or endpoints |
Targeted validation of changed behavior |
| Replay mirror |
Recorded traffic replayed offline |
Testing without live shadow infrastructure |
Implementation Architecture
Key Components
| Component |
Responsibility |
| Traffic splitter |
Duplicates requests to shadow system |
| Shadow router |
Forwards mirrored requests, manages timeouts |
| Response comparator |
Compares prod vs shadow responses |
| Discrepancy logger |
Records differences with full context |
| Metrics collector |
Tracks match rates, latency, error rates |
| Kill switch |
Disables shadow traffic instantly if issues arise |
Deployment Topology
| Topology |
How it works |
Trade-offs |
| Proxy-based |
Load balancer or API gateway mirrors requests |
Simple setup, adds proxy hop |
| Application-level |
Application code sends async copy of request |
Fine-grained control, code coupling |
| Infrastructure-level |
Service mesh (Istio, Linkerd) mirrors traffic |
No code changes, requires mesh |
| Log replay |
Capture request logs, replay against shadow |
No live infrastructure needed, not real-time |
Implementation Patterns
Proxy-Based Mirroring
Configure the load balancer or API gateway to:
- Forward the original request to the production backend
- Clone the request and send it to the shadow backend
- Return only the production response to the client
- Shadow response is logged but never returned
- Shadow request timeout is independent of production
Application-Level Mirroring
- Intercept the incoming request at the application layer
- Process the request normally through the production path
- Asynchronously send a copy of the request to the shadow service
- Do not block the production response on the shadow response
- Compare responses in a background worker
Response Comparison Strategy
Compare responses field by field with configurable rules:
| Field type |
Comparison approach |
| IDs, timestamps |
Ignore (expected to differ) |
| Computed values |
Compare within tolerance (e.g., floating point) |
| Collections |
Compare as sets (ignore ordering unless significant) |
| Status codes |
Exact match required |
| Error responses |
Categorize and compare error types |
| Headers |
Compare relevant headers only (Content-Type, Cache-Control) |
Handling Stateful Requests
Shadow mode works best with read-only requests. For stateful (write) requests:
| Approach |
Description |
| Skip writes |
Only mirror read requests to shadow |
| Isolated state |
Shadow has its own database seeded from production |
| Dry-run writes |
Shadow validates the write but does not persist |
| Record-only |
Log what shadow would have written, compare intent |
Gradual Rollout
| Phase |
Traffic % |
Duration |
Goal |
| 1. Smoke test |
1% |
Hours |
Verify shadow receives and processes requests |
| 2. Canary |
5-10% |
Days |
Identify obvious discrepancies |
| 3. Validation |
25-50% |
Days-weeks |
Build confidence in match rate |
| 4. Full mirror |
100% |
Days-weeks |
Final validation before cutover |
Validation Metrics
| Metric |
Target |
Description |
| Response match rate |
> 99.9% |
Percentage of identical responses |
| Shadow latency (P50) |
Within 2x of prod |
Shadow performance baseline |
| Shadow latency (P99) |
Monitored |
Tail latency under real load |
| Shadow error rate |
< prod error rate |
Shadow should not produce more errors |
| Shadow availability |
Monitored |
Shadow uptime (not a blocker) |
| Discrepancy categories |
Trending to zero |
Known differences resolved over time |
Common Pitfalls
| Pitfall |
Mitigation |
| Shadow affects production performance |
Async mirroring, independent timeouts, kill switch |
| Shadow writes to shared resources |
Isolate shadow databases, queues, and external services |
| Non-deterministic responses cause false mismatches |
Configure comparison rules to ignore timestamps, IDs, nonces |
| Shadow receives stale data |
Seed shadow database from recent production snapshot |
| Traffic amplification overwhelms shadow |
Use sampled mirroring, auto-scaling, or circuit breakers |
| Request ordering differs between prod and shadow |
Compare request-by-request, not sequence-dependent |
| Authentication tokens expire for shadow |
Mint shadow-specific tokens or bypass auth in shadow |
Integration with Dual Write
Shadow mode and dual write are complementary migration techniques:
| Migration phase |
Technique |
Purpose |
| Early validation |
Shadow mode (reads) |
Verify the new system returns correct responses |
| Data sync |
Dual write |
Keep both stores authoritative during transition |
| Pre-cutover |
Both simultaneously |
Shadow validates reads, dual write maintains data |
| Cutover |
Dual write reversal |
New system becomes primary, old becomes secondary |
| Post-cutover |
Shadow mode (reversed) |
Mirror to old system to verify nothing broke |
Strangler Fig Context
Both patterns are tactics within the broader Strangler Fig migration strategy:
- Identify a component to migrate
- Shadow traffic to validate the replacement
- Dual write to synchronize data stores
- Cut over reads, then writes
- Decommission the old component
- Repeat for the next component
Kill Switch Requirements
Shadow mode must have an immediate disable mechanism:
- Feature flag or configuration toggle (no deployment required)
- Disables within seconds, not minutes
- Monitored — alerts if shadow causes production impact
- Tested before enabling shadow traffic
Monitoring Checklist
Agentic Optimizations
| Context |
Approach |
| Architecture review |
Verify shadow isolation (no shared writes), kill switch exists |
| Code review |
Check async mirroring does not block production path |
| Implementation |
Start with proxy-based mirroring at 1%, increase gradually |
| Testing |
Verify kill switch works, confirm production is unaffected when shadow fails |
Quick Reference
| Term |
Definition |
| Shadow system |
The new system receiving mirrored traffic |
| Production system |
The live system serving real users |
| Traffic splitter |
Component that duplicates requests |
| Match rate |
Percentage of shadow responses matching production |
| Kill switch |
Mechanism to instantly disable shadow traffic |
| Dark launching |
Synonym for shadow mode — feature is live but invisible to users |
| Canary traffic |
Small percentage of mirrored requests for initial validation |
| Strangler fig |
Broader migration strategy of incrementally replacing components |
1---2name: shadow-mode3description: Shadow mode / dark-launch for validating new systems under production load. Use when testing replacement services, comparing behavior, or planning traffic mirroring for migrations.4---5
6# Shadow Mode Migration Pattern
7
8Shadow mode mirrors production traffic to a new system without affecting users. The shadow system's responses are discarded — only the production response reaches the user — but both responses are logged and compared to validate correctness.
9
10## When to Use This Skill
11
12| Use this skill when... | Use dual-write instead when... |
13|------------------------|-------------------------------|
14| Validating read behavior of a replacement service | Both systems need to persist writes |
15| Testing performance under real production load | You need the new store to be authoritative |
16| Comparing response correctness before cutover | Migrating data stores that must stay in sync |
17| Evaluating a new service version safely | The new system needs to receive and store mutations |
18| Load testing a new deployment with real traffic | You need strong consistency between systems |
19
20## Core Concepts
21
22### Traffic Flow
23
24```
25Client Request
26 │
27 ▼
28┌─────────────┐
29│ Router / │
30│ Proxy │
31├──────┬──────┤
32│ │ │
33▼ │ ▼
34Prod │ Shadow
35System │ System
36│ │ │
37▼ │ ▼
38Prod │ Shadow
39Response│ Response
40│ │ │
41▼ │ (discard)
42Client │ │
43 │ ▼
44 │ Compare &
45 │ Log
46 ▼
47```
48
49### Shadow Modes
50
51| Mode | Description | Use case |
52|------|------------|----------|
53| Full mirror | 100% of traffic duplicated | Final validation before cutover |
54| Sampled mirror | Percentage of traffic (e.g., 10%) | Early validation, capacity-constrained shadow |
55| Selective mirror | Specific request types or endpoints | Targeted validation of changed behavior |
56| Replay mirror | Recorded traffic replayed offline | Testing without live shadow infrastructure |
57
58## Implementation Architecture
59
60### Key Components
61
62| Component | Responsibility |
63|-----------|---------------|
64| Traffic splitter | Duplicates requests to shadow system |
65| Shadow router | Forwards mirrored requests, manages timeouts |
66| Response comparator | Compares prod vs shadow responses |
67| Discrepancy logger | Records differences with full context |
68| Metrics collector | Tracks match rates, latency, error rates |
69| Kill switch | Disables shadow traffic instantly if issues arise |
70
71### Deployment Topology
72
73| Topology | How it works | Trade-offs |
74|----------|-------------|------------|
75| Proxy-based | Load balancer or API gateway mirrors requests | Simple setup, adds proxy hop |
76| Application-level | Application code sends async copy of request | Fine-grained control, code coupling |
77| Infrastructure-level | Service mesh (Istio, Linkerd) mirrors traffic | No code changes, requires mesh |
78| Log replay | Capture request logs, replay against shadow | No live infrastructure needed, not real-time |
79
80## Implementation Patterns
81
82### Proxy-Based Mirroring
83
84Configure the load balancer or API gateway to:
85
861. Forward the original request to the production backend
872. Clone the request and send it to the shadow backend
883. Return only the production response to the client
894. Shadow response is logged but never returned
905. Shadow request timeout is independent of production
91
92### Application-Level Mirroring
93
941. Intercept the incoming request at the application layer
952. Process the request normally through the production path
963. Asynchronously send a copy of the request to the shadow service
974. Do not block the production response on the shadow response
985. Compare responses in a background worker
99
100### Response Comparison Strategy
101
102Compare responses field by field with configurable rules:
103
104| Field type | Comparison approach |
105|-----------|-------------------|
106| IDs, timestamps | Ignore (expected to differ) |
107| Computed values | Compare within tolerance (e.g., floating point) |
108| Collections | Compare as sets (ignore ordering unless significant) |
109| Status codes | Exact match required |
110| Error responses | Categorize and compare error types |
111| Headers | Compare relevant headers only (Content-Type, Cache-Control) |
112
113### Handling Stateful Requests
114
115Shadow mode works best with read-only requests. For stateful (write) requests:
116
117| Approach | Description |
118|----------|------------|
119| Skip writes | Only mirror read requests to shadow |
120| Isolated state | Shadow has its own database seeded from production |
121| Dry-run writes | Shadow validates the write but does not persist |
122| Record-only | Log what shadow would have written, compare intent |
123
124## Gradual Rollout
125
126| Phase | Traffic % | Duration | Goal |
127|-------|-----------|----------|------|
128| 1. Smoke test | 1% | Hours | Verify shadow receives and processes requests |
129| 2. Canary | 5-10% | Days | Identify obvious discrepancies |
130| 3. Validation | 25-50% | Days-weeks | Build confidence in match rate |
131| 4. Full mirror | 100% | Days-weeks | Final validation before cutover |
132
133## Validation Metrics
134
135| Metric | Target | Description |
136|--------|--------|-------------|
137| Response match rate | > 99.9% | Percentage of identical responses |
138| Shadow latency (P50) | Within 2x of prod | Shadow performance baseline |
139| Shadow latency (P99) | Monitored | Tail latency under real load |
140| Shadow error rate | < prod error rate | Shadow should not produce more errors |
141| Shadow availability | Monitored | Shadow uptime (not a blocker) |
142| Discrepancy categories | Trending to zero | Known differences resolved over time |
143
144## Common Pitfalls
145
146| Pitfall | Mitigation |
147|---------|-----------|
148| Shadow affects production performance | Async mirroring, independent timeouts, kill switch |
149| Shadow writes to shared resources | Isolate shadow databases, queues, and external services |
150| Non-deterministic responses cause false mismatches | Configure comparison rules to ignore timestamps, IDs, nonces |
151| Shadow receives stale data | Seed shadow database from recent production snapshot |
152| Traffic amplification overwhelms shadow | Use sampled mirroring, auto-scaling, or circuit breakers |
153| Request ordering differs between prod and shadow | Compare request-by-request, not sequence-dependent |
154| Authentication tokens expire for shadow | Mint shadow-specific tokens or bypass auth in shadow |
155
156## Integration with Dual Write
157
158Shadow mode and dual write are complementary migration techniques:
159
160| Migration phase | Technique | Purpose |
161|----------------|-----------|---------|
162| Early validation | Shadow mode (reads) | Verify the new system returns correct responses |
163| Data sync | Dual write | Keep both stores authoritative during transition |
164| Pre-cutover | Both simultaneously | Shadow validates reads, dual write maintains data |
165| Cutover | Dual write reversal | New system becomes primary, old becomes secondary |
166| Post-cutover | Shadow mode (reversed) | Mirror to old system to verify nothing broke |
167
168### Strangler Fig Context
169
170Both patterns are tactics within the broader Strangler Fig migration strategy:
171
1721. **Identify** a component to migrate
1732. **Shadow** traffic to validate the replacement
1743. **Dual write** to synchronize data stores
1754. **Cut over** reads, then writes
1765. **Decommission** the old component
1776. Repeat for the next component
178
179## Kill Switch Requirements
180
181Shadow mode must have an immediate disable mechanism:
182
183- Feature flag or configuration toggle (no deployment required)
184- Disables within seconds, not minutes
185- Monitored — alerts if shadow causes production impact
186- Tested before enabling shadow traffic
187
188## Monitoring Checklist
189
190- [ ] Production latency impact (should be zero or negligible)
191- [ ] Shadow request success rate
192- [ ] Shadow response latency distribution
193- [ ] Response match rate by endpoint
194- [ ] Discrepancy log volume and categories
195- [ ] Shadow system resource utilization
196- [ ] Kill switch status and responsiveness
197
198## Agentic Optimizations
199
200| Context | Approach |
201|---------|----------|
202| Architecture review | Verify shadow isolation (no shared writes), kill switch exists |
203| Code review | Check async mirroring does not block production path |
204| Implementation | Start with proxy-based mirroring at 1%, increase gradually |
205| Testing | Verify kill switch works, confirm production is unaffected when shadow fails |
206
207## Quick Reference
208
209| Term | Definition |
210|------|-----------|
211| Shadow system | The new system receiving mirrored traffic |
212| Production system | The live system serving real users |
213| Traffic splitter | Component that duplicates requests |
214| Match rate | Percentage of shadow responses matching production |
215| Kill switch | Mechanism to instantly disable shadow traffic |
216| Dark launching | Synonym for shadow mode — feature is live but invisible to users |
217| Canary traffic | Small percentage of mirrored requests for initial validation |
218| Strangler fig | Broader migration strategy of incrementally replacing components |
laurigates