Data Resiliency Testing And Failure Injection
Overview
Use this skill when the goal is to prove that a data system recovers safely under failure, not only when everything goes right. It helps agents design controlled drills for retries, restarts, dependency outages, state recovery, duplicate prevention, backlog catch-up, and publish protection.
When to Use
- hardening a production pipeline before broad rollout
- testing failover, restart, replay, or checkpoint recovery behavior
- validating that retries do not duplicate or corrupt data
- proving recovery objectives for orchestrators, jobs, streams, or warehouse publishes
- converting a past incident into a repeatable resilience drill
Do not treat resilience testing as random breakage. The point is to validate recovery behavior with explicit safety limits and evidence.
Workflow
Define the failure modes that matter.
Prioritize:
- source outage or delayed upstream delivery
- worker or task restart
- orchestrator retry and timeout behavior
- duplicate event or duplicate file delivery
- checkpoint or incremental-state recovery
- credential, secret, or network dependency failure
- partial publish or downstream unavailability
Define the resilience objectives.
Include:
- acceptable data loss behavior
- recovery time objective
- replay or backlog catch-up expectation
- duplicate-prevention requirement
- publish block or quarantine behavior
- alert and escalation expectation
Choose the safest drill environment.
Prefer:
- staging or isolated non-production
- canary datasets or partitions
- synthetic or masked test data
- bounded windows and rollback-ready test scope
Inject one failure mode at a time.
Use controlled exercises such as:
- killing a task or worker
- pausing an upstream dependency
- delaying input arrival
- replaying a duplicate input
- forcing an expired secret or denied permission in a safe environment
- simulating partial output and validating publish closure
Validate the recovery path.
Check:
- whether the system resumes or fails safely
- whether alerts fire with useful context
- whether duplicates are prevented
- whether backlog catch-up stays bounded
- whether publish remains blocked until validation passes
Record guardrails and automate the highest-value drills.
The best resilience test is one the team can rerun after changes, not a one-time exercise that gets forgotten.
Load companion skills by failure mode.
- replay or backfill drills:
safe-backfill-and-replay-orchestration
- Kafka lag, DLQ, or schema drift:
kafka-resilience-and-schema-evolution
- serverless Spark checkpoint recovery:
spark-serverless-reliability-and-state-management
- live diagnosis before drills:
mcp-data-observability-integration
- drill patterns:
references/data-resiliency-testing-patterns.md
Common Rationalizations
| Rationalization |
Reality |
| "If the job retries, we are resilient enough." |
Retry alone does not prove replay safety, duplicate prevention, or publish protection. |
| "We can test recovery during a real incident." |
Real incidents are the worst time to discover the recovery path is unclear or unsafe. |
| "Failure injection is too risky for data systems." |
Uncontrolled failure is riskier than bounded, reviewable drills in safe environments. |
| "The scheduler health page already proves resilience." |
Scheduler status does not prove data correctness, backlog catch-up, or downstream safety. |
Red Flags
- no list of prioritized failure modes exists
- retries are enabled without idempotency proof
- resilience drills have no rollback or blast-radius limits
- checkpoint or incremental-state recovery has never been tested
- alerts fire but recovery ownership is unclear
- a past incident has no corresponding regression drill
Verification
1---2name: data-resiliency-testing-and-failure-injection3description: Guides agents through resiliency testing for data platforms. Use when designing or running failure drills, recovery validation, failover tests, replay-safety checks, dependency outage exercises, or fault injection for pipelines and publishes.4---56# Data Resiliency Testing And Failure Injection78## Overview910Use this skill when the goal is to prove that a data system recovers safely under failure, not only when everything goes right. It helps agents design controlled drills for retries, restarts, dependency outages, state recovery, duplicate prevention, backlog catch-up, and publish protection.1112## When to Use1314- hardening a production pipeline before broad rollout15- testing failover, restart, replay, or checkpoint recovery behavior16- validating that retries do not duplicate or corrupt data17- proving recovery objectives for orchestrators, jobs, streams, or warehouse publishes18- converting a past incident into a repeatable resilience drill1920Do not treat resilience testing as random breakage. The point is to validate recovery behavior with explicit safety limits and evidence.2122## Workflow23241. Define the failure modes that matter.25 Prioritize:26 - source outage or delayed upstream delivery27 - worker or task restart28 - orchestrator retry and timeout behavior29 - duplicate event or duplicate file delivery30 - checkpoint or incremental-state recovery31 - credential, secret, or network dependency failure32 - partial publish or downstream unavailability33342. Define the resilience objectives.35 Include:36 - acceptable data loss behavior37 - recovery time objective38 - replay or backlog catch-up expectation39 - duplicate-prevention requirement40 - publish block or quarantine behavior41 - alert and escalation expectation42433. Choose the safest drill environment.44 Prefer:45 - staging or isolated non-production46 - canary datasets or partitions47 - synthetic or masked test data48 - bounded windows and rollback-ready test scope49504. Inject one failure mode at a time.51 Use controlled exercises such as:52 - killing a task or worker53 - pausing an upstream dependency54 - delaying input arrival55 - replaying a duplicate input56 - forcing an expired secret or denied permission in a safe environment57 - simulating partial output and validating publish closure58595. Validate the recovery path.60 Check:61 - whether the system resumes or fails safely62 - whether alerts fire with useful context63 - whether duplicates are prevented64 - whether backlog catch-up stays bounded65 - whether publish remains blocked until validation passes66676. Record guardrails and automate the highest-value drills.68 The best resilience test is one the team can rerun after changes, not a one-time exercise that gets forgotten.69707. Load companion skills by failure mode.71 - replay or backfill drills: `safe-backfill-and-replay-orchestration`72 - Kafka lag, DLQ, or schema drift: `kafka-resilience-and-schema-evolution`73 - serverless Spark checkpoint recovery: `spark-serverless-reliability-and-state-management`74 - live diagnosis before drills: `mcp-data-observability-integration`75 - drill patterns: `references/data-resiliency-testing-patterns.md`7677## Common Rationalizations7879| Rationalization | Reality |80| --- | --- |81| "If the job retries, we are resilient enough." | Retry alone does not prove replay safety, duplicate prevention, or publish protection. |82| "We can test recovery during a real incident." | Real incidents are the worst time to discover the recovery path is unclear or unsafe. |83| "Failure injection is too risky for data systems." | Uncontrolled failure is riskier than bounded, reviewable drills in safe environments. |84| "The scheduler health page already proves resilience." | Scheduler status does not prove data correctness, backlog catch-up, or downstream safety. |8586## Red Flags8788- no list of prioritized failure modes exists89- retries are enabled without idempotency proof90- resilience drills have no rollback or blast-radius limits91- checkpoint or incremental-state recovery has never been tested92- alerts fire but recovery ownership is unclear93- a past incident has no corresponding regression drill9495## Verification9697- [ ] High-impact failure modes are named and prioritized98- [ ] Recovery objectives and acceptable failure behavior are explicit99- [ ] The drill scope is bounded and safe to run100- [ ] Recovery evidence covers alerts, replay safety, duplicate prevention, and publish protection101- [ ] At least one incident-derived failure mode is turned into a repeatable drill
vaquarkhan