majiayu000 FDB Simulation Workloads
Patterns for designing Rust workloads that find bugs through autonomous chaos testing.
Determinism Rules (Critical)
Breaking any rule destroys reproducibility. The simulator requires identical seeds to produce identical execution paths.
| Instead of | Use | Why |
|---|---|---|
HashMap / HashSet |
BTreeMap / BTreeSet |
Iteration order must be deterministic |
rand::random() |
context.rnd() |
Seeded randomness for reproducibility |
SystemTime::now() |
context.now() |
Simulation controls time |
| Manual retry loops | db.run() |
Proper retry handling with maybe_committed |
tokio::spawn() |
Never | Simulation uses custom executor |
Operation Alphabet
Design operations across three categories:
| Category | Purpose | Examples |
|---|---|---|
| Normal | Mirror production traffic distribution | 80% reads, 15% writes, 5% complex updates |
| Adversarial | Edge cases customers will send | Empty strings, max-length values, null bytes, boundary integers |
| Nemesis | Deliberately break things | Delete random data, crash mid-batch, conflict storms, approach 10MB limit |
Invariant Patterns
Verify correctness continuously during execution, not just at the end.
| Pattern | Description | Example |
|---|---|---|
| Reference Model | In-memory copy of expected state; compare in check phase | BTreeMap tracking all expected key-values |
| Conservation Laws | Quantities that must remain constant | Total money across accounts unchanged |
| Structural Integrity | Data structure validity checks | Index entries point to existing records |
| Operation Logging | Log intent in same transaction | Eliminates maybe_committed uncertainty |
Three-Crate Architecture
Separate production frameworks from simulation-testable code:
my-project/
├── my-fdb-service/ # Core FDB operations - NO tokio
├── my-grpc-server/ # Production layer (tokio + tonic)
└── my-fdb-workloads/ # Simulation tests
The service crate contains pure db.run() transaction logic. The server crate wraps it for production. The workloads crate tests actual service code under chaos.
Common Pitfalls
- Initialization on all clients - Use
if self.client_id == 0to run setup once - Ignoring
maybe_committed- Check the flag indb.run()closure for idempotency - Storing database references - Each phase receives fresh references; don't cache them
- Wrapping
FdbError- Keep errors unwrapped so retry mechanism detects retryable errors - Assuming setup is failure-free - FDB knobs randomize; always use
db.run()with retry logic
Full Reference
For detailed examples, patterns, and rationale: https://pierrezemb.fr/posts/writing-rust-fdb-workloads-that-find-bugs/