sethdford Concurrency Debugging
Identifying bugs caused by concurrent access and timing issues.
Context
You are debugging concurrency issues. These are the hardest bugs to find and reproduce.
Domain Context
- Race Condition: Two threads access shared data; order matters; result is non-deterministic
- Deadlock: Threads waiting for each other; system hangs
- Happens-Before: Guarantee that one event completes before another
- Memory Visibility: When one thread sees writes from another thread
Instructions
- Reproduce Consistently: Add logging, delays, thread barriers
- Check Synchronization: Are shared variables protected?
- Verify Lock Ordering: Do all threads acquire locks in same order?
- Analyze Thread Dumps: What is each thread waiting for?
- Look for Wait-Free Patterns: Are threads spinning, not blocking?
- Use Tools: ThreadSanitizer, Helgrind (Valgrind), or language-specific tools
- Add Happens-Before: Use happens-before semantics (memory barriers, atomic ops)
Anti-Patterns
- Single-threaded debugging then assuming it works threaded; concurrency bugs are different
- Using sleep() for synchronization; racey and slow
- Assuming x++ is atomic; it's not, use atomic operations
- Multiple locks protecting same data; hard to reason about
- Not testing under load; concurrency bugs often need high contention to appear
Further Reading
- Brian Gorelick, Concurrency in Practice
- Java Concurrency in Practice (excellent; applies to other languages too)
- ThreadSanitizer documentation