# Concurrency Reasoning > Concurrency Reasoning - Skill: `annatchijova/concurrency-reasoning` (Agent Skill) - Install (CLI): `npx skillmds@latest add annatchijova/concurrency-reasoning` - Raw SKILL.md: https://api.skillmd.com/api/skills/annatchijova/concurrency-reasoning/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - Author: annatchijova (https://skillmd.com/u/annatchijova) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/annatchijova/concurrency-reasoning --- # Concurrency Reasoning Concurrency bugs are not hard because concurrency is exotic. They are hard because the reasoning most people apply is about **simultaneity** — "two things happening at the same time" — and the actual subject is **ordering and visibility**. Simultaneity is not observable, not reproducible, and not what breaks. Ordering is. The second reason they are hard: the failing interleaving is rare, so the test passes, the review passes, and the defect ships with full confidence behind it. A concurrency bug is a defect whose evidence is *statistical*, in a discipline that treats a green test as proof. Composes with: `atomic-state-mutation` (the same problem confined to a database transaction), `falsifiable-testing` (a flake is often a race reporting itself), `incident-timeline-reconstruction` (no global clock, same lesson), `irreversible-action-gate` (check-then-act is TOCTOU), `deterministic-core` (nondeterminism you did not choose). --- ## Part 1 — Name the shared mutable state, or stop A data race needs exactly three ingredients: 1. **State reachable from more than one execution context** 2. **Concurrent access to it** 3. **At least one write** Remove any one and the race is gone — and they are not equally expensive to remove. In order of preference: - **Eliminate the sharing.** Per-worker state, a value copied instead of referenced, immutability, single ownership, a message passed instead of memory shared. This is the only fix that cannot be got wrong later by someone who does not know the rule. - **Eliminate the write.** Compute once before the concurrency starts; publish an immutable snapshot; recompute rather than mutate. - **Make the operation atomic.** A compare-and-swap, an atomic counter, a single `UPDATE ... WHERE`, a unique constraint doing the arbitration for you. - **Serialize with a lock.** Correct, and the most fragile, because its correctness lives in the discipline of every future caller rather than in the structure of the data. Before any of that: **write down what is shared.** If you cannot name the shared mutable state in a sentence, you are not yet reasoning about the bug — and a lock added at that stage protects an unknown region against an unknown invariant. --- ## Part 2 — "I added a lock" is not an argument Locks do not protect *variables*. They protect **invariants** — a relationship that must hold across several reads and writes. So the questions are: 1. **What is the invariant?** ("balance never goes negative"; "the cache entry and its expiry timestamp are always consistent"; "at most one worker owns this job".) 2. **Over which region must it hold?** That region is the critical section, and it must span the **entire read-modify-write, including the decision** — not just the write. 3. **Does every path that touches the state take the same lock?** One unguarded accessor invalidates all the others. Locks are a whole-program property masquerading as a local one. **Check-then-act is the canonical bug.** `if not exists: create`, `if balance >= amount: debit`, `if not locked: take_lock`, `if file_missing: write`. The condition was true when checked and false when acted on, because someone else ran in between. Fix it by collapsing the check and the act into one atomic operation — a conditional write, a unique constraint, a CAS, `INSERT ... ON CONFLICT` — or by holding the invariant's lock across both. This is the same time-of-check/time-of-use shape as in `irreversible-action-gate`: the preview and the action must refer to the same state. **Scope the lock to the invariant, not to the function.** Too narrow and the invariant breaks between two guarded regions; too wide and you have serialized your system and turned a correctness tool into a throughput bug. Both failures look like "we have a lock". --- ## Part 3 — Happens-before, not "at the same time" Two facts that reshape the reasoning: **Ordering.** Without an explicit synchronization relationship, there is no "before". Two operations are ordered only if something establishes it: a lock acquired and released, a channel send and receive, a thread start or join, an atomic with the right memory ordering, a database transaction boundary, a causal dependency in a message. Absent that link, every interleaving is legal — including the ones you did not imagine and the ones your hardware and compiler are free to invent by reordering. **Visibility.** A write performed without synchronization may **never become visible** to another context. This is not a slow update; it is a permanently stale read, and it defeats the entire genre of "I'll just check the flag". `while not self.done: pass` can loop forever against a `done` that was set minutes ago. In distributed systems the same lesson arrives wearing different clothes: there is no global clock, so timestamps from two machines cannot order two events (exactly the clock-domain problem in `incident-timeline-reconstruction`). Order by causality — sequence numbers, versions, fencing tokens, request IDs — never by comparing wall clocks. --- ## Part 4 — The concurrency you never declared Most concurrency bugs are found in code whose author never wrote a thread. You are concurrent the moment any of these is true: - The service runs as **more than one process, pod, or replica**. "Single-threaded code" stops being single-anything at the second instance. - Anything **retries** — a client, a proxy, a queue with at-least-once delivery, a user reloading the page. - A **scheduled job can overlap with itself** because the previous run has not finished. This is the classic silent doubler. - **Webhooks and callbacks** arrive more than once, or out of order, by design. - The UI lets someone **click twice**, or two people act on the same entity. - Async/await, callbacks, signal handlers, or a garbage-collector finalizer interleave inside a single thread — no parallelism required, same reordering. For each: ask what happens if this runs **twice, concurrently, with the second starting before the first commits**. That single question finds more real defects than any static analysis. --- ## Part 5 — Idempotency beats "exactly once" Exactly-once *delivery* does not exist; exactly-once *effect* does, and you build it rather than configure it: - **Idempotency key** supplied by the caller, stored with the result. A repeat returns the stored result instead of doing the work again. - **Let the database arbitrate.** A unique constraint is a race resolver that cannot be forgotten by the next caller — far more durable than a check. - **Guard state transitions on the current state**: `UPDATE ... WHERE status = 'pending'` and check the affected row count. Zero rows means someone else won, which is information, not an error. - **Fencing tokens** for anything holding a "lease": a lock that can expire must hand out a monotonically increasing token, and the resource must reject stale ones. Otherwise a paused worker wakes up and writes with a lock it no longer holds — the failure mode that makes distributed locks dangerous rather than merely unreliable. And retries have a systemic cost: synchronized retries turn a blip into an outage. Bound them, add jitter, add a budget or circuit breaker, and never retry a non-idempotent operation whose outcome you do not know — check first, or make it idempotent so you do not have to. --- ## Part 6 — Deadlock and liveness - **Lock ordering.** Deadlock needs a cycle. Impose a global order on lock acquisition and document it; almost all deadlocks are two code paths taking the same two locks in opposite orders. - **Never hold a lock across I/O.** A network call under a mutex converts remote latency into local unavailability, and a remote hang into a permanent one. - **Everything blocking gets a timeout** — locks, queues, connections, waits. A system without timeouts does not fail; it stops, which is harder to diagnose and much harder to recover. - **Distinguish hung from slow.** They have different causes and opposite fixes. Instrument wait time explicitly rather than inferring it from total latency. - **Watch for starvation and convoying**, not just deadlock: a lock that is always available to someone and never to *this* caller is a liveness bug with no error message. --- ## Part 7 — Testing what only sometimes happens A passing concurrency test proves that **one interleaving** worked, on one machine, once. Treat that for what it is: - **Use the detectors.** Race detectors, thread sanitizers, and lock-order checkers find real defects that no test will reproduce on schedule. Run them in CI, not on demand. - **Stress with variation, not repetition.** Randomize timing, injected delays, and ordering; run with more workers than cores; run the same operation concurrently N times and assert the invariant afterwards (the sum is right, exactly one winner, no duplicates) rather than asserting on a single call's return. - **Assert invariants, not outcomes.** Under concurrency the per-call result is legitimately nondeterministic; the invariant is not. This is the property-based rung of `falsifiable-testing` applied to schedules. - **Force the interleaving you fear.** A hook, a barrier, or an injected pause that makes the rare ordering deterministic turns "we think it's fixed" into a regression test that fails on the unfixed code — the red-first rule, applied to a bug that will not cooperate on its own. - **A flake under parallelism is a finding.** It is the race reporting itself, on its own schedule. Retrying it deletes the report and leaves the defect (`falsifiable-testing`). --- ## Deliverable checklist ```markdown ## Concurrency review Shared mutable state: Concurrency sources: threads · async · replicas · retries · overlapping cron · duplicate webhooks · double-click · signal/callback (which apply) Invariant: Critical section: Check-then-act: any? → collapsed into an atomic op / guarded across both steps Ordering: what establishes happens-before between the accesses Visibility: is every cross-context read synchronized (no unsynchronized flag polling) Runs twice concurrently: what happens? → idempotency key / unique constraint / state-guarded update + affected-row check / fencing token Locks: ordering documented · no I/O held under lock · timeouts on every wait Tests: detector in CI · stress with randomized timing · invariant assertions · the feared interleaving forced deterministically ``` --- ## How to respond when this skill is active - Ask what the shared mutable state is before discussing any fix; refuse to evaluate a lock whose invariant has not been stated. - Prefer eliminating sharing over guarding it, and say why: a structural fix cannot be forgotten by the next caller. - Name check-then-act on sight and offer the atomic form (conditional update, unique constraint, CAS) rather than a wider lock. - Reframe "at the same time" as ordering and visibility; point out unsynchronized reads that may never observe a write. - Ask "what if this runs twice, concurrently, with the second starting before the first commits" for every handler, job, and webhook. - Treat a flaky-under-parallelism test as a defect report, and force the interleaving into a deterministic regression test before declaring the fix. - For database-scoped versions of these problems, hand off to `atomic-state-mutation` rather than reimplementing transaction reasoning here.