imtiazrayhan Trace asynchronous work from intent to verified outcome.
Workflow
- Define the required outcome. Identify the user or business effect, acceptable delay, duplication tolerance, loss tolerance, ordering need, and recovery objective.
- Map the lifecycle. Trace producer commit, enqueue, broker delivery, lease or visibility timeout, worker processing, database writes, external effects, acknowledgement, retry, dead-letter handling, and reconciliation.
- State delivery assumptions. Verify what the scheduler or broker actually guarantees under crashes, timeouts, partitions, and redelivery. Do not infer exactly-once effects from product terminology.
- Audit idempotency and transactions. Find the stable operation key, deduplication store, uniqueness boundary, transaction scope, outbox or inbox pattern, and behavior when an external side effect succeeds before local state commits.
- Review failure policy. Check exception classification, backoff, jitter, maximum attempts, retry budget, timeout hierarchy, poison-message isolation, dead-letter ownership, and replay procedure.
- Inspect concurrency and lifecycle. Verify prefetch, worker count, per-key ordering, rate limits, locks, heartbeat or lease extension, graceful shutdown, deploy draining, checkpointing, and resource cleanup.
- Assess observability. Require correlation from request or schedule to job and effect. Measure enqueue rate, completion rate, latency, oldest age, attempts, terminal failures, dead letters, throughput, and saturation.
- Exercise recovery. Define tests for crash before and after side effects, duplicate delivery, broker outage, dependency timeout, malformed payload, deploy interruption, backlog drain, and dead-letter replay.
[!WARNING] A retry policy without idempotency can amplify an outage into duplicate charges, emails, allocations, or data corruption.
Output
Return a lifecycle diagram in text, verified delivery assumptions, prioritized findings with code or configuration evidence, missing telemetry, failure-injection cases, and a remediation plan. Separate containment from durable fixes.