Distributed Systems Investigator
Trace one concrete unit of work through the complete system before proposing a fix. Prefer event IDs, partition keys, correlation IDs, consumer offsets, durable rows, and timestamps over broad log searching.
Read references/investigation-playbook.md for the evidence matrix and failure patterns. When working in Sculptors Agent, also read the local service README and use its current topology and commands as the source of truth.
Workflow
Define the expected journey.
- Name the ingress, durable boundaries, queue topics, consumer groups, workers, stores, and final observable outcome.
- Separate acknowledged delivery, durable persistence, processing completion, and derived-index visibility.
Choose a trace key.
- Prefer an event ID plus tenant/workspace/entity identity and partition key.
- Record the expected ordering and idempotency key.
- Do not infer end-to-end success from HTTP success alone.
Inspect each boundary in order.
- Ingress acceptance and validation
- Producer publish and partition assignment
- Broker offset and consumer-group ownership
- Worker claim, retry, timeout, and commit behavior
- Canonical database transaction
- Analytics/audit persistence
- Derived vector or search indexing
- Final retrieval or user-facing response
Classify the first broken invariant.
- Never-produced
- Produced but unconsumed
- Consumed but uncommitted
- Committed before side effect
- Partially persisted
- Persisted but not indexed
- Indexed with stale or wrong identity
- Correct data hidden by retrieval/filtering
Reproduce at the smallest safe scale.
- Start with one event, then a small ordered batch, then concurrency.
- Preserve production partitioning and retry semantics.
- Avoid cleanup or replay until evidence is captured.
Implement only after diagnosis when the user asked for a fix.
- Add an invariant test that fails before the change.
- Make the smallest change at the first broken boundary.
- Verify retries, duplicates, ordering, shutdown, and partial failure.
Evidence Standard
For every conclusion, identify:
- verified evidence,
- inference,
- missing evidence,
- alternative explanations,
- the command or artifact that would falsify the conclusion.
Do not call an incident resolved because queues are empty. Verify the final canonical and derived states for known trace keys.
Safety
- Treat replay, topic reset, record deletion, consumer-group offset changes, and production writes as state-changing operations requiring explicit authority.
- Never print secrets or full sensitive payloads.
- Prefer read-only broker, database, and log inspection during diagnosis.
- Preserve unrelated local changes and existing operational artifacts.
Verification
Use the repository's established checks. For Sculptors Agent, typical evidence includes targeted pytest cases, preflight checks, Kafka group/offset inspection, canonical Supabase rows, ClickHouse event/audit rows, Qdrant payload coverage, and the existing memory quality/e2e reports.
1---2name: distributed-systems-investigator3description: Diagnose and verify failures across distributed event pipelines, queues, consumer groups, workers, APIs, databases, vector stores, analytics stores, retries, dead-letter paths, and asynchronous consistency boundaries. Use for Kafka lag, missing events, duplicate processing, out-of-order writes, worker stalls, partial persistence, Supabase/Qdrant/ClickHouse divergence, throughput collapse, backpressure, and end-to-end data-loss investigations.4---56# Distributed Systems Investigator78Trace one concrete unit of work through the complete system before proposing a fix. Prefer event IDs, partition keys, correlation IDs, consumer offsets, durable rows, and timestamps over broad log searching.910Read [references/investigation-playbook.md](references/investigation-playbook.md) for the evidence matrix and failure patterns. When working in Sculptors Agent, also read the local service README and use its current topology and commands as the source of truth.1112## Workflow13141. Define the expected journey.15 - Name the ingress, durable boundaries, queue topics, consumer groups, workers, stores, and final observable outcome.16 - Separate acknowledged delivery, durable persistence, processing completion, and derived-index visibility.17182. Choose a trace key.19 - Prefer an event ID plus tenant/workspace/entity identity and partition key.20 - Record the expected ordering and idempotency key.21 - Do not infer end-to-end success from HTTP success alone.22233. Inspect each boundary in order.24 - Ingress acceptance and validation25 - Producer publish and partition assignment26 - Broker offset and consumer-group ownership27 - Worker claim, retry, timeout, and commit behavior28 - Canonical database transaction29 - Analytics/audit persistence30 - Derived vector or search indexing31 - Final retrieval or user-facing response32334. Classify the first broken invariant.34 - Never-produced35 - Produced but unconsumed36 - Consumed but uncommitted37 - Committed before side effect38 - Partially persisted39 - Persisted but not indexed40 - Indexed with stale or wrong identity41 - Correct data hidden by retrieval/filtering42435. Reproduce at the smallest safe scale.44 - Start with one event, then a small ordered batch, then concurrency.45 - Preserve production partitioning and retry semantics.46 - Avoid cleanup or replay until evidence is captured.47486. Implement only after diagnosis when the user asked for a fix.49 - Add an invariant test that fails before the change.50 - Make the smallest change at the first broken boundary.51 - Verify retries, duplicates, ordering, shutdown, and partial failure.5253## Evidence Standard5455For every conclusion, identify:5657- verified evidence,58- inference,59- missing evidence,60- alternative explanations,61- the command or artifact that would falsify the conclusion.6263Do not call an incident resolved because queues are empty. Verify the final canonical and derived states for known trace keys.6465## Safety6667- Treat replay, topic reset, record deletion, consumer-group offset changes, and production writes as state-changing operations requiring explicit authority.68- Never print secrets or full sensitive payloads.69- Prefer read-only broker, database, and log inspection during diagnosis.70- Preserve unrelated local changes and existing operational artifacts.7172## Verification7374Use the repository's established checks. For Sculptors Agent, typical evidence includes targeted pytest cases, preflight checks, Kafka group/offset inspection, canonical Supabase rows, ClickHouse event/audit rows, Qdrant payload coverage, and the existing memory quality/e2e reports.
xhalilbey