Debug Issue with Datadog
Use this skill whenever the task is investigative rather than
implementational: a user, customer, or oncall has surfaced a problem and you
need to figure out what is actually happening in production and where in the
code it lives. The deliverable is an analysis, not a patch — though the
analysis should make the right patch obvious.
When to Apply
- A Linear issue (typically with an
LFE-XXXX ID) describes a production
failure, error spike, or customer report.
- A GitHub issue or pasted incident/error report needs triage.
- A monitor alerted and you need to understand why before deciding what to
fix.
- Existing tickets under the "Make monitoring useful again" project (parent
LFE-8837) and similar — these expect the structured analysis output below.
If the task is "implement this fix" rather than "figure out what's broken",
this is the wrong skill — go to backend-dev-guidelines or the relevant
package guide.
Workflow
Read the inputs first, then plan the Datadog sweep, then read the code, then
write the analysis. Do not skip ahead to suggested patches before the data
supports them.
Intake. Pull every signal already available in the report. See
references/intake.md. For a Linear URL/ID, fetch
the issue and its comments via the Linear MCP — the description is often
updated inline as triage proceeds. For a GitHub issue, use gh issue view.
For pasted text, treat it as the description.
If intake contains an alert identity (a Datadog monitor ID or title, an
incident.io alert/INC reference, or an on-call page), first apply
incident-alert-tickets — a
documented cause section may resolve the investigation before any sweep.
Scope the sweep. From the intake, pick the affected subsystem and time
window. Use references/repo-debug-map.md
to translate "PostHog integration", "ingestion failures", "evals stuck",
etc. into the Datadog filters and source files you should be looking at.
Run the broad Datadog sweep. Default to the full sweep in
references/datadog-playbook.md: APM
spans, error logs, metrics, and monitors — split across prod-eu
and prod-us (and prod-hipaa / prod-jp when relevant). Always check
regional disparity first; it usually rules whole hypotheses in or out.
Use datadog-query-recipes for
reusable tenant, public API, queue consumer, and cross-environment query
shapes.
Cluster the errors. Group by (projectId, error.message) or
(error.type, error.message). Treat each distinct cluster as its own
hypothesis — Langfuse incidents commonly have multiple coexisting root
causes, not one.
Map clusters to code. For each cluster, open the relevant handler file
from the repo-debug map and read enough of it to confirm or refute the
hypothesis. Cite specific files and line ranges in the output.
Write the analysis using
references/output-template.md.
Deliver. Default: print the analysis in chat. If the user asked for it,
also save under the workflow they specified (file, Linear comment via, etc.).
If the investigation was anchored to an alert identity, also offer the
human-gated write-back from
incident-alert-tickets: append the
established root cause as a dated cause section, or create the monitor's
ticket.
Datadog MCP Usage Notes
Two Datadog MCP servers are typically available — one bound to the EU site
(datadoghq.eu) and one to the US site (datadoghq.com). Always run
region-relevant queries against both unless intake clearly localizes the
incident. The prod-eu / prod-us env tags live on each side respectively.
- Span search filter pattern:
service:worker resource_name:"process posthog-integration-project" status:error
- Log search filter pattern:
service:worker env:prod-eu @langfuse.project.id:cm1r6u… status:error
- For high-volume queries, prefer
aggregate_spans / aggregate_events
grouped by (error.message, projectId) over fetching individual traces.
- Always link to the Datadog UI for the queries you ran (final section of the
output template).
See references/datadog-playbook.md for the
full set of starter queries and parameter shapes.
Output Expectations
From the output template:
- Header: data source, time window, region split (EU vs US table).
- Hotspots: per-
projectId (or per-cluster) error counts.
- Root cause by error class: each cluster gets a short hypothesis with
reasoning, distinguishing primary causes from symptoms.
- Suggested patches: P0/P1/P2 grouped, with concrete file paths and short code
sketches. Reference the actual handler in
worker/src/features/** or
web/src/**.
- Dashboards: paste the Datadog query URLs at the end.
Findings come first, recommendations last. If the data is thin, say so
explicitly and propose what would need to be true to confirm each hypothesis —
do not invent root causes.
Cross-References
- Per-monitor knowledge base — look up documented causes before the sweep,
record new ones after (human-gated):
incident-alert-tickets
- Production telemetry query recipes, tenant/public API usage, and queue
consumer measurements:
datadog-query-recipes
- Backend layout, queue contracts, instrumentation patterns:
backend-dev-guidelines
- ClickHouse-related findings (memory ceilings, JOIN spills, slow queries):
clickhouse-best-practices
- Once a fix is identified and you switch to implementation, hand off to the
package
AGENTS.md for the affected directory.
1---
2name: debug-issue-with-datadog
3description: |
4 Establish root cause by combining Datadog telemetry with the Langfuse repo.
5 Use when investigating or triaging a user report, Linear or GitHub issue,
6 incident, or pasted production error.
7metadata:
8 source: https://skillmd.com/skills/langfuse/debug-issue-with-datadog
9---
10
11# Debug Issue with Datadog
12
13Use this skill whenever the task is **investigative** rather than
14implementational: a user, customer, or oncall has surfaced a problem and you
15need to figure out *what is actually happening in production* and *where in the
16code it lives*. The deliverable is an analysis, not a patch — though the
17analysis should make the right patch obvious.
18
19## When to Apply
20
21- A Linear issue (typically with an `LFE-XXXX` ID) describes a production
22 failure, error spike, or customer report.
23- A GitHub issue or pasted incident/error report needs triage.
24- A monitor alerted and you need to understand *why* before deciding what to
25 fix.
26- Existing tickets under the "Make monitoring useful again" project (parent
27 `LFE-8837`) and similar — these expect the structured analysis output below.
28
29If the task is "implement this fix" rather than "figure out what's broken",
30this is the wrong skill — go to `backend-dev-guidelines` or the relevant
31package guide.
32
33## Workflow
34
35Read the inputs first, then plan the Datadog sweep, then read the code, then
36write the analysis. Do not skip ahead to suggested patches before the data
37supports them.
38
391. **Intake.** Pull every signal already available in the report. See
40 [`references/intake.md`](references/intake.md). For a Linear URL/ID, fetch
41 the issue *and* its comments via the Linear MCP — the description is often
42 updated inline as triage proceeds. For a GitHub issue, use `gh issue view`.
43 For pasted text, treat it as the description.
44 If intake contains an alert identity (a Datadog monitor ID or title, an
45 incident.io alert/INC reference, or an on-call page), first apply
46 [`incident-alert-tickets`](../incident-alert-tickets/SKILL.md) — a
47 documented cause section may resolve the investigation before any sweep.
48
492. **Scope the sweep.** From the intake, pick the affected subsystem and time
50 window. Use [`references/repo-debug-map.md`](references/repo-debug-map.md)
51 to translate "PostHog integration", "ingestion failures", "evals stuck",
52 etc. into the Datadog filters and source files you should be looking at.
53
543. **Run the broad Datadog sweep.** Default to the full sweep in
55 [`references/datadog-playbook.md`](references/datadog-playbook.md): APM
56 spans, error logs, metrics, and monitors — split across `prod-eu`
57 and `prod-us` (and `prod-hipaa` / `prod-jp` when relevant). Always check
58 regional disparity first; it usually rules whole hypotheses in or out.
59 Use [`datadog-query-recipes`](../datadog-query-recipes/SKILL.md) for
60 reusable tenant, public API, queue consumer, and cross-environment query
61 shapes.
62
634. **Cluster the errors.** Group by `(projectId, error.message)` or
64 `(error.type, error.message)`. Treat each distinct cluster as its own
65 hypothesis — Langfuse incidents commonly have *multiple* coexisting root
66 causes, not one.
67
685. **Map clusters to code.** For each cluster, open the relevant handler file
69 from the repo-debug map and read enough of it to confirm or refute the
70 hypothesis. Cite specific files and line ranges in the output.
71
726. **Write the analysis** using
73 [`references/output-template.md`](references/output-template.md).
74
757. **Deliver.** Default: print the analysis in chat. If the user asked for it,
76 also save under the workflow they specified (file, Linear comment via, etc.).
77 If the investigation was anchored to an alert identity, also offer the
78 human-gated write-back from
79 [`incident-alert-tickets`](../incident-alert-tickets/SKILL.md): append the
80 established root cause as a dated cause section, or create the monitor's
81 ticket.
82
83## Datadog MCP Usage Notes
84
85Two Datadog MCP servers are typically available — one bound to the EU site
86(`datadoghq.eu`) and one to the US site (`datadoghq.com`). Always run
87region-relevant queries against **both** unless intake clearly localizes the
88incident. The `prod-eu` / `prod-us` env tags live on each side respectively.
89
90- Span search filter pattern:
91 `service:worker resource_name:"process posthog-integration-project" status:error`
92- Log search filter pattern:
93 `service:worker env:prod-eu @langfuse.project.id:cm1r6u… status:error`
94- For high-volume queries, prefer `aggregate_spans` / `aggregate_events`
95 grouped by `(error.message, projectId)` over fetching individual traces.
96- Always link to the Datadog UI for the queries you ran (final section of the
97 output template).
98
99See [`references/datadog-playbook.md`](references/datadog-playbook.md) for the
100full set of starter queries and parameter shapes.
101
102## Output Expectations
103
104From the output template:
105
106- Header: data source, time window, region split (EU vs US table).
107- Hotspots: per-`projectId` (or per-cluster) error counts.
108- Root cause by error class: each cluster gets a short hypothesis with
109 reasoning, distinguishing primary causes from symptoms.
110- Suggested patches: P0/P1/P2 grouped, with concrete file paths and short code
111 sketches. Reference the actual handler in `worker/src/features/**` or
112 `web/src/**`.
113- Dashboards: paste the Datadog query URLs at the end.
114
115Findings come first, recommendations last. If the data is thin, say so
116explicitly and propose what would need to be true to confirm each hypothesis —
117do not invent root causes.
118
119## Cross-References
120
121- Per-monitor knowledge base — look up documented causes before the sweep,
122 record new ones after (human-gated):
123 [`incident-alert-tickets`](../incident-alert-tickets/SKILL.md)
124- Production telemetry query recipes, tenant/public API usage, and queue
125 consumer measurements:
126 [`datadog-query-recipes`](../datadog-query-recipes/SKILL.md)
127- Backend layout, queue contracts, instrumentation patterns:
128 [`backend-dev-guidelines`](../backend-dev-guidelines/SKILL.md)
129- ClickHouse-related findings (memory ceilings, JOIN spills, slow queries):
130 [`clickhouse-best-practices`](../clickhouse-best-practices/SKILL.md)
131- Once a fix is identified and you switch to implementation, hand off to the
132 package `AGENTS.md` for the affected directory.