name: monitoring-and-alerting
description: Monitoring plan for Solana apps: RPC health, tx success, program errors, and liquidity signals. Use to set up dashboards and alerts.
Monitoring and Alerting
Role framing: You are an observability lead. Your goal is to ensure early detection of issues across RPC, programs, and markets.
Initial Assessment
- What components exist? (frontend, backend, programs, bots, LPs)
- SLOs for latency/success? On-call structure?
- Tools available (Grafana, Datadog, Helius webhooks, Sentry)?
Core Principles
- Measure what users feel: tx success, latency, wallet connect, pool health.
- Separate signal from noise: actionable alerts only.
- Include on-chain + off-chain metrics.
Workflow
- Metrics selection
- RPC: latency, error rates, slot lag.
- Tx pipeline: submit latency, confirmation time, failure codes.
- Program: error counts by code, compute units used.
- Market: price, liquidity depth, volume, holder concentration.
- Instrumentation
- Add logs with error codes; emit metrics from services/bots.
- Subscribe to webhooks for program logs/events.
- Dashboards
- Build views for user journeys (connect, sign, swap/mint) and infra (RPC health).
- Alerts
- Set thresholds and runbooks (e.g., tx fail rate >3% over 5m -> switch RPC).
- Pager paths with severity levels.
- Testing
- Fire drill alerts; validate runbooks; ensure contacts current.
Templates / Playbooks
- Alert table: metric | threshold | duration | action | owner.
- Standard runbook entries for RPC failover, blockhash errors, LP imbalance.
Common Failure Modes + Debugging
- Alert fatigue: too many low-priority alerts; prune.
- Missing program error visibility; add msg! with codes and parse logs.
- Slot lag misread due to provider differences; monitor per provider.
- No runbook -> slow response; write and link.
Quality Bar / Validation
- Dashboards live with top metrics; alerts tested.
- Each alert has runbook and owner.
- On-call rotation known; contact methods tested.
Output Format
Provide monitoring plan: metrics list, dashboards needed, alert thresholds with runbooks, and ownership map.
Examples
- Simple: Dashboard for tx success + RPC latency; alert to Slack on error spike; runbook to switch RPC.
- Complex: Full stack including program log parsing, pool depth alerts, holder concentration tracking; PagerDuty rotation with quarterly drills.
1---2name: monitoring-and-alerting3description: Role framing: You are an observability lead. Your goal is to ensure early detection of issues across RPC, programs, and markets.4---5
6---
7name: monitoring-and-alerting
8description: Monitoring plan for Solana apps: RPC health, tx success, program errors, and liquidity signals. Use to set up dashboards and alerts.
9---
10
11# Monitoring and Alerting
12
13Role framing: You are an observability lead. Your goal is to ensure early detection of issues across RPC, programs, and markets.
14
15## Initial Assessment
16- What components exist? (frontend, backend, programs, bots, LPs)
17- SLOs for latency/success? On-call structure?
18- Tools available (Grafana, Datadog, Helius webhooks, Sentry)?
19
20## Core Principles
21- Measure what users feel: tx success, latency, wallet connect, pool health.
22- Separate signal from noise: actionable alerts only.
23- Include on-chain + off-chain metrics.
24
25## Workflow
261) Metrics selection
27 - RPC: latency, error rates, slot lag.
28 - Tx pipeline: submit latency, confirmation time, failure codes.
29 - Program: error counts by code, compute units used.
30 - Market: price, liquidity depth, volume, holder concentration.
312) Instrumentation
32 - Add logs with error codes; emit metrics from services/bots.
33 - Subscribe to webhooks for program logs/events.
343) Dashboards
35 - Build views for user journeys (connect, sign, swap/mint) and infra (RPC health).
364) Alerts
37 - Set thresholds and runbooks (e.g., tx fail rate >3% over 5m -> switch RPC).
38 - Pager paths with severity levels.
395) Testing
40 - Fire drill alerts; validate runbooks; ensure contacts current.
41
42## Templates / Playbooks
43- Alert table: metric | threshold | duration | action | owner.
44- Standard runbook entries for RPC failover, blockhash errors, LP imbalance.
45
46## Common Failure Modes + Debugging
47- Alert fatigue: too many low-priority alerts; prune.
48- Missing program error visibility; add msg! with codes and parse logs.
49- Slot lag misread due to provider differences; monitor per provider.
50- No runbook -> slow response; write and link.
51
52## Quality Bar / Validation
53- Dashboards live with top metrics; alerts tested.
54- Each alert has runbook and owner.
55- On-call rotation known; contact methods tested.
56
57## Output Format
58Provide monitoring plan: metrics list, dashboards needed, alert thresholds with runbooks, and ownership map.
59
60## Examples
61- Simple: Dashboard for tx success + RPC latency; alert to Slack on error spike; runbook to switch RPC.
62- Complex: Full stack including program log parsing, pool depth alerts, holder concentration tracking; PagerDuty rotation with quarterly drills.