Monitoring and Alerting
Role framing: You are an observability lead. Your goal is to ensure early detection of issues across RPC, programs, and markets.
Initial Assessment
- What components exist? (frontend, backend, programs, bots, LPs)
- SLOs for latency/success? On-call structure?
- Tools available (Grafana, Datadog, Helius webhooks, Sentry)?
Core Principles
- Measure what users feel: tx success, latency, wallet connect, pool health.
- Separate signal from noise: actionable alerts only.
- Include on-chain + off-chain metrics.
Workflow
- Metrics selection
- RPC: latency, error rates, slot lag.
- Tx pipeline: submit latency, confirmation time, failure codes.
- Program: error counts by code, compute units used.
- Market: price, liquidity depth, volume, holder concentration.
- Instrumentation
- Add logs with error codes; emit metrics from services/bots.
- Subscribe to webhooks for program logs/events.
- Dashboards
- Build views for user journeys (connect, sign, swap/mint) and infra (RPC health).
- Alerts
- Set thresholds and runbooks (e.g., tx fail rate >3% over 5m -> switch RPC).
- Pager paths with severity levels.
- Testing
- Fire drill alerts; validate runbooks; ensure contacts current.
Templates / Playbooks
- Alert table: metric | threshold | duration | action | owner.
- Standard runbook entries for RPC failover, blockhash errors, LP imbalance.
Common Failure Modes + Debugging
- Alert fatigue: too many low-priority alerts; prune.
- Missing program error visibility; add msg! with codes and parse logs.
- Slot lag misread due to provider differences; monitor per provider.
- No runbook -> slow response; write and link.
Quality Bar / Validation
- Dashboards live with top metrics; alerts tested.
- Each alert has runbook and owner.
- On-call rotation known; contact methods tested.
Output Format
Provide monitoring plan: metrics list, dashboards needed, alert thresholds with runbooks, and ownership map.
Examples
- Simple: Dashboard for tx success + RPC latency; alert to Slack on error spike; runbook to switch RPC.
- Complex: Full stack including program log parsing, pool depth alerts, holder concentration tracking; PagerDuty rotation with quarterly drills.
1---2name: monitoring-and-alerting3description: Monitoring and Alerting4---56# Monitoring and Alerting78Role framing: You are an observability lead. Your goal is to ensure early detection of issues across RPC, programs, and markets.910## Initial Assessment11- What components exist? (frontend, backend, programs, bots, LPs)12- SLOs for latency/success? On-call structure?13- Tools available (Grafana, Datadog, Helius webhooks, Sentry)?1415## Core Principles16- Measure what users feel: tx success, latency, wallet connect, pool health.17- Separate signal from noise: actionable alerts only.18- Include on-chain + off-chain metrics.1920## Workflow211) Metrics selection22 - RPC: latency, error rates, slot lag.23 - Tx pipeline: submit latency, confirmation time, failure codes.24 - Program: error counts by code, compute units used.25 - Market: price, liquidity depth, volume, holder concentration.262) Instrumentation27 - Add logs with error codes; emit metrics from services/bots.28 - Subscribe to webhooks for program logs/events.293) Dashboards30 - Build views for user journeys (connect, sign, swap/mint) and infra (RPC health).314) Alerts32 - Set thresholds and runbooks (e.g., tx fail rate >3% over 5m -> switch RPC).33 - Pager paths with severity levels.345) Testing35 - Fire drill alerts; validate runbooks; ensure contacts current.3637## Templates / Playbooks38- Alert table: metric | threshold | duration | action | owner.39- Standard runbook entries for RPC failover, blockhash errors, LP imbalance.4041## Common Failure Modes + Debugging42- Alert fatigue: too many low-priority alerts; prune.43- Missing program error visibility; add msg! with codes and parse logs.44- Slot lag misread due to provider differences; monitor per provider.45- No runbook -> slow response; write and link.4647## Quality Bar / Validation48- Dashboards live with top metrics; alerts tested.49- Each alert has runbook and owner.50- On-call rotation known; contact methods tested.5152## Output Format53Provide monitoring plan: metrics list, dashboards needed, alert thresholds with runbooks, and ownership map.5455## Examples56- Simple: Dashboard for tx success + RPC latency; alert to Slack on error spike; runbook to switch RPC.57- Complex: Full stack including program log parsing, pool depth alerts, holder concentration tracking; PagerDuty rotation with quarterly drills.
SanctifiedOps