Grafana Monitoring

Grafana Monitoring

Access to [YOUR_PRODUCT]'s Grafana Cloud instance for infrastructure monitoring, alerting, and observability.

Instance: https://.grafana.net

Tool

Call the native MCP tool grafana__<tool-name> directly (server=grafana, tool=). Parameters go in a JSON object — examples below use shorthand key=value.

Datasources

Services

Key Tools

Logs (Loki)

See references/loki-logql.md for complete LogQL guide.

# Errors from web server
grafana__query_loki_logs datasourceUid="grafanacloud-logs" logQL='{service_name="web-server", level=~"ERROR|Error|Fatal"}'

# Search for exception
grafana__query_loki_logs datasourceUid="grafanacloud-logs" logQL='{service_name="web-server"} |= "NullReferenceException"'

# Job failures
grafana__query_loki_logs datasourceUid="grafanacloud-logs" logQL='{service_name=~"jobs-heavy|jobs-light"} |= "failed"'

# Check log volume before querying
grafana__query_loki_stats datasourceUid="grafanacloud-logs" logQL='{service_name="web-server"}'

# Discover labels
grafana__list_loki_label_names datasourceUid="grafanacloud-logs"
grafana__list_loki_label_values datasourceUid="grafanacloud-logs" labelName="service_name"

# Auto-detected log patterns
grafana__query_loki_patterns datasourceUid="grafanacloud-logs"

Metrics (Prometheus)

See references/prometheus-metrics.md for available metrics and PromQL patterns.

# HTTP request latency (p95)
grafana__query_prometheus datasourceUid="grafanacloud-prom" expr='histogram_quantile(0.95, sum by (le) (rate(http_server_request_duration_seconds_bucket{job="web-server"}[1h])))' startRfc3339="now-1h" endRfc3339="now" stepSeconds=300

# Error rate percentage
grafana__query_prometheus datasourceUid="grafanacloud-prom" expr='100 * sum(rate(http_server_request_duration_seconds_count{http_response_status_code=~"5.."}[5m])) / sum(rate(http_server_request_duration_seconds_count[5m]))' startRfc3339="now-1h" endRfc3339="now" stepSeconds=60

# Chat latency by search mode
grafana__query_prometheus datasourceUid="grafanacloud-prom" expr='sum by (search_mode) (rate(gen_ai_client_operation_duration_seconds_sum[1h]))' startRfc3339="now-1h" endRfc3339="now" stepSeconds=300

# Discover metrics
grafana__list_prometheus_metric_names datasourceUid="grafanacloud-prom"
grafana__list_prometheus_metric_metadata datasourceUid="grafanacloud-prom"
grafana__list_prometheus_label_names datasourceUid="grafanacloud-prom"

Aggregated metrics note: Grafana Cloud aggregates high-cardinality metrics. Always use aggregation functions (sum, rate, histogram_quantile), never query raw series.

Traces (Tempo)

See references/tempo-traceql.md for TraceQL syntax and trace analysis.

# Find slow requests (>10s)
grafana__tempo_traceql-search datasourceUid="grafanacloud-traces" query='{span:duration > 10s}'

# Average duration by search mode
grafana__tempo_traceql-metrics-instant datasourceUid="grafanacloud-traces" query='{span.search.mode != ""} | avg_over_time(duration) by (span.search.mode)'

# Error traces by service
grafana__tempo_traceql-metrics-range datasourceUid="grafanacloud-traces" query='{span:status = error} | rate() by (resource.service.name)'

# Get specific trace
grafana__tempo_get-trace traceId="abc123"

# Discover trace attributes
grafana__tempo_get-attribute-names datasourceUid="grafanacloud-traces"
grafana__tempo_get-attribute-values datasourceUid="grafanacloud-traces" attributeName="service.name"

Dashboards

See references/dashboards.md for full dashboard list and deeplink generation.

# Search dashboards
grafana__search_dashboards query="[YOUR_PRODUCT]"

# Get dashboard summary
grafana__get_dashboard_summary uid="cdiz9piuoa3gge"

# Get panel queries
grafana__get_dashboard_panel_queries dashboardUid="cdiz9piuoa3gge"

# Generate deeplink
grafana__generate_deeplink resourceType="dashboard" dashboardUid="cdiz9piuoa3gge"

Key dashboards:

• Agents GenAI Observability (cdiz9piuoa3gge) — LLM operations, tokens, costs
• Hangfire Jobs Monitoring (sc5pdf5) — Background job health
• [YOUR_PRODUCT] Web Server Test (040cee27-2c68-47e4-8e73-e69975e8cc4f) — API performance

Alerts

# Check firing alerts
grafana__list_alert_groups

# List all alert rules
grafana__list_alert_rules

# Get specific alert rule
grafana__get_alert_rule_by_uid uid="abc"

# Get specific alert group
grafana__get_alert_group groupId="abc"

Incidents

# List incidents
grafana__list_incidents

# Get incident details
grafana__get_incident incidentId="abc"

# Create incident
grafana__create_incident title="API latency spike" severity="minor"

# Add investigation notes
grafana__add_activity_to_incident incidentId="abc" body="Root cause: connection pool exhaustion"

Error Investigation (Sift)

Automated error pattern detection — use before manual log trawling.

# Auto-detect error patterns
grafana__find_error_pattern_logs datasourceUid="grafanacloud-logs"

# Find slow requests
grafana__find_slow_requests datasourceUid="grafanacloud-logs"

# List/get sift investigations
grafana__list_sift_investigations
grafana__get_sift_analysis analysisId="abc"

Annotations

# Mark deployment on dashboards
grafana__create_annotation text="Deployed v2.1" dashboardUID="cdiz9piuoa3gge"

# Search annotations (correlate events)
grafana__get_annotations

OnCall

# Who is on call now
grafana__get_current_oncall_users

# List schedules
grafana__list_oncall_schedules

Datasources

# List all datasources (discover UIDs)
grafana__list_datasources

Investigation Workflows

"Why is the API slow?"

1. query_prometheus — HTTP latency p95
2. tempo_traceql-search — find slow traces (>5s)
3. query_loki_logs — check logs around slow request timestamps

"Are there errors in production?"

1. find_error_pattern_logs — auto-detect error patterns (Sift)
2. query_loki_logs — error logs by service
3. tempo_traceql-search — error traces
4. list_alert_groups — check firing alerts

"How is search performance?"

1. tempo_traceql-metrics-instant — avg duration by search mode
2. query_prometheus — LLM operation latency by search mode
3. query_loki_logs — specific search errors

"What is the indexing status of a repository?"

1. query_loki_logs — {service_name="indexer"} — recent activity
2. query_loki_logs — {service_name="indexer"} |= "<repo-name>" — filter by repo
3. query_loki_logs — {service_name="indexer"} |= "error" — check for indexing errors
4. query_prometheus — indexer metrics (if available)

General investigation flow

1. list_datasources — get datasource UIDs (if unsure)
2. find_error_pattern_logs / find_slow_requests — Sift auto-detection
3. query_loki_logs — search for errors
4. query_prometheus — check related metrics
5. tempo_traceql-search — trace analysis
6. search_dashboards — find relevant dashboards
7. create_incident — if critical

Time Ranges

• Prometheus: Use relative (now-1h, now-24h) or RFC3339
• Tempo: RFC3339 format, max 25h for metrics queries
• Always verify current date before time-sensitive queries

Output Format

Present results concisely for Telegram:

• Status indicators for alerts/incidents
• Key metric values with time context
• Log excerpts (max 5 lines, most relevant)
• Link to dashboard when available

Related Skills

• posthog-analytics — product analytics (user behavior, funnels)
• metrics-briefing — CEO-level metric interpretation