Monitoring Grafana

Monitoring Grafana

Discovery

What discovery provides:

• datasources: All datasources with uid, name, type, isDefault (grouped by type)
• dashboards: Available dashboards with uid, title, folderTitle, tags
• folders: Dashboard folders with uid, title
• alerts: Alert rules with total count, state breakdown, and firing alerts list
• incidents: Active incidents with id, title, status
• prometheus: Per-datasource label discovery (labelNames, keyLabels for job/namespace/instance)
• loki: Per-datasource label discovery (labelNames, keyLabels for app/namespace/pod)
• capabilities: siftAvailable - whether Sift plugin is installed (for find_error_pattern_logs)
• hints.loki: Query limits, forbidden regex patterns, and safe alternatives

Why run discovery:

• Get datasourceUid required for all Prometheus/Loki queries
• Know available labels before building queries
• Check if Sift plugin is available (determines if find_error_pattern_logs will work)
• See firing alerts that need immediate attention

Smart Query Rules

// ✅ CORRECT - always specify at least one label await query_loki_logs({ datasourceUid, logql: '{app="backend"}', limit: 50 }); await query_loki_stats({ datasourceUid, logql: '{namespace="prod"}' });

Use discovery (`list_loki_label_values`) to find valid label values if you don't know them.
</critical>

| Bad ❌ | Good ✅ | Why |
|--------|---------|-----|
| `expr: 'node_cpu_total'` | `expr: 'avg(rate(node_cpu_total[5m]))'` | Aggregation reduces 100+ series to 1 |
| `queryType: "range"` for current value | `queryType: "instant"` | Single point vs 60+ points |
| `limit: 1000` | `limit: 20-50` first | Sample before fetching all |
| `startTime: "now-24h"` | `startTime: "now-15m"` | Recent data sufficient for troubleshooting |
| `query_loki_logs()` first | `find_error_pattern_logs()` first | Patterns before raw logs |

**Query progression**: Aggregated summary → Filter to problem area → Small sample → (rarely) Full data

## Tools

**Datasources:** `list_datasources(type?)`, `get_datasource(uid?, name?)`

**Dashboards:** `search_dashboards(query?)`, `get_dashboard_by_uid(uid)`, `get_dashboard_summary(uid)`, `get_dashboard_panel_queries(uid, panelId)`, `get_dashboard_property(uid, jsonPath)`, `update_dashboard(dashboard?, uid?, operations?, folderUid?, message?, overwrite?)`, `run_panel_query(dashboardUid, panelIds, queryIndex?, start?, end?, variables?)`, `search_folders(query?)`, `create_folder(title, uid?, parentUid?)`

**Prometheus:** `query_prometheus(datasourceUid, expr, startTime, queryType?, endTime?, stepSeconds?)`, `query_prometheus_histogram(datasourceUid, metric, percentile, labels?, rateInterval?, startTime?, endTime?, stepSeconds?)`, `list_prometheus_metric_names(datasourceUid)`, `list_prometheus_label_names(datasourceUid, match?)`, `list_prometheus_label_values(datasourceUid, labelName, match?)`, `list_prometheus_metric_metadata(datasourceUid)`

**Loki:** `query_loki_logs(datasourceUid, logql, startRfc3339?, endRfc3339?, limit?, direction?)`, `query_loki_stats(datasourceUid, logql, startRfc3339?, endRfc3339?)`, `list_loki_label_names(datasourceUid, startRfc3339?, endRfc3339?)`, `list_loki_label_values(datasourceUid, labelName, startRfc3339?, endRfc3339?)`, `find_error_pattern_logs(name, labels, start?, end?)`, `find_slow_requests(name, labels, start?, end?)`, `query_loki_patterns(datasourceUid, logql, startRfc3339?, endRfc3339?, step?)`

**Alerting:** `alerting_manage_rules(operation: "list"|"get"|"create"|"update"|"delete", rule_uid?, ...)`, `list_alert_groups(...)`, `alerting_manage_routing(operation: "get_notification_policies"|"get_contact_points"|..., ...)`

**Annotations:** `get_annotations(...)`, `create_annotation(...)`, `update_annotation(...)`, `get_annotation_tags(...)`

**Incidents:** `list_incidents(...)`, `get_incident(incidentId)`, `create_incident(...)`, `add_activity_to_incident(incidentId, body, eventTime?)`, `list_oncall_schedules(...)`, `get_current_oncall_users(scheduleId)`

**Search:** `search_logs(datasourceUid, pattern, table?, start?, end?, limit?)`

**Elasticsearch:** `query_elasticsearch(datasourceUid, query, start?, end?, limit?)`

**CloudWatch:** `query_cloudwatch(datasourceUid, namespace, metricName, dimensions?, statistic?, period?, start?, end?, region, accountId?)`

**ClickHouse:** `query_clickhouse(datasourceUid, query, start?, end?, variables?, limit?)`, `list_clickhouse_tables(datasourceUid)`

**Pyroscope:** `list_pyroscope_label_names(data_source_uid, ...)`, `list_pyroscope_label_values(...)`, `list_pyroscope_profile_types(data_source_uid)`, `fetch_pyroscope_profile(...)`

**Rendering:** `get_panel_image(dashboardUid, panelId?, width?, height?, timeRange?, variables?, theme?, scale?)`

**Navigation:** `generate_deeplink(resourceType, dashboardUid?, datasourceUid?, panelId?, queryParams?, timeRange?)`

**Admin:** `list_teams(query?)`, `list_users_by_org()`, `list_all_roles(delegatableOnly?)`, `get_role_details(roleUID)`

**Examples:** `get_query_examples(datasourceType)`

**Sift:** `start_sift_investigation(...)`, `get_sift_investigation_status(...)`, `list_sift_investigations(...)`

**Asserts:** `get_assertions(startTime, endTime, entityType, entityName, ...)`

## Quick Patterns

**Discover:**
```typescript
const datasources = await list_datasources({ type: "prometheus" });
const labels = await list_prometheus_label_names({ datasourceUid: "ds-uid" });
const apps = await list_loki_label_values({ datasourceUid: "ds-uid", labelName: "app" });

LogQL: {label="value"} |= "filter" - Examples: {app="backend"}, {app="backend"} |= "error", {app="backend"} | json | status >= 500

import type { QueryLokiLogsResult, QueryLokiStatsResult } from "./_skills/connections/grafana/monitoring-grafana/schemas";

// Stats query (label matchers only, no line filters)
const stats = await query_loki_stats({
  datasourceUid, logql: '{app="backend", namespace="prod"}'
}) as QueryLokiStatsResult;

// Log query with line filter
const response = await query_loki_logs({
  datasourceUid, logql: '{app="backend"} |= "error"', limit: 100, direction: "backward"
}) as QueryLokiLogsResult;
const logs = response.data;
// Strip JSON quotes bug: timestamp.replace(/^"|"$/g, '') then new Date(Number(ts) / 1e6)

// ✅ CORRECT - regex alternation '{namespace="prod"} |~ "error|ERROR|Exception|FATAL"'

// ✅ BEST - broad query + TypeScript filtering const response = await query_loki_logs({...logql: '{namespace="prod"}', limit: 100}) as QueryLokiLogsResult; const errors = response.data.filter(l => /error|exception|fatal/i.test(l.line));

</critical>

<critical>
**RATE LIMITS:** Max 2-3 concurrent queries (429 error otherwise). **Prefer single broad query + TypeScript filtering** over parallel queries.
```typescript
const response = await query_loki_logs({datasourceUid, logql: '{namespace="prod"}', limit: 100}) as QueryLokiLogsResult;
const errors = response.data.filter(l => /error/i.test(l.line)); // Filter in code

// ✅ CORRECT - label matchers only await query_loki_stats({datasourceUid, logql: '{app="backend", namespace="prod"}', ...});

// Then filter with query_loki_logs if needed const logs = await query_loki_logs({datasourceUid, logql: '{app="backend"} |= "error"', limit: 100});

</critical>

<critical>
**SIFT PLUGIN:** `find_error_pattern_logs()` / `find_slow_requests()` may return 404. Always use try-catch with manual fallback:
```typescript
try {
  patterns = await find_error_pattern_logs({name: "error-analysis", labels: {namespace: "prod"}});
} catch (e: any) {
  if (e.message?.includes('404')) {
    const patternMap = new Map<string, number>();
    logs.forEach(log => {
      const p = log.line.substring(0, 100).trim();
      patternMap.set(p, (patternMap.get(p) || 0) + 1);
    });
    patterns = Array.from(patternMap.entries()).sort((a, b) => b[1] - a[1]).slice(0, 10);
  } else throw e;
}

Best Workflow: Single broad query → TypeScript categorization → pattern detection

const response = await query_loki_logs({datasourceUid, logql: '{namespace="prod"}', limit: 100, direction: 'backward'}) as QueryLokiLogsResult;
const logs = response.data;
const errors = logs.filter(l => /error/i.test(l.line));
const patternMap = new Map<string, number>();
errors.forEach(log => {
  const p = log.line.substring(0, 100).trim();
  patternMap.set(p, (patternMap.get(p) || 0) + 1);
});
const top5 = Array.from(patternMap.entries()).sort((a, b) => b[1] - a[1]).slice(0, 5);

PromQL: avg(rate(...[5m])) by (label), topk(5, ...), histogram_quantile(0.95, ...)

// Current state (instant query)
const p95 = await query_prometheus({datasourceUid, expr: 'histogram_quantile(0.95, rate(http_duration_bucket[5m]))', startTime: "now", queryType: "instant"});
// Trend (short range)
const trend = await query_prometheus({datasourceUid, expr: 'avg(rate(http_requests[5m])) by (status)', startTime: "now-15m", queryType: "range", stepSeconds: 60});

Dashboards: search_dashboards({query}), get_dashboard_summary({uid}), get_dashboard_panel_queries({uid, panelId})

Alerts: alerting_manage_rules({operation: "list"}), alerting_manage_rules({operation: "get", rule_uid: uid})

Workflows

Errors: find_error_pattern_logs({name: "...", labels: {...}}) → query_loki_logs(limit:20) for top pattern Performance: query_prometheus(instant) P95 → if high, query_prometheus(range, now-15m) trend Alerts: alerting_manage_rules({operation: "list"}) filter state:"firing" → query_prometheus(now-15m) for alert metric

Time & Errors

Time: Prometheus: now-1h, Loki: RFC3339 2024-01-07T00:00:00Z

Common Errors:

• Empty selector {} → Loki rejects logql: '{}' with parse error: unexpected }, expecting IDENTIFIER. Always include at least one label matcher (e.g., {app="backend"}, {namespace="prod"}). Use list_loki_label_values() to discover available labels.
• Timestamp quotes → Strip: ts.replace(/^"|"$/g, '') then new Date(Number(ts)/1e6)
• Escape sequences \w → Use [A-Za-z0-9_]
• 429 rate limit → Single query + code filtering
• 404 Plugin → Use manual fallback
• query_loki_stats with filters → Stats only supports label matchers, no line filters (|=, |~, | json)

Critical Anti-Patterns

Raw metrics first: Use aggregation (sum, avg, rate, topk) before fetching series Range for current state: Use queryType: "instant" not "range" Long time windows: Start now-15m not now-1h or now-24h High log limits: Use limit: 20-50 not 500+ for exploration Logs before patterns: Use find_error_pattern_logs() or query_loki_stats() first Complex regex: Use [A-Za-z0-9_] not \w, or filter in TypeScript 3+ parallel queries: Rate limit 2-3 max, prefer single query + code filter Assuming Sift: Always try-catch with manual fallback (404 error)

Output Format

Present results as a structured report:

Monitoring Grafana Report
═════════════════════════
Resources discovered: [count]

Resource       Status    Key Metric    Issues
──────────────────────────────────────────────
[name]         [ok/warn] [value]       [findings]

Summary: [total] resources | [ok] healthy | [warn] warnings | [crit] critical
Action Items: [list of prioritized findings]

Target ≤50 lines of output. Use tables for multi-resource comparisons.

Anti-Hallucination Rules

1. NEVER assume resource names — always discover via CLI/API in Phase 1 before referencing in Phase 2.
2. NEVER fabricate metric names or dimensions — verify against the service documentation or --help output.
3. NEVER mix CLI commands between service versions — confirm which version/API you are targeting.
4. ALWAYS use the discovery → verify → analyze chain — every resource referenced must have been discovered first.
5. ALWAYS handle empty results gracefully — an empty response is valid data, not an error to retry.

Counter-Rationalizations