Grafana Lens
You have full native Grafana access — query data, create dashboards, set alerts, receive alert notifications, annotate events, explore datasources, push custom data, and deliver visualizations inline. Works with ANY data in Grafana, not just agent metrics.
Musts
- Always call
grafana_explore_datasources first when you need a datasource UID — never guess UIDs
- Always call
grafana_search before creating a dashboard — avoid duplicates
- Always call
grafana_get_dashboard before grafana_share_dashboard — you need exact panel IDs
- Always call
grafana_get_dashboard before grafana_update_dashboard — you need panel IDs and current structure
- Prefer
grafana_query for direct answers over creating dashboards — "what's my cost?" needs a number, not a URL
- Prefer
grafana_query over grafana_create_dashboard + grafana_share_dashboard for simple data questions — a number is faster than a chart
- Use
grafana_query_logs for log searches — LogQL for logs, PromQL for metrics, TraceQL for traces. Never use grafana_query for Loki datasources
- Use
grafana_query_traces for trace searches — TraceQL for traces, PromQL for metrics, LogQL for logs. Never use grafana_query or grafana_query_logs for Tempo datasources
- All tools work with ANY Prometheus datasource — not just
openclaw_lens_* metrics
- When you see "GRAFANA ALERTS" in prompt context, investigate immediately with
grafana_check_alerts — use the suggestedInvestigation field to go directly to querying (it provides the tool, query, and datasource)
- Run
grafana_check_alerts with action setup once before alert notifications can reach the agent — this creates the webhook contact point
- Push data before querying or dashboarding it — data is pushed via OTLP and available immediately
- Prefer
grafana_explain_metric for "what is this metric?" questions over manual grafana_query — it returns current value, trend, stats, and metadata in one call
- Use
queryNames from push response for PromQL queries — don't guess metric names (counters get _total suffix)
- Use
openclaw_ext_ prefix for custom metrics — grafana_push_metrics auto-prepends it if missing
- Follow statistics-first discipline for log investigation — always run count/rate LogQL before reading individual entries. Use
grafana_query_logs with metric-over-logs queries (count_over_time, rate, topk) before switching to raw log entries
- Silence alerts during investigation — use
grafana_check_alerts with action silence to prevent repeat notifications while investigating
- Use
list_rules for complete alert health — grafana_check_alerts with action list_rules returns all rules with live eval state (normal/firing/pending/nodata/error), health, and lastEvaluation — no need to cross-reference with list action
- Use
dashboardUid + panelId to re-run panel queries — don't manually extract PromQL/LogQL from get_dashboard output. Both grafana_query and grafana_query_logs accept these params to auto-resolve the panel's query expression and datasource. The tool handles template variable replacement and datasource routing automatically
- Confirm with user before deleting dashboards or alert rules —
grafana_update_dashboard with operation delete and grafana_check_alerts with action delete_rule are permanent and cannot be undone
Quick Decision Tree
- "What is [metric]?" / "Why did it spike?" →
grafana_explain_metric
- "What's the current value of X?" / complex PromQL →
grafana_query
- "Find error logs" / "Search logs for..." →
grafana_query_logs
- "Find slow traces" / "Show trace for session X" / "Debug distributed spans" →
grafana_query_traces
- "Debug this session" / "Why did it fail?" / "What went wrong?" →
grafana_query_traces (search error/slow) → grafana_query_traces (get → follow correlationHint) → grafana_query_logs → grafana_query → grafana_annotate
- "Show me a chart" / "Visualize..." →
grafana_search → grafana_get_dashboard → grafana_share_dashboard
- "Create a dashboard for..." →
grafana_search (check duplicates) → grafana_create_dashboard
- "Add a panel to my dashboard" →
grafana_get_dashboard → grafana_update_dashboard
- "Delete this dashboard" →
grafana_update_dashboard with operation delete (confirm with user first)
- "Alert me when..." →
grafana_check_alerts (setup) → grafana_create_alert
- "List my alert rules" / "What alerts do I have?" →
grafana_check_alerts with action list_rules
- "Delete alert rule X" →
grafana_check_alerts with action list_rules → delete_rule with ruleUid
- "Track my [custom data]" / "Record my [past data]" →
grafana_push_metrics (with optional timestamp for historical data, auto-registers, returns queryNames) → grafana_query with queryNames
- "What data sources do I have?" →
grafana_explore_datasources
- "What metrics are available?" →
grafana_list_metrics
- "Set up monitoring" / "Monitor my agent" / "What dashboards should I have?" →
grafana_search (check existing) → grafana_create_dashboard with llm-command-center → follow suggestedNext chain through remaining templates
- "GenAI observability" / "OTel gen_ai metrics" / "Standard AI monitoring" →
grafana_create_dashboard with genai-observability template
- "What happened in session X?" / "Debug this session" →
grafana_create_dashboard with session-explorer template → paste session ID
- "Show me LLM traces" / "Show agent logs" →
grafana_create_dashboard with llm-command-center template (Loki + Tempo)
- "How much am I spending?" / "Cost analysis" →
grafana_create_dashboard with cost-intelligence template
- "Which tools are slow?" / "Tool errors" →
grafana_create_dashboard with tool-performance template
- "Queue health" / "Webhook issues" / "Stuck sessions" →
grafana_create_dashboard with sre-operations template
- "System health check" / "Status report" / "Review all dashboards" →
grafana_explore_datasources → grafana_check_alerts (list + list_rules) → grafana_search → grafana_get_dashboard (audit=true for each) → summarize
- "Audit my dashboard" / "Which panels are broken?" →
grafana_get_dashboard (audit=true) → review auditSummary + per-panel health
- "Am I being attacked?" / "Security check" / "Security status" →
grafana_security_check
- "Set up security monitoring" →
grafana_check_alerts (setup) → grafana_create_dashboard (security-overview) → grafana_create_alert (webhook error burst, cost spike, tool loops, injection signals)
- "Investigate security alert" →
grafana_security_check → grafana_query_logs (correlate) → grafana_annotate (mark investigation) → grafana_check_alerts (silence)
- "Investigate this alert" / "Why is X broken?" / "Debug this issue" / "Triage" / "Root cause" →
grafana_investigate (multi-signal triage) → follow suggestedHypotheses.testWith for deep-dives
- "Is this metric normal?" / "Is there an anomaly?" →
grafana_explain_metric (returns anomaly z-score + seasonality vs 1d/7d ago for 24h period)
- "RED analysis" / "What's the error rate?" / "Service health" → RED Method queries (see sre-investigation.md §2)
- "Alert fatigue" / "Which alerts are noisy?" / "Alert health" →
grafana_check_alerts with action analyze — fatigue report
- "Postmortem" / "Incident summary" / "What happened?" →
grafana_investigate → 5-Phase methodology → postmortem template (see sre-investigation.md §9)
- "Compare before/after deployment" →
grafana_annotate (list, tags: ["deploy"]) → grafana_explain_metric (compareWith: "previous")
Tool Inventory
| Tool |
What It Does |
grafana_explore_datasources |
Discover configured datasources (UIDs, types, query routing) — tells you which tool + query language to use for each datasource |
grafana_list_metrics |
Discover available metrics or label values from a datasource. Use compact: true with metadata: true for minimal fields in multi-tool chains |
grafana_query |
Run PromQL instant/range queries — get numbers directly |
grafana_query_logs |
Run LogQL queries against Loki — search and filter logs |
grafana_query_traces |
Run TraceQL queries against Tempo — search traces or get full trace by ID |
grafana_create_dashboard |
Create dashboards from templates or custom JSON |
grafana_update_dashboard |
Add/remove/update panels, change dashboard metadata, or delete dashboard |
grafana_get_dashboard |
Get dashboard summary (panels, queries). Use compact: true for overview scans, audit: true to health-check all panels in one call |
grafana_search |
Search existing dashboards by title, tags, or starred status |
grafana_share_dashboard |
Render panel as image and deliver inline via messaging |
grafana_create_alert |
Create Grafana-native alert rules on any metric |
grafana_annotate |
Create or list annotations (events) on dashboards |
grafana_check_alerts |
Check, acknowledge, list/delete rules, silence/unsilence, or set up Grafana alert webhook notifications. Use compact: true with list_rules for minimal fields |
grafana_push_metrics |
Push custom data (calendar, git, fitness, finance) via OTLP |
grafana_explain_metric |
Get metric context: current value, trend, stats, metadata, drill-down queries — agent interprets |
grafana_security_check |
Run 6 parallel security checks and return threat-level assessment (green/yellow/red) — "Am I being attacked?" |
grafana_investigate |
Multi-signal investigation triage — gathers metrics, logs, traces, and context in parallel, generates hypotheses with specific tool+params for follow-up |
Tool Details
grafana_explore_datasources
When: First step when user mentions data, metrics, or monitoring. Gets datasource UIDs needed by grafana_query, grafana_query_logs, grafana_query_traces, grafana_list_metrics, grafana_create_alert, and grafana_explain_metric.
Params: None required.
Example: {}
Returns: List of datasources with uid, name, type, isDefault, plus routing hints: queryTool (which agent tool to use, e.g. "grafana_query", "grafana_query_logs", or "grafana_query_traces"), queryLanguage (e.g. "PromQL", "LogQL", "TraceQL"), and supported (boolean — whether an agent tool can query this datasource). Use queryTool to pick the right tool for each datasource.
grafana_list_metrics
When: User asks "what metrics are available?" or you need to discover metrics before querying or composing dashboards. Also when grouping metrics by function — metadata mode adds category to each openclaw_* metric. Use purpose when user asks about a specific concern (e.g., "performance metrics", "cost metrics").
Params: datasourceUid (required), prefix (filter by prefix), search (targeted discovery — server-side regex, only matching metrics returned), purpose ("performance" | "cost" | "reliability" | "capacity" — pre-filter by intent, composable with prefix and search), label (list label values instead), metadata (boolean — enriched results with type/help/category), compact (boolean — with metadata, returns only name/type/category, ~60% smaller).
Example names: { "datasourceUid": "prom1", "prefix": "openclaw_lens_" }
Example search: { "datasourceUid": "prom1", "search": "steps" }
Example purpose: { "datasourceUid": "prom1", "purpose": "performance", "metadata": true }
Example combined: { "datasourceUid": "prom1", "prefix": "openclaw_ext_", "search": "fitness" }
Example metadata: { "datasourceUid": "prom1", "metadata": true, "prefix": "openclaw_" }
Example compact: { "datasourceUid": "prom1", "metadata": true, "compact": true }
Returns names: { metrics: ["metric1", "metric2", ...] }. Truncated at 200.
Returns metadata: { metadataSource, categorySummary: { cost: 3, usage: 4, session: 5, ... }, metrics: [{ name, type, help, category?, source? }, ...] }. Use this before composing custom dashboards — type tells you counter vs gauge vs histogram, category groups openclaw_* metrics by function. Search also matches help text. Categories: cost, usage, session, queue, messaging, webhook, tools, agent, custom. categorySummary gives counts per category for quick overview (omitted when no openclaw_* metrics). Purpose maps: performance → session + tools, cost → cost + usage, reliability → webhook + messaging + agent, capacity → queue + session. metadataSource: "prometheus" when Prometheus metadata endpoint has data, "synthetic" when OTLP-only (metadata synthesized from known metric registry — histogram sub-metrics deduplicated, type/help from Grafana Lens definitions). On OTLP stacks, includes hint explaining why metadata is synthetic. source: "synthetic" on individual entries from the registry; source: "custom" on entries from the custom metrics store.
Returns compact: { metadataSource, categorySummary: {...}, metrics: [{ name, type, category? }, ...] }. Same as metadata but drops help, source, labelNames — use in multi-tool chains where you need metric names and types but not full descriptions.
Example label: { "datasourceUid": "prom1", "label": "job" }
Returns label: { label, count, totalCount, values: ["value1", "value2", ...] }. Truncated at 200.
grafana_query
When: User asks a data question that needs a direct answer, not a dashboard. Also for re-running an existing dashboard panel's query with different time ranges.
Params: datasourceUid, expr (PromQL), queryType (instant/range), start (range only, required), end (range only, default "now"), step (range only, optional — auto-calculated from time range if omitted, targeting ~300 datapoints), dashboardUid (optional — resolve query from panel), panelId (optional — use with dashboardUid).
Example instant: { "datasourceUid": "prom1", "expr": "sum(increase(openclaw_lens_cost_by_model_total[1d])) or vector(0)" }
Example range (auto-step): { "datasourceUid": "prom1", "expr": "rate(openclaw_tokens_total[5m])", "queryType": "range", "start": "now-30d" }
Example range (explicit step): { "datasourceUid": "prom1", "expr": "rate(openclaw_tokens_total[5m])", "queryType": "range", "start": "now-1h", "end": "now", "step": "60" }
Example panel re-run: { "dashboardUid": "openclaw-command-center", "panelId": 10, "queryType": "range", "start": "now-7d" }
Tip: start/end accept Unix seconds or relative expressions like "now-1h", "now-7d". For range queries, just set start — end defaults to "now" and step is auto-calculated. Override step only when you need specific resolution.
Tip (panel re-run): Set dashboardUid + panelId to re-run a panel's query without manually extracting PromQL. The tool auto-resolves expr and datasourceUid from the panel definition. Template variables are replaced with wildcards. You can still override expr or datasourceUid explicitly if needed. Get panel IDs from grafana_get_dashboard.
Returns instant: { metrics: [{ metric: {...}, value: "1.23", timestamp: "...", healthContext?: { status, thresholds, description, direction } }], datasourceUid, resultCount, warnings?, hint? } — healthContext is included for well-known openclaw_lens_* gauge metrics, providing SRE-grade health assessment: status ("healthy"/"warning"/"critical"), thresholds (warning/critical values), description (what the metric means), direction ("higher_is_worse"/"lower_is_worse"). Omitted for unknown metrics. Capped at 50 results; when exceeded includes truncated: true, totalResults, and truncationHint advising to narrow the query.
Returns range: { series: [{ metric: {...}, values: [{ time, value }...] }], datasourceUid, resultCount, warnings?, hint? } — truncated to 20 points per series and 50 series max. When series are truncated includes truncated: true, totalSeries, and truncationHint. When step is auto-calculated, includes step: { value: "288s", display: "5m", auto: true }.
Returns (panel re-run): Includes resolvedFrom: "panel", panelTitle, panelType, templateVarsReplaced alongside normal query results. If the panel uses a Loki datasource, returns an error directing you to use grafana_query_logs instead.
Returns (warnings): When Prometheus flags a non-fatal issue (e.g., rate() on a gauge), warnings: [{ cause, suggestion, example? }] is included. Example: rate() on a gauge → cause says "rate() applied to 'metric' which appears to be a gauge", suggestion says "use delta() or deriv() instead", example shows the corrected query.
Returns (hint): When the query returns zero results, hint: { cause, suggestion } explains why (metric may not exist, label filters may not match) and suggests using grafana_list_metrics to verify.
Returns (error with guidance): On query failure, includes guidance: { cause, suggestion, example? } alongside the raw error. Pattern-matched for common PromQL mistakes: unclosed parenthesis, missing range selector, timeout, auth failure, rate on gauge, etc. Omitted when the error is unrecognized.
Tip (chaining): Both instant and range responses include datasourceUid — pass it directly to grafana_create_alert or other tools without re-calling grafana_explore_datasources. This enables zero-friction query→alert chains.
grafana_query_logs
When: User asks about logs, errors, or needs to investigate issues by searching log data. Also for session debugging, OTel log investigation, and re-running existing log panel queries.
Params: datasourceUid, expr (LogQL), queryType (instant/range, default range), start/end (default now-1h/now), step (metric queries only), limit (default 100), direction (backward/forward), lineLimit (max chars per log line, default 500, max 2000), extractFields (boolean, default false — extract structured OTel attributes into a clean fields object), dashboardUid (optional — resolve query from panel), panelId (optional — use with dashboardUid).
Example log search: { "datasourceUid": "loki1", "expr": "{job=\"api\"} |= \"error\"" }
Example with filters: { "datasourceUid": "loki1", "expr": "{job=\"api\"} |~ \"timeout|refused\"", "limit": 50, "direction": "forward" }
Example full stack traces: { "datasourceUid": "loki1", "expr": "{job=\"api\"} |= \"Exception\"", "lineLimit": 2000 }
Example session debugging: { "datasourceUid": "loki1", "expr": "{service_name=\"openclaw\"} | json | component=\"lifecycle\"", "extractFields": true }
Example metric query: { "datasourceUid": "loki1", "expr": "rate({job=\"api\"}[5m])", "queryType": "range", "start": "now-6h", "end": "now", "step": "60" }
Example panel re-run: { "dashboardUid": "openclaw-command-center", "panelId": 18, "start": "now-24h", "extractFields": true }
Returns streams: { entries: [{ labels: {...}, timestamp: "...", line: "..." }], datasourceUid, totalEntries, truncated } — capped at 100 entries, lines at 500 chars (set lineLimit: 2000 for full stack traces).
Returns streams (extractFields): { entries: [{ labels: {...cleaned...}, timestamp: "...", line: "...", fields: { component, event_name, session_id, trace_id, model, duration_s, ... } }], datasourceUid } — infrastructure noise labels removed, openclaw_ prefix stripped from field keys, numeric values auto-converted. Also parses JSON log bodies if present.
Returns streams (traceCorrelation): When extractFields: true and entries contain trace_id, includes traceCorrelation: { traceIds: [...], tool: "grafana_query_traces", tip } — up to 5 unique trace IDs ready for grafana_query_traces with queryType: "get".
Returns metric: Same shape as grafana_query range/instant results (matrix capped at 50 series, vector capped at 50 results — includes datasourceUid, truncated, totalSeries/totalResults, and truncationHint when exceeded).
Returns (panel re-run): Includes resolvedFrom: "panel", panelTitle, panelType, templateVarsReplaced alongside normal results. If the panel uses a Prometheus datasource, returns an error directing you to use grafana_query instead.
Returns (error with guidance): On query failure, includes guidance: { cause, suggestion, example? } alongside the raw error. Pattern-matched for common LogQL mistakes: bare text without stream selector, empty {}, unclosed braces, missing label matchers, auth failure, timeout. Omitted when the error is unrecognized.
Tip: LogQL: {label="value"} selects streams, |= substring filter, |~ regex, != exclude. Metric wrappers: rate(), count_over_time(), bytes_rate(). Use extractFields: true when investigating OTel/lifecycle logs — it surfaces trace_id, session_id, event_name, model, and other attributes as first-class fields instead of buried in raw labels.
Tip (panel re-run): Same as grafana_query — set dashboardUid + panelId to auto-resolve LogQL and datasource. The tool routes Prometheus panels to grafana_query with a helpful error.
grafana_query_traces
When: User asks about traces, distributed tracing, slow spans, session trace hierarchies, or needs to debug request flows across services.
Params: datasourceUid, query (TraceQL expression or trace ID), queryType (search/get, default search), start/end (default now-1h/now), limit (default 20, max 50), minDuration/maxDuration (e.g., "1s", "10s"), dashboardUid (optional — resolve query from panel), panelId (optional — use with dashboardUid).
Example search: { "datasourceUid": "tempo1", "query": "{ resource.service.name = \"openclaw\" }" }
Example search slow: { "datasourceUid": "tempo1", "query": "{ resource.service.name = \"openclaw\" }", "minDuration": "5s" }
Example search with time: { "datasourceUid": "tempo1", "query": "{ span.gen_ai.system = \"anthropic\" }", "start": "now-24h", "limit": 50 }
Example get: { "datasourceUid": "tempo1", "query": "abc123def456789...", "queryType": "get" }
Example panel re-run: { "dashboardUid": "openclaw-session-explorer", "panelId": 12, "start": "now-24h" }
Returns search: { traces: [{ traceId, rootServiceName, rootTraceName, startTime, durationMs, spanCount? }], datasourceUid, totalTraces, truncated?, correlationHint? } — capped at 50 traces. When exceeded includes truncated: true and truncationHint. When traces are found, includes correlationHint: { logQuery, tool, tip } with a ready-to-use LogQL expression for grafana_query_logs.
Returns get: { traceId, spans: [{ traceId, spanId, parentSpanId?, operationName, serviceName, startTime, durationMs, status, kind?, attributes: {...} }], datasourceUid, totalSpans, truncated? } — flattened OTLP spans with resolved attributes (string/number/boolean). Capped at 200 spans. Sorted by start time (earliest first).
Returns (panel re-run): Includes resolvedFrom: "panel", panelTitle, panelType, templateVarsReplaced alongside normal results. If the panel uses a Prometheus or Loki datasource, returns an error directing you to use the correct tool.
Returns (error with guidance): On query failure, includes guidance: { cause, suggestion, example? } alongside the raw error. Pattern-matched for common TraceQL mistakes: syntax errors, invalid attributes, auth failure, timeout, not-found, invalid trace ID. Omitted when the error is unrecognized.
Returns (no results): When search returns zero traces, includes hint: { cause, suggestion } suggesting to broaden the query or check the datasource.
Tip: TraceQL: { } matches all traces, resource.service.name for service filter, span.http.status_code for HTTP spans, name for operation name, duration for span duration, status for error/ok filtering. Use minDuration/maxDuration to find performance outliers. Trace-to-Log: search and get results include correlationHint.logQuery — pass it directly to grafana_query_logs to find correlated logs. Log-to-Trace: grafana_query_logs results (with extractFields: true) include traceCorrelation.traceIds — pass any ID to grafana_query_traces with queryType: "get".
Tip (panel re-run): Same as grafana_query — set dashboardUid + panelId to auto-resolve TraceQL and datasource. The tool routes Prometheus/Loki panels to the correct tool with a helpful error.
grafana_create_dashboard
When: User wants a persistent dashboard for ongoing monitoring.
Params: template or dashboard (custom JSON) — one required. Optional: title (overrides template default), folderUid (target folder), overwrite (default true).
Returns: { uid, url, status, message, suggestedNext?: [{ template, reason }], validation?: DashboardValidation }. For template-based dashboards, suggestedNext lists complementary templates to deploy next. For custom JSON dashboards, validation dry-runs each panel's PromQL and reports per-panel health — check validation.panelsError for broken queries.
Choose the right template (3-tier SRE drill-down hierarchy):
Tier 1 → System: Start here for overall health.
Tier 2 → Session: Click a session from Tier 1 to investigate.
Tier 3 → Deep Dive: Cost, tool, or SRE details.
| Template |
Tier |
Domain |
Variables |
Use When |
llm-command-center |
Tier 1 |
System overview |
$prometheus, $loki, $tempo, $provider, $model, $channel |
Golden signals, session table with click-to-drill-down, cost, cache, live feeds |
session-explorer |
Tier 2 |
Session debug |
$prometheus, $loki, $tempo, $session (textbox) |
Per-session trace hierarchy, LLM calls, tool calls, conversation flow |
cost-intelligence |
Tier 3a |
Cost analysis |
$prometheus, $loki, $provider, $model |
Spending trends, model attribution, cache savings, per-session cost table |
tool-performance |
Tier 3b |
Tool analytics |
$prometheus, $loki, $tempo, $tool |
Tool leaderboard, latency ranking, error rates, tool traces |
sre-operations |
Tier 3c |
SRE operations |
$prometheus, $loki |
Queue health, webhooks, stuck sessions, tool loops |
genai-observability |
— |
OTel gen_ai standard |
$prometheus, $loki, $tempo, $model, $provider |
Industry-standard AI monitoring: token analytics, LLM performance, traces, logs, cache efficiency. Works with any gen_ai data. |
node-exporter |
— |
System/DevOps |
$datasource, $instance |
Server CPU, memory, disk, network |
http-service |
— |
Web/DevOps |
$datasource, $job |
HTTP request rate, errors, latency (RED signals) |
metric-explorer |
— |
Any domain |
$datasource, $metric |
Deep-dive into any single metric from a dropdown |
multi-kpi |
— |
Any domain |
$datasource, $metric1..$metric4 |
4-metric KPI overview (business, fitness, finance, IoT) |
weekly-review |
— |
Any domain |
$datasource, $metric1, $metric2 |
Weekly overview of 2 external metrics with trends + all openclaw_ext_* table |
All AI templates have Loki log-to-trace correlation via Tempo + stable UIDs for cross-dashboard navigation.
Example AI health: { "template": "llm-command-center", "title": "My AI Dashboard" }
Example session debug: { "template": "session-explorer", "title": "Session Debug" }
Example cost analysis: { "template": "cost-intelligence", "title": "My AI Costs" }
Example tool analytics: { "template": "tool-performance", "title": "Tool Health" }
Example SRE ops: { "template": "sre-operations", "title": "SRE Health" }
Example GenAI observability: { "template": "genai-observability", "title": "GenAI Observability" }
Example system: { "template": "node-exporter", "title": "Server Health" }
Example generic: { "template": "metric-explorer", "title": "Explore My Data" }
Example multi-KPI: { "template": "multi-kpi", "title": "Business KPIs" }
Example weekly review: { "template": "weekly-review", "title": "My Weekly Review" }
Example custom with validation: { "dashboard": { "title": "Model Comparison", "panels": [{ "id": 1, "title": "Cost by Model", "type": "timeseries", "targets": [{ "refId": "A", "expr": "sum by (model) (rate(openclaw_lens_cost_by_token_type[1h]))", "datasource": { "uid": "prometheus" } }] }] } }
Custom dashboard validation (returned only for dashboard param, not templates):
validation: { panelsTotal: 3, panelsValid: 1, panelsNoData: 1, panelsError: 1, panelsSkipped: 0, details: [{ panelId: 1, title: "Cost by Model", status: "ok", queries: [{ refId: "A", expr: "...", valid: true, sampleValue: 0.42 }] }, { panelId: 2, title: "Latency", status: "nodata" }, { panelId: 3, title: "Bad Query", status: "error", error: "parse error at char 5" }] }
Panel statuses: ok (query returned data), nodata (valid query, no results — metric may not exist yet), error (PromQL syntax error or datasource issue), skipped (no datasource UID found). Dashboard is always created regardless — validation is informational.
grafana_update_dashboard
When: User wants to add a panel, remove a panel, change a query, update dashboard settings, or delete a dashboard.
Params: uid (required), operation (required: add_panel, remove_panel, update_panel, update_metadata, delete).
add_panel params: panel (object with title, type, targets). Auto-layouts below existing panels.
remove_panel / update_panel params: panelId (preferred) or panelTitle (case-insensitive substring fallback). updates (object) for update_panel.
update_metadata params: title, description, tags, time (e.g., { "from": "now-7d", "to": "now" }), refresh (e.g., "1m").
delete params: None besides uid — permanently removes the dashboard. Always confirm with user first.
Example add: { "uid": "abc123", "operation": "add_panel", "panel": { "title": "Error Rate", "type": "timeseries", "targets": [{ "refId": "A", "expr": "rate(errors_total[5m])", "datasource": { "uid": "prom1" } }] } }
Example add (no datasource): { "uid": "abc123", "operation": "add_panel", "panel": { "title": "Latency", "type": "timeseries", "targets": [{ "refId": "A", "expr": "histogram_quantile(0.99, rate(http_duration_bucket[5m]))" }] } } — validation skipped if no datasource UID found, panel still saved.
Example remove: { "uid": "abc123", "operation": "remove_panel", "panelId": 3 }
Example update panel: { "uid": "abc123", "operation": "update_panel", "panelId": 1, "updates": { "title": "New Title", "targets": [{ "refId": "A", "expr": "new_query" }] } }
Example update metadata: { "uid": "abc123", "operation": "update_metadata", "title": "My Dashboard v2", "time": { "from": "now-7d", "to": "now" }, "refresh": "5m" }
Example delete: { "uid": "abc123", "operation": "delete" }
Returns update: { status: "updated", uid, url, version, operation, panelCount, affectedPanel?: { id, title }, changedFields?: [...], queryValidation?: { validated, results, datasourceUid?, skippedReason? } }.
Returns queryValidation: For add_panel and update_panel (when targets change), PromQL queries are dry-run against Grafana. Each result: { refId, expr, valid: boolean, error?: string, sampleValue?: number }. Panel is always saved — validation is informational. If valid: false, check the error field for PromQL syntax issues. If skippedReason is set, no datasource UID was found — include datasource: { uid: "..." } on targets to enable validation.
Returns delete: { status: "deleted", uid, title, message }.
Tip: targets in update_panel replaces entirely — include all targets, not just changed ones. Include datasource.uid on targets for query validation feedback.
grafana_get_dashboard
When: Need to inspect a dashboard's panels — find panel IDs for sharing, verify structure, scan multiple dashboards for an overview, or audit which panels are returning data.
Params: uid (required). Optional: compact (boolean, default false) — return panel titles and types only, no queries or metadata (~70% smaller). audit (boolean, default false) — dry-run each panel's query and add health status.
Example (full): { "uid": "abc123" }
Example (compact overview): { "uid": "abc123", "compact": true }
Example (audit): { "uid": "abc123", "audit": true }
Returns (full): { uid, title, description?, url, tags, time?, refresh?, panelCount, panels: [{ id, title, type, queries: [{ refId, expr }] }], folderUid, created?, updated? }.
Returns (compact): { uid, title, url, tags, panelCount, panels: [{ id, title, type }] }.
Returns (audit): Same as full, plus each panel gets health: { status: "ok"|"nodata"|"error"|"skipped", error?, sampleValue? } and the response includes auditSummary: { ok, nodata, error, skipped }. Resolves template variable datasources ($prometheus, $loki) and replaces expression template vars with wildcards.
Tip: Use audit: true when the user asks "which panels are broken?" or "audit my dashboard" — it replaces N separate grafana_query calls with one tool call. Use compact: true for lightweight overview scans. Omit both when you need query details (before update or share).
grafana_search
When: User mentions a dashboard by name, before creating one (check duplicates), or for reporting/audit workflows.
Params: query (required). Optional: tags (array — filter by tags), starred (boolean — only starred), sort ("alpha-asc"/"alpha-desc"), limit (number, default 100), enrich (boolean — add updatedAt + panelCount per result, default false).
Example: { "query": "cost" }
Example with tags: { "query": "", "tags": ["production"] }
Example starred: { "query": "", "starred": true, "limit": 10 }
Example enriched: { "query": "", "enrich": true }
Returns: { count, enriched, dashboards: [{ uid, title, url, tags, folderTitle?, folderUid?, updatedAt?, panelCount? }] }. folderTitle/folderUid always included when dashboard is in a folder. updatedAt (ISO 8601) and panelCount only present when enrich: true — enables staleness detection and reporting without per-dashboard get_dashboard calls.
Tip: Use enrich: true for reporting workflows ("which dashboards are stale?", "give me a summary of all dashboards"). Skip enrichment for simple lookups. After finding a dashboard, use grafana_get_dashboard to inspect panels, grafana_share_dashboard to render a chart, or grafana_update_dashboard to modify it.
grafana_share_dashboard
When: User says "show me" or "send me" a chart/dashboard.
Params: dashboardUid, panelId (required). Optional: from (default "now-6h"), to (default "now"), width (default 1000), height (default 500), theme ("light"/"dark", default "dark").
Example: { "dashboardUid": "abc123", "panelId": 2, "from": "now-6h", "to": "now" }
Returns: Image rendered inline (tier 1), or snapshot URL (tier 2), or deep link (tier 3). Always delivers something. Includes deliveryTier ("image" | "snapshot" | "link"), rendererAvailable (boolean — false when Image Renderer plugin is missing), renderFailureReason (why image rendering failed), and remediation (how to fix it). Tier 3 also includes snapshotFailureReason.
Tip: Use grafana_get_dashboard first to find panel IDs. If rendererAvailable is false, tell the user to install the grafana-image-renderer plugin.
grafana_create_alert
When: User wants notifications when a metric crosses a threshold.
Params: title, datasourceUid, expr (PromQL), threshold (all required). Optional: evaluation ("instant"/"rate"/"increase", default "instant"), evaluationWindow (default "5m", used with rate/increase), condition (gt/lt/gte/lte, default gt), for (duration, default 5m), folderUid, labels (e.g., { "severity": "warning" }), annotations (e.g., { "summary": "Cost too high" }), noDataState (NoData/Alerting/OK, default NoData).
IMPORTANT: For counter metrics (*_total), always use evaluation: "rate" (per-second rate) or evaluation: "increase" (total change over window). Raw counter values always increase and will immediately breach any threshold. Use "instant" (default) only for gauges.
Example gauge alert: { "title": "High Cost Alert", "datasourceUid": "prom1", "expr": "openclaw_lens_daily_cost_usd", "threshold": 5, "condition": "gt" }
Example rate alert: { "title": "High Error Rate", "datasourceUid": "prom1", "expr": "openclaw_lens_webhook_error_total", "threshold": 0.1, "evaluation": "rate" }
Example increase alert: { "title": "Token Burst", "datasourceUid": "prom1", "expr": "openclaw_lens_tokens_total", "threshold": 10000, "evaluation": "increase", "evaluationWindow": "1h" }
Returns: { uid, title, status: "created", datasourceUid, url, evaluation?: { mode, window, evaluatedExpr }, metricValidation: { valid, error?, sampleValue? }, message }. The datasourceUid echoes back which datasource the rule targets (verify correctness). metricValidation dry-runs the expression before creation — valid: true + sampleValue confirms data exists; valid: false + error warns of typos/missing metrics. Alert is always created regardless (metric may not have data yet). When evaluation is "rate" or "increase", validation runs the wrapped expression.
Note: Auto-creates a "Grafana Lens Alerts" folder if no folderUid is specified.
grafana_annotate
When: User deploys, changes config, or wants to mark an event for correlation.
Params: action ("create" default, or "list").
Create params: text (required), tags, dashboardUid, panelId, time (epoch ms or relative like "now-2h", default now), timeEnd (epoch ms or relative).
List params: from, to (epoch ms or relative like "now-7d", "now-24h", "now"), tags, limit (default 20).
Time formats: All time params accept epoch ms (e.g., 1700000000000) OR Grafana-style relative strings ("now", "now-1h", "now-7d", "now-30m"). Prefer relative strings — they're simpler and avoid arithmetic errors.
Example create: { "text": "Deployed v2.1.0", "tags": ["deploy", "production"] }
Example create past: { "text": "Incident started", "time": "now-2h", "timeEnd": "now-30m", "tags": ["incident"] }
Example list recent: { "action": "list", "from": "now-7d", "to": "now", "tags": ["deploy"] }
Example list: { "action": "list", "tags": ["deploy"], "limit": 10 }
Returns create: { status: "created", id, message, time, comparisonHint: { beforeWindow: { from, to }, afterWindow: { from, to }, suggestion } }. The comparisonHint provides ready-to-use ISO 8601 time ranges (30-min windows) for before/after comparison via grafana_query — no manual time math needed. For region annotations (with timeEnd), afterWindow starts at timeEnd.
Returns list: { annotations: [{ id, text, tags, time, timeEnd?, dashboardUID?, panelId? }] }.
grafana_check_alerts
When: Prompt context shows "GRAFANA ALERTS", need to manage alert rules (list/delete), set up the alert webhook, silence alerts during investigation, or acknowledge an investigated alert.
Params: action ("list" default, "acknowledge", "list_rules", "delete_rule", "silence", "unsilence", "setup").
List params: None — returns all pending (unacknowledged) alerts. Instances capped at 5 per alert.
Acknowledge params: alertId (required) — marks an alert as investigated.
List rules params: compact (boolean, default false —
…(truncated)
1---2name: grafana-lens3description: Grafana tools for data visualization, monitoring, alerting, security, and SRE investigation. Use grafana_query, grafana_query_logs, grafana_query_traces, grafana_create_dashboard, grafana_update_dashboard, grafana_create_alert, grafana_share_dashboard, grafana_annotate, grafana_explore_datasources, grafana_list_metrics, grafana_search, grafana_get_dashboard, grafana_check_alerts, grafana_push_metrics, grafana_explain_metric, grafana_security_check, and grafana_investigate. Trigger when asked about metrics, dashboards, monitoring, alerts, costs, token usage, data visualization, PromQL, Prometheus, LogQL, Loki, log queries, error logs, log search, TraceQL, Tempo, traces, distributed tracing, span search, find slow traces, debug session traces, annotations, deployments, sharing charts, investigating alert notifications, pushing custom data (calendar, git, fitness, finance) to Grafana for visualization, pushing historical data, backfilling metrics, recording past data with timestamps, modifying dashboards, adding4---56# Grafana Lens78You have full native Grafana access — query data, create dashboards, set alerts, receive alert notifications, annotate events, explore datasources, push custom data, and deliver visualizations inline. Works with ANY data in Grafana, not just agent metrics.910## Musts1112- **Always call `grafana_explore_datasources` first** when you need a datasource UID — never guess UIDs13- **Always call `grafana_search` before creating a dashboard** — avoid duplicates14- **Always call `grafana_get_dashboard` before `grafana_share_dashboard`** — you need exact panel IDs15- **Always call `grafana_get_dashboard` before `grafana_update_dashboard`** — you need panel IDs and current structure16- **Prefer `grafana_query` for direct answers** over creating dashboards — "what's my cost?" needs a number, not a URL17- **Prefer `grafana_query` over `grafana_create_dashboard` + `grafana_share_dashboard`** for simple data questions — a number is faster than a chart18- **Use `grafana_query_logs` for log searches** — LogQL for logs, PromQL for metrics, TraceQL for traces. Never use `grafana_query` for Loki datasources19- **Use `grafana_query_traces` for trace searches** — TraceQL for traces, PromQL for metrics, LogQL for logs. Never use `grafana_query` or `grafana_query_logs` for Tempo datasources20- **All tools work with ANY Prometheus datasource** — not just `openclaw_lens_*` metrics21- **When you see "GRAFANA ALERTS" in prompt context**, investigate immediately with `grafana_check_alerts` — use the `suggestedInvestigation` field to go directly to querying (it provides the tool, query, and datasource)22- **Run `grafana_check_alerts` with action `setup` once** before alert notifications can reach the agent — this creates the webhook contact point23- **Push data before querying or dashboarding it** — data is pushed via OTLP and available immediately24- **Prefer `grafana_explain_metric` for "what is this metric?" questions** over manual `grafana_query` — it returns current value, trend, stats, and metadata in one call25- **Use `queryNames` from push response for PromQL queries** — don't guess metric names (counters get `_total` suffix)26- **Use `openclaw_ext_` prefix for custom metrics** — `grafana_push_metrics` auto-prepends it if missing27- **Follow statistics-first discipline for log investigation** — always run count/rate LogQL before reading individual entries. Use `grafana_query_logs` with metric-over-logs queries (`count_over_time`, `rate`, `topk`) before switching to raw log entries28- **Silence alerts during investigation** — use `grafana_check_alerts` with action `silence` to prevent repeat notifications while investigating29- **Use `list_rules` for complete alert health** — `grafana_check_alerts` with action `list_rules` returns all rules with live eval state (normal/firing/pending/nodata/error), health, and lastEvaluation — no need to cross-reference with `list` action30- **Use `dashboardUid` + `panelId` to re-run panel queries** — don't manually extract PromQL/LogQL from `get_dashboard` output. Both `grafana_query` and `grafana_query_logs` accept these params to auto-resolve the panel's query expression and datasource. The tool handles template variable replacement and datasource routing automatically31- **Confirm with user before deleting dashboards or alert rules** — `grafana_update_dashboard` with operation `delete` and `grafana_check_alerts` with action `delete_rule` are permanent and cannot be undone3233## Quick Decision Tree3435- "What is [metric]?" / "Why did it spike?" → `grafana_explain_metric`36- "What's the current value of X?" / complex PromQL → `grafana_query`37- "Find error logs" / "Search logs for..." → `grafana_query_logs`38- "Find slow traces" / "Show trace for session X" / "Debug distributed spans" → `grafana_query_traces`39- "Debug this session" / "Why did it fail?" / "What went wrong?" → `grafana_query_traces` (search error/slow) → `grafana_query_traces` (get → follow `correlationHint`) → `grafana_query_logs` → `grafana_query` → `grafana_annotate`40- "Show me a chart" / "Visualize..." → `grafana_search` → `grafana_get_dashboard` → `grafana_share_dashboard`41- "Create a dashboard for..." → `grafana_search` (check duplicates) → `grafana_create_dashboard`42- "Add a panel to my dashboard" → `grafana_get_dashboard` → `grafana_update_dashboard`43- "Delete this dashboard" → `grafana_update_dashboard` with operation `delete` (confirm with user first)44- "Alert me when..." → `grafana_check_alerts` (setup) → `grafana_create_alert`45- "List my alert rules" / "What alerts do I have?" → `grafana_check_alerts` with action `list_rules`46- "Delete alert rule X" → `grafana_check_alerts` with action `list_rules` → `delete_rule` with `ruleUid`47- "Track my [custom data]" / "Record my [past data]" → `grafana_push_metrics` (with optional `timestamp` for historical data, auto-registers, returns `queryNames`) → `grafana_query` with `queryNames`48- "What data sources do I have?" → `grafana_explore_datasources`49- "What metrics are available?" → `grafana_list_metrics`50- "Set up monitoring" / "Monitor my agent" / "What dashboards should I have?" → `grafana_search` (check existing) → `grafana_create_dashboard` with `llm-command-center` → follow `suggestedNext` chain through remaining templates51- "GenAI observability" / "OTel gen_ai metrics" / "Standard AI monitoring" → `grafana_create_dashboard` with `genai-observability` template52- "What happened in session X?" / "Debug this session" → `grafana_create_dashboard` with `session-explorer` template → paste session ID53- "Show me LLM traces" / "Show agent logs" → `grafana_create_dashboard` with `llm-command-center` template (Loki + Tempo)54- "How much am I spending?" / "Cost analysis" → `grafana_create_dashboard` with `cost-intelligence` template55- "Which tools are slow?" / "Tool errors" → `grafana_create_dashboard` with `tool-performance` template56- "Queue health" / "Webhook issues" / "Stuck sessions" → `grafana_create_dashboard` with `sre-operations` template57- "System health check" / "Status report" / "Review all dashboards" → `grafana_explore_datasources` → `grafana_check_alerts` (list + list_rules) → `grafana_search` → `grafana_get_dashboard` (audit=true for each) → summarize58- "Audit my dashboard" / "Which panels are broken?" → `grafana_get_dashboard` (audit=true) → review `auditSummary` + per-panel `health`59- "Am I being attacked?" / "Security check" / "Security status" → `grafana_security_check`60- "Set up security monitoring" → `grafana_check_alerts` (setup) → `grafana_create_dashboard` (`security-overview`) → `grafana_create_alert` (webhook error burst, cost spike, tool loops, injection signals)61- "Investigate security alert" → `grafana_security_check` → `grafana_query_logs` (correlate) → `grafana_annotate` (mark investigation) → `grafana_check_alerts` (silence)62- "Investigate this alert" / "Why is X broken?" / "Debug this issue" / "Triage" / "Root cause" → `grafana_investigate` (multi-signal triage) → follow `suggestedHypotheses.testWith` for deep-dives63- "Is this metric normal?" / "Is there an anomaly?" → `grafana_explain_metric` (returns `anomaly` z-score + `seasonality` vs 1d/7d ago for 24h period)64- "RED analysis" / "What's the error rate?" / "Service health" → RED Method queries (see sre-investigation.md §2)65- "Alert fatigue" / "Which alerts are noisy?" / "Alert health" → `grafana_check_alerts` with action `analyze` — fatigue report66- "Postmortem" / "Incident summary" / "What happened?" → `grafana_investigate` → 5-Phase methodology → postmortem template (see sre-investigation.md §9)67- "Compare before/after deployment" → `grafana_annotate` (list, tags: ["deploy"]) → `grafana_explain_metric` (compareWith: "previous")6869## Tool Inventory7071| Tool | What It Does |72|------|-------------|73| `grafana_explore_datasources` | Discover configured datasources (UIDs, types, query routing) — tells you which tool + query language to use for each datasource |74| `grafana_list_metrics` | Discover available metrics or label values from a datasource. Use `compact: true` with `metadata: true` for minimal fields in multi-tool chains |75| `grafana_query` | Run PromQL instant/range queries — get numbers directly |76| `grafana_query_logs` | Run LogQL queries against Loki — search and filter logs |77| `grafana_query_traces` | Run TraceQL queries against Tempo — search traces or get full trace by ID |78| `grafana_create_dashboard` | Create dashboards from templates or custom JSON |79| `grafana_update_dashboard` | Add/remove/update panels, change dashboard metadata, or delete dashboard |80| `grafana_get_dashboard` | Get dashboard summary (panels, queries). Use `compact: true` for overview scans, `audit: true` to health-check all panels in one call |81| `grafana_search` | Search existing dashboards by title, tags, or starred status |82| `grafana_share_dashboard` | Render panel as image and deliver inline via messaging |83| `grafana_create_alert` | Create Grafana-native alert rules on any metric |84| `grafana_annotate` | Create or list annotations (events) on dashboards |85| `grafana_check_alerts` | Check, acknowledge, list/delete rules, silence/unsilence, or set up Grafana alert webhook notifications. Use `compact: true` with `list_rules` for minimal fields |86| `grafana_push_metrics` | Push custom data (calendar, git, fitness, finance) via OTLP |87| `grafana_explain_metric` | Get metric context: current value, trend, stats, metadata, drill-down queries — agent interprets |88| `grafana_security_check` | Run 6 parallel security checks and return threat-level assessment (green/yellow/red) — "Am I being attacked?" |89| `grafana_investigate` | Multi-signal investigation triage — gathers metrics, logs, traces, and context in parallel, generates hypotheses with specific tool+params for follow-up |9091## Tool Details9293### `grafana_explore_datasources`94**When**: First step when user mentions data, metrics, or monitoring. Gets datasource UIDs needed by `grafana_query`, `grafana_query_logs`, `grafana_query_traces`, `grafana_list_metrics`, `grafana_create_alert`, and `grafana_explain_metric`.95**Params**: None required.96**Example**: `{}`97**Returns**: List of datasources with `uid`, `name`, `type`, `isDefault`, plus routing hints: `queryTool` (which agent tool to use, e.g. `"grafana_query"`, `"grafana_query_logs"`, or `"grafana_query_traces"`), `queryLanguage` (e.g. `"PromQL"`, `"LogQL"`, `"TraceQL"`), and `supported` (boolean — whether an agent tool can query this datasource). Use `queryTool` to pick the right tool for each datasource.9899### `grafana_list_metrics`100**When**: User asks "what metrics are available?" or you need to discover metrics before querying or composing dashboards. Also when grouping metrics by function — metadata mode adds `category` to each `openclaw_*` metric. Use `purpose` when user asks about a specific concern (e.g., "performance metrics", "cost metrics").101**Params**: `datasourceUid` (required), `prefix` (filter by prefix), `search` (targeted discovery — server-side regex, only matching metrics returned), `purpose` (`"performance"` | `"cost"` | `"reliability"` | `"capacity"` — pre-filter by intent, composable with prefix and search), `label` (list label values instead), `metadata` (boolean — enriched results with type/help/category), `compact` (boolean — with metadata, returns only name/type/category, ~60% smaller).102**Example names**: `{ "datasourceUid": "prom1", "prefix": "openclaw_lens_" }`103**Example search**: `{ "datasourceUid": "prom1", "search": "steps" }`104**Example purpose**: `{ "datasourceUid": "prom1", "purpose": "performance", "metadata": true }`105**Example combined**: `{ "datasourceUid": "prom1", "prefix": "openclaw_ext_", "search": "fitness" }`106**Example metadata**: `{ "datasourceUid": "prom1", "metadata": true, "prefix": "openclaw_" }`107**Example compact**: `{ "datasourceUid": "prom1", "metadata": true, "compact": true }`108**Returns names**: `{ metrics: ["metric1", "metric2", ...] }`. Truncated at 200.109**Returns metadata**: `{ metadataSource, categorySummary: { cost: 3, usage: 4, session: 5, ... }, metrics: [{ name, type, help, category?, source? }, ...] }`. Use this before composing custom dashboards — type tells you counter vs gauge vs histogram, category groups `openclaw_*` metrics by function. Search also matches help text. Categories: `cost`, `usage`, `session`, `queue`, `messaging`, `webhook`, `tools`, `agent`, `custom`. `categorySummary` gives counts per category for quick overview (omitted when no `openclaw_*` metrics). Purpose maps: `performance` → session + tools, `cost` → cost + usage, `reliability` → webhook + messaging + agent, `capacity` → queue + session. `metadataSource`: `"prometheus"` when Prometheus metadata endpoint has data, `"synthetic"` when OTLP-only (metadata synthesized from known metric registry — histogram sub-metrics deduplicated, type/help from Grafana Lens definitions). On OTLP stacks, includes `hint` explaining why metadata is synthetic. `source: "synthetic"` on individual entries from the registry; `source: "custom"` on entries from the custom metrics store.110**Returns compact**: `{ metadataSource, categorySummary: {...}, metrics: [{ name, type, category? }, ...] }`. Same as metadata but drops `help`, `source`, `labelNames` — use in multi-tool chains where you need metric names and types but not full descriptions.111**Example label**: `{ "datasourceUid": "prom1", "label": "job" }`112**Returns label**: `{ label, count, totalCount, values: ["value1", "value2", ...] }`. Truncated at 200.113114### `grafana_query`115**When**: User asks a data question that needs a direct answer, not a dashboard. Also for re-running an existing dashboard panel's query with different time ranges.116**Params**: `datasourceUid`, `expr` (PromQL), `queryType` (`instant`/`range`), `start` (range only, required), `end` (range only, default `"now"`), `step` (range only, optional — auto-calculated from time range if omitted, targeting ~300 datapoints), `dashboardUid` (optional — resolve query from panel), `panelId` (optional — use with `dashboardUid`).117**Example instant**: `{ "datasourceUid": "prom1", "expr": "sum(increase(openclaw_lens_cost_by_model_total[1d])) or vector(0)" }`118**Example range (auto-step)**: `{ "datasourceUid": "prom1", "expr": "rate(openclaw_tokens_total[5m])", "queryType": "range", "start": "now-30d" }`119**Example range (explicit step)**: `{ "datasourceUid": "prom1", "expr": "rate(openclaw_tokens_total[5m])", "queryType": "range", "start": "now-1h", "end": "now", "step": "60" }`120**Example panel re-run**: `{ "dashboardUid": "openclaw-command-center", "panelId": 10, "queryType": "range", "start": "now-7d" }`121**Tip**: `start`/`end` accept Unix seconds or relative expressions like `"now-1h"`, `"now-7d"`. For range queries, just set `start` — `end` defaults to `"now"` and `step` is auto-calculated. Override `step` only when you need specific resolution.122**Tip (panel re-run)**: Set `dashboardUid` + `panelId` to re-run a panel's query without manually extracting PromQL. The tool auto-resolves `expr` and `datasourceUid` from the panel definition. Template variables are replaced with wildcards. You can still override `expr` or `datasourceUid` explicitly if needed. Get panel IDs from `grafana_get_dashboard`.123**Returns instant**: `{ metrics: [{ metric: {...}, value: "1.23", timestamp: "...", healthContext?: { status, thresholds, description, direction } }], datasourceUid, resultCount, warnings?, hint? }` — `healthContext` is included for well-known `openclaw_lens_*` gauge metrics, providing SRE-grade health assessment: `status` ("healthy"/"warning"/"critical"), `thresholds` (warning/critical values), `description` (what the metric means), `direction` ("higher_is_worse"/"lower_is_worse"). Omitted for unknown metrics. Capped at 50 results; when exceeded includes `truncated: true`, `totalResults`, and `truncationHint` advising to narrow the query.124**Returns range**: `{ series: [{ metric: {...}, values: [{ time, value }...] }], datasourceUid, resultCount, warnings?, hint? }` — truncated to 20 points per series and 50 series max. When series are truncated includes `truncated: true`, `totalSeries`, and `truncationHint`. When step is auto-calculated, includes `step: { value: "288s", display: "5m", auto: true }`.125**Returns (panel re-run)**: Includes `resolvedFrom: "panel"`, `panelTitle`, `panelType`, `templateVarsReplaced` alongside normal query results. If the panel uses a Loki datasource, returns an error directing you to use `grafana_query_logs` instead.126**Returns (warnings)**: When Prometheus flags a non-fatal issue (e.g., `rate()` on a gauge), `warnings: [{ cause, suggestion, example? }]` is included. Example: `rate()` on a gauge → cause says "rate() applied to 'metric' which appears to be a gauge", suggestion says "use delta() or deriv() instead", example shows the corrected query.127**Returns (hint)**: When the query returns zero results, `hint: { cause, suggestion }` explains why (metric may not exist, label filters may not match) and suggests using `grafana_list_metrics` to verify.128**Returns (error with guidance)**: On query failure, includes `guidance: { cause, suggestion, example? }` alongside the raw error. Pattern-matched for common PromQL mistakes: unclosed parenthesis, missing range selector, timeout, auth failure, rate on gauge, etc. Omitted when the error is unrecognized.129**Tip (chaining)**: Both instant and range responses include `datasourceUid` — pass it directly to `grafana_create_alert` or other tools without re-calling `grafana_explore_datasources`. This enables zero-friction query→alert chains.130131### `grafana_query_logs`132**When**: User asks about logs, errors, or needs to investigate issues by searching log data. Also for session debugging, OTel log investigation, and re-running existing log panel queries.133**Params**: `datasourceUid`, `expr` (LogQL), `queryType` (`instant`/`range`, default `range`), `start`/`end` (default `now-1h`/`now`), `step` (metric queries only), `limit` (default 100), `direction` (`backward`/`forward`), `lineLimit` (max chars per log line, default 500, max 2000), `extractFields` (boolean, default false — extract structured OTel attributes into a clean `fields` object), `dashboardUid` (optional — resolve query from panel), `panelId` (optional — use with `dashboardUid`).134**Example log search**: `{ "datasourceUid": "loki1", "expr": "{job=\"api\"} |= \"error\"" }`135**Example with filters**: `{ "datasourceUid": "loki1", "expr": "{job=\"api\"} |~ \"timeout|refused\"", "limit": 50, "direction": "forward" }`136**Example full stack traces**: `{ "datasourceUid": "loki1", "expr": "{job=\"api\"} |= \"Exception\"", "lineLimit": 2000 }`137**Example session debugging**: `{ "datasourceUid": "loki1", "expr": "{service_name=\"openclaw\"} | json | component=\"lifecycle\"", "extractFields": true }`138**Example metric query**: `{ "datasourceUid": "loki1", "expr": "rate({job=\"api\"}[5m])", "queryType": "range", "start": "now-6h", "end": "now", "step": "60" }`139**Example panel re-run**: `{ "dashboardUid": "openclaw-command-center", "panelId": 18, "start": "now-24h", "extractFields": true }`140**Returns streams**: `{ entries: [{ labels: {...}, timestamp: "...", line: "..." }], datasourceUid, totalEntries, truncated }` — capped at 100 entries, lines at 500 chars (set `lineLimit: 2000` for full stack traces).141**Returns streams (extractFields)**: `{ entries: [{ labels: {...cleaned...}, timestamp: "...", line: "...", fields: { component, event_name, session_id, trace_id, model, duration_s, ... } }], datasourceUid }` — infrastructure noise labels removed, `openclaw_` prefix stripped from field keys, numeric values auto-converted. Also parses JSON log bodies if present.142**Returns streams (traceCorrelation)**: When `extractFields: true` and entries contain `trace_id`, includes `traceCorrelation: { traceIds: [...], tool: "grafana_query_traces", tip }` — up to 5 unique trace IDs ready for `grafana_query_traces` with `queryType: "get"`.143**Returns metric**: Same shape as `grafana_query` range/instant results (matrix capped at 50 series, vector capped at 50 results — includes `datasourceUid`, `truncated`, `totalSeries`/`totalResults`, and `truncationHint` when exceeded).144**Returns (panel re-run)**: Includes `resolvedFrom: "panel"`, `panelTitle`, `panelType`, `templateVarsReplaced` alongside normal results. If the panel uses a Prometheus datasource, returns an error directing you to use `grafana_query` instead.145**Returns (error with guidance)**: On query failure, includes `guidance: { cause, suggestion, example? }` alongside the raw error. Pattern-matched for common LogQL mistakes: bare text without stream selector, empty `{}`, unclosed braces, missing label matchers, auth failure, timeout. Omitted when the error is unrecognized.146**Tip**: LogQL: `{label="value"}` selects streams, `|=` substring filter, `|~` regex, `!=` exclude. Metric wrappers: `rate()`, `count_over_time()`, `bytes_rate()`. Use `extractFields: true` when investigating OTel/lifecycle logs — it surfaces `trace_id`, `session_id`, `event_name`, `model`, and other attributes as first-class fields instead of buried in raw labels.147**Tip (panel re-run)**: Same as `grafana_query` — set `dashboardUid` + `panelId` to auto-resolve LogQL and datasource. The tool routes Prometheus panels to `grafana_query` with a helpful error.148149### `grafana_query_traces`150**When**: User asks about traces, distributed tracing, slow spans, session trace hierarchies, or needs to debug request flows across services.151**Params**: `datasourceUid`, `query` (TraceQL expression or trace ID), `queryType` (`search`/`get`, default `search`), `start`/`end` (default `now-1h`/`now`), `limit` (default 20, max 50), `minDuration`/`maxDuration` (e.g., `"1s"`, `"10s"`), `dashboardUid` (optional — resolve query from panel), `panelId` (optional — use with `dashboardUid`).152**Example search**: `{ "datasourceUid": "tempo1", "query": "{ resource.service.name = \"openclaw\" }" }`153**Example search slow**: `{ "datasourceUid": "tempo1", "query": "{ resource.service.name = \"openclaw\" }", "minDuration": "5s" }`154**Example search with time**: `{ "datasourceUid": "tempo1", "query": "{ span.gen_ai.system = \"anthropic\" }", "start": "now-24h", "limit": 50 }`155**Example get**: `{ "datasourceUid": "tempo1", "query": "abc123def456789...", "queryType": "get" }`156**Example panel re-run**: `{ "dashboardUid": "openclaw-session-explorer", "panelId": 12, "start": "now-24h" }`157**Returns search**: `{ traces: [{ traceId, rootServiceName, rootTraceName, startTime, durationMs, spanCount? }], datasourceUid, totalTraces, truncated?, correlationHint? }` — capped at 50 traces. When exceeded includes `truncated: true` and `truncationHint`. When traces are found, includes `correlationHint: { logQuery, tool, tip }` with a ready-to-use LogQL expression for `grafana_query_logs`.158**Returns get**: `{ traceId, spans: [{ traceId, spanId, parentSpanId?, operationName, serviceName, startTime, durationMs, status, kind?, attributes: {...} }], datasourceUid, totalSpans, truncated? }` — flattened OTLP spans with resolved attributes (string/number/boolean). Capped at 200 spans. Sorted by start time (earliest first).159**Returns (panel re-run)**: Includes `resolvedFrom: "panel"`, `panelTitle`, `panelType`, `templateVarsReplaced` alongside normal results. If the panel uses a Prometheus or Loki datasource, returns an error directing you to use the correct tool.160**Returns (error with guidance)**: On query failure, includes `guidance: { cause, suggestion, example? }` alongside the raw error. Pattern-matched for common TraceQL mistakes: syntax errors, invalid attributes, auth failure, timeout, not-found, invalid trace ID. Omitted when the error is unrecognized.161**Returns (no results)**: When search returns zero traces, includes `hint: { cause, suggestion }` suggesting to broaden the query or check the datasource.162**Tip**: TraceQL: `{ }` matches all traces, `resource.service.name` for service filter, `span.http.status_code` for HTTP spans, `name` for operation name, `duration` for span duration, `status` for error/ok filtering. Use `minDuration`/`maxDuration` to find performance outliers. **Trace-to-Log**: search and get results include `correlationHint.logQuery` — pass it directly to `grafana_query_logs` to find correlated logs. **Log-to-Trace**: `grafana_query_logs` results (with `extractFields: true`) include `traceCorrelation.traceIds` — pass any ID to `grafana_query_traces` with `queryType: "get"`.163**Tip (panel re-run)**: Same as `grafana_query` — set `dashboardUid` + `panelId` to auto-resolve TraceQL and datasource. The tool routes Prometheus/Loki panels to the correct tool with a helpful error.164165### `grafana_create_dashboard`166**When**: User wants a persistent dashboard for ongoing monitoring.167**Params**: `template` or `dashboard` (custom JSON) — one required. Optional: `title` (overrides template default), `folderUid` (target folder), `overwrite` (default `true`).168**Returns**: `{ uid, url, status, message, suggestedNext?: [{ template, reason }], validation?: DashboardValidation }`. For template-based dashboards, `suggestedNext` lists complementary templates to deploy next. For custom JSON dashboards, `validation` dry-runs each panel's PromQL and reports per-panel health — check `validation.panelsError` for broken queries.169170**Choose the right template (3-tier SRE drill-down hierarchy):**171172**Tier 1 → System:** Start here for overall health.173**Tier 2 → Session:** Click a session from Tier 1 to investigate.174**Tier 3 → Deep Dive:** Cost, tool, or SRE details.175176| Template | Tier | Domain | Variables | Use When |177|----------|------|--------|-----------|----------|178| `llm-command-center` | **Tier 1** | System overview | `$prometheus`, `$loki`, `$tempo`, `$provider`, `$model`, `$channel` | Golden signals, session table with click-to-drill-down, cost, cache, live feeds |179| `session-explorer` | **Tier 2** | Session debug | `$prometheus`, `$loki`, `$tempo`, `$session` (textbox) | Per-session trace hierarchy, LLM calls, tool calls, conversation flow |180| `cost-intelligence` | Tier 3a | Cost analysis | `$prometheus`, `$loki`, `$provider`, `$model` | Spending trends, model attribution, cache savings, per-session cost table |181| `tool-performance` | Tier 3b | Tool analytics | `$prometheus`, `$loki`, `$tempo`, `$tool` | Tool leaderboard, latency ranking, error rates, tool traces |182| `sre-operations` | Tier 3c | SRE operations | `$prometheus`, `$loki` | Queue health, webhooks, stuck sessions, tool loops |183| `genai-observability` | — | **OTel gen_ai standard** | `$prometheus`, `$loki`, `$tempo`, `$model`, `$provider` | Industry-standard AI monitoring: token analytics, LLM performance, traces, logs, cache efficiency. Works with any gen_ai data. |184| `node-exporter` | — | System/DevOps | `$datasource`, `$instance` | Server CPU, memory, disk, network |185| `http-service` | — | Web/DevOps | `$datasource`, `$job` | HTTP request rate, errors, latency (RED signals) |186| `metric-explorer` | — | **Any domain** | `$datasource`, `$metric` | Deep-dive into any single metric from a dropdown |187| `multi-kpi` | — | **Any domain** | `$datasource`, `$metric1`..`$metric4` | 4-metric KPI overview (business, fitness, finance, IoT) |188| `weekly-review` | — | **Any domain** | `$datasource`, `$metric1`, `$metric2` | Weekly overview of 2 external metrics with trends + all openclaw_ext_* table |189190All AI templates have Loki log-to-trace correlation via Tempo + stable UIDs for cross-dashboard navigation.191192**Example AI health**: `{ "template": "llm-command-center", "title": "My AI Dashboard" }`193**Example session debug**: `{ "template": "session-explorer", "title": "Session Debug" }`194**Example cost analysis**: `{ "template": "cost-intelligence", "title": "My AI Costs" }`195**Example tool analytics**: `{ "template": "tool-performance", "title": "Tool Health" }`196**Example SRE ops**: `{ "template": "sre-operations", "title": "SRE Health" }`197**Example GenAI observability**: `{ "template": "genai-observability", "title": "GenAI Observability" }`198**Example system**: `{ "template": "node-exporter", "title": "Server Health" }`199**Example generic**: `{ "template": "metric-explorer", "title": "Explore My Data" }`200**Example multi-KPI**: `{ "template": "multi-kpi", "title": "Business KPIs" }`201**Example weekly review**: `{ "template": "weekly-review", "title": "My Weekly Review" }`202**Example custom with validation**: `{ "dashboard": { "title": "Model Comparison", "panels": [{ "id": 1, "title": "Cost by Model", "type": "timeseries", "targets": [{ "refId": "A", "expr": "sum by (model) (rate(openclaw_lens_cost_by_token_type[1h]))", "datasource": { "uid": "prometheus" } }] }] } }`203204**Custom dashboard validation** (returned only for `dashboard` param, not templates):205`validation: { panelsTotal: 3, panelsValid: 1, panelsNoData: 1, panelsError: 1, panelsSkipped: 0, details: [{ panelId: 1, title: "Cost by Model", status: "ok", queries: [{ refId: "A", expr: "...", valid: true, sampleValue: 0.42 }] }, { panelId: 2, title: "Latency", status: "nodata" }, { panelId: 3, title: "Bad Query", status: "error", error: "parse error at char 5" }] }`206Panel statuses: `ok` (query returned data), `nodata` (valid query, no results — metric may not exist yet), `error` (PromQL syntax error or datasource issue), `skipped` (no datasource UID found). Dashboard is always created regardless — validation is informational.207208### `grafana_update_dashboard`209**When**: User wants to add a panel, remove a panel, change a query, update dashboard settings, or delete a dashboard.210**Params**: `uid` (required), `operation` (required: `add_panel`, `remove_panel`, `update_panel`, `update_metadata`, `delete`).211**add_panel params**: `panel` (object with `title`, `type`, `targets`). Auto-layouts below existing panels.212**remove_panel / update_panel params**: `panelId` (preferred) or `panelTitle` (case-insensitive substring fallback). `updates` (object) for update_panel.213**update_metadata params**: `title`, `description`, `tags`, `time` (e.g., `{ "from": "now-7d", "to": "now" }`), `refresh` (e.g., `"1m"`).214**delete params**: None besides `uid` — permanently removes the dashboard. Always confirm with user first.215**Example add**: `{ "uid": "abc123", "operation": "add_panel", "panel": { "title": "Error Rate", "type": "timeseries", "targets": [{ "refId": "A", "expr": "rate(errors_total[5m])", "datasource": { "uid": "prom1" } }] } }`216**Example add (no datasource)**: `{ "uid": "abc123", "operation": "add_panel", "panel": { "title": "Latency", "type": "timeseries", "targets": [{ "refId": "A", "expr": "histogram_quantile(0.99, rate(http_duration_bucket[5m]))" }] } }` — validation skipped if no datasource UID found, panel still saved.217**Example remove**: `{ "uid": "abc123", "operation": "remove_panel", "panelId": 3 }`218**Example update panel**: `{ "uid": "abc123", "operation": "update_panel", "panelId": 1, "updates": { "title": "New Title", "targets": [{ "refId": "A", "expr": "new_query" }] } }`219**Example update metadata**: `{ "uid": "abc123", "operation": "update_metadata", "title": "My Dashboard v2", "time": { "from": "now-7d", "to": "now" }, "refresh": "5m" }`220**Example delete**: `{ "uid": "abc123", "operation": "delete" }`221**Returns update**: `{ status: "updated", uid, url, version, operation, panelCount, affectedPanel?: { id, title }, changedFields?: [...], queryValidation?: { validated, results, datasourceUid?, skippedReason? } }`.222**Returns queryValidation**: For `add_panel` and `update_panel` (when targets change), PromQL queries are dry-run against Grafana. Each result: `{ refId, expr, valid: boolean, error?: string, sampleValue?: number }`. Panel is always saved — validation is informational. If `valid: false`, check the `error` field for PromQL syntax issues. If `skippedReason` is set, no datasource UID was found — include `datasource: { uid: "..." }` on targets to enable validation.223**Returns delete**: `{ status: "deleted", uid, title, message }`.224**Tip**: `targets` in update_panel replaces entirely — include all targets, not just changed ones. Include `datasource.uid` on targets for query validation feedback.225226### `grafana_get_dashboard`227**When**: Need to inspect a dashboard's panels — find panel IDs for sharing, verify structure, scan multiple dashboards for an overview, or audit which panels are returning data.228**Params**: `uid` (required). Optional: `compact` (boolean, default `false`) — return panel titles and types only, no queries or metadata (~70% smaller). `audit` (boolean, default `false`) — dry-run each panel's query and add `health` status.229**Example (full)**: `{ "uid": "abc123" }`230**Example (compact overview)**: `{ "uid": "abc123", "compact": true }`231**Example (audit)**: `{ "uid": "abc123", "audit": true }`232**Returns (full)**: `{ uid, title, description?, url, tags, time?, refresh?, panelCount, panels: [{ id, title, type, queries: [{ refId, expr }] }], folderUid, created?, updated? }`.233**Returns (compact)**: `{ uid, title, url, tags, panelCount, panels: [{ id, title, type }] }`.234**Returns (audit)**: Same as full, plus each panel gets `health: { status: "ok"|"nodata"|"error"|"skipped", error?, sampleValue? }` and the response includes `auditSummary: { ok, nodata, error, skipped }`. Resolves template variable datasources (`$prometheus`, `$loki`) and replaces expression template vars with wildcards.235**Tip**: Use `audit: true` when the user asks "which panels are broken?" or "audit my dashboard" — it replaces N separate `grafana_query` calls with one tool call. Use `compact: true` for lightweight overview scans. Omit both when you need query details (before update or share).236237### `grafana_search`238**When**: User mentions a dashboard by name, before creating one (check duplicates), or for reporting/audit workflows.239**Params**: `query` (required). Optional: `tags` (array — filter by tags), `starred` (boolean — only starred), `sort` (`"alpha-asc"`/`"alpha-desc"`), `limit` (number, default 100), `enrich` (boolean — add `updatedAt` + `panelCount` per result, default false).240**Example**: `{ "query": "cost" }`241**Example with tags**: `{ "query": "", "tags": ["production"] }`242**Example starred**: `{ "query": "", "starred": true, "limit": 10 }`243**Example enriched**: `{ "query": "", "enrich": true }`244**Returns**: `{ count, enriched, dashboards: [{ uid, title, url, tags, folderTitle?, folderUid?, updatedAt?, panelCount? }] }`. `folderTitle`/`folderUid` always included when dashboard is in a folder. `updatedAt` (ISO 8601) and `panelCount` only present when `enrich: true` — enables staleness detection and reporting without per-dashboard `get_dashboard` calls.245**Tip**: Use `enrich: true` for reporting workflows ("which dashboards are stale?", "give me a summary of all dashboards"). Skip enrichment for simple lookups. After finding a dashboard, use `grafana_get_dashboard` to inspect panels, `grafana_share_dashboard` to render a chart, or `grafana_update_dashboard` to modify it.246247### `grafana_share_dashboard`248**When**: User says "show me" or "send me" a chart/dashboard.249**Params**: `dashboardUid`, `panelId` (required). Optional: `from` (default `"now-6h"`), `to` (default `"now"`), `width` (default `1000`), `height` (default `500`), `theme` (`"light"`/`"dark"`, default `"dark"`).250**Example**: `{ "dashboardUid": "abc123", "panelId": 2, "from": "now-6h", "to": "now" }`251**Returns**: Image rendered inline (tier 1), or snapshot URL (tier 2), or deep link (tier 3). Always delivers something. Includes `deliveryTier` (`"image"` | `"snapshot"` | `"link"`), `rendererAvailable` (boolean — false when Image Renderer plugin is missing), `renderFailureReason` (why image rendering failed), and `remediation` (how to fix it). Tier 3 also includes `snapshotFailureReason`.252**Tip**: Use `grafana_get_dashboard` first to find panel IDs. If `rendererAvailable` is false, tell the user to install the grafana-image-renderer plugin.253254### `grafana_create_alert`255**When**: User wants notifications when a metric crosses a threshold.256**Params**: `title`, `datasourceUid`, `expr` (PromQL), `threshold` (all required). Optional: `evaluation` (`"instant"`/`"rate"`/`"increase"`, default `"instant"`), `evaluationWindow` (default `"5m"`, used with `rate`/`increase`), `condition` (`gt`/`lt`/`gte`/`lte`, default `gt`), `for` (duration, default `5m`), `folderUid`, `labels` (e.g., `{ "severity": "warning" }`), `annotations` (e.g., `{ "summary": "Cost too high" }`), `noDataState` (`NoData`/`Alerting`/`OK`, default `NoData`).257**IMPORTANT**: For counter metrics (`*_total`), always use `evaluation: "rate"` (per-second rate) or `evaluation: "increase"` (total change over window). Raw counter values always increase and will immediately breach any threshold. Use `"instant"` (default) only for gauges.258**Example gauge alert**: `{ "title": "High Cost Alert", "datasourceUid": "prom1", "expr": "openclaw_lens_daily_cost_usd", "threshold": 5, "condition": "gt" }`259**Example rate alert**: `{ "title": "High Error Rate", "datasourceUid": "prom1", "expr": "openclaw_lens_webhook_error_total", "threshold": 0.1, "evaluation": "rate" }`260**Example increase alert**: `{ "title": "Token Burst", "datasourceUid": "prom1", "expr": "openclaw_lens_tokens_total", "threshold": 10000, "evaluation": "increase", "evaluationWindow": "1h" }`261**Returns**: `{ uid, title, status: "created", datasourceUid, url, evaluation?: { mode, window, evaluatedExpr }, metricValidation: { valid, error?, sampleValue? }, message }`. The `datasourceUid` echoes back which datasource the rule targets (verify correctness). `metricValidation` dry-runs the expression before creation — `valid: true` + `sampleValue` confirms data exists; `valid: false` + `error` warns of typos/missing metrics. Alert is always created regardless (metric may not have data yet). When `evaluation` is `"rate"` or `"increase"`, validation runs the wrapped expression.262**Note**: Auto-creates a "Grafana Lens Alerts" folder if no `folderUid` is specified.263264### `grafana_annotate`265**When**: User deploys, changes config, or wants to mark an event for correlation.266**Params**: `action` (`"create"` default, or `"list"`).267**Create params**: `text` (required), `tags`, `dashboardUid`, `panelId`, `time` (epoch ms or relative like `"now-2h"`, default now), `timeEnd` (epoch ms or relative).268**List params**: `from`, `to` (epoch ms or relative like `"now-7d"`, `"now-24h"`, `"now"`), `tags`, `limit` (default `20`).269**Time formats**: All time params accept epoch ms (e.g., `1700000000000`) OR Grafana-style relative strings (`"now"`, `"now-1h"`, `"now-7d"`, `"now-30m"`). Prefer relative strings — they're simpler and avoid arithmetic errors.270**Example create**: `{ "text": "Deployed v2.1.0", "tags": ["deploy", "production"] }`271**Example create past**: `{ "text": "Incident started", "time": "now-2h", "timeEnd": "now-30m", "tags": ["incident"] }`272**Example list recent**: `{ "action": "list", "from": "now-7d", "to": "now", "tags": ["deploy"] }`273**Example list**: `{ "action": "list", "tags": ["deploy"], "limit": 10 }`274**Returns create**: `{ status: "created", id, message, time, comparisonHint: { beforeWindow: { from, to }, afterWindow: { from, to }, suggestion } }`. The `comparisonHint` provides ready-to-use ISO 8601 time ranges (30-min windows) for before/after comparison via `grafana_query` — no manual time math needed. For region annotations (with `timeEnd`), `afterWindow` starts at `timeEnd`.275**Returns list**: `{ annotations: [{ id, text, tags, time, timeEnd?, dashboardUID?, panelId? }] }`.276277### `grafana_check_alerts`278**When**: Prompt context shows "GRAFANA ALERTS", need to manage alert rules (list/delete), set up the alert webhook, silence alerts during investigation, or acknowledge an investigated alert.279**Params**: `action` (`"list"` default, `"acknowledge"`, `"list_rules"`, `"delete_rule"`, `"silence"`, `"unsilence"`, `"setup"`).280**List params**: None — returns all pending (unacknowledged) alerts. Instances capped at 5 per alert.281**Acknowledge params**: `alertId` (required) — marks an alert as investigated.282**List rules params**: `compact` (boolean, default false — 283284…(truncated)