Grafana Integration
Connection
- Base URL:
$GRAFANA_URL (e.g. https://your-org.grafana.net)
- Auth:
Authorization: Bearer $GRAFANA_API_TOKEN (service account token)
- Credentials:
GRAFANA_URL, GRAFANA_API_TOKEN from Settings → Data sources
- Caching: 10-minute cache for metadata; query results are NOT cached (time-sensitive)
- Key datasource: Prometheus UID
grafanacloud-prom
Server Lib & API Routes
- File:
server/lib/grafana.ts
Exported Functions
| Function |
Description |
listDashboards(query?) |
Search dashboards by query |
getDashboard(uid) |
Full dashboard JSON with panels |
getDatasources() |
List all datasources |
getAlertRules() |
All alert rules (flattened from groups) |
getAlertInstances() |
Currently firing alert instances |
queryDatasource(uid, queries[], from?, to?) |
Proxy to Grafana's /api/ds/query |
API Routes
| Route |
Description |
GET /api/grafana/dashboards |
Search dashboards |
GET /api/grafana/dashboard?uid=... |
Full dashboard JSON |
GET /api/grafana/datasources |
List datasources |
GET /api/grafana/alerts |
Alert rules and firing instances |
POST /api/grafana/query |
Query datasource (Prometheus, Loki, etc.) |
Agent Action
Use grafana for agent-facing Grafana work. Do not call /api/grafana/*
directly from the agent.
| Mode |
Args |
Description |
dashboards |
search |
Search dashboards |
dashboard |
uid |
Full dashboard JSON |
datasources |
|
List datasources |
alerts |
|
Alert rules and firing instances |
query |
datasourceUid, queries, from, to |
Query a datasource |
Dashboard
/adhoc/engineering — Engineering dashboard (mirrors a Grafana dashboard by UID)
Grafana Response Frame Format
{ results: { [refId]: { frames: [{ schema: { fields }, data: { values } }] } } }
values[0] = timestamps (epoch ms), values[1+] = metric values
- Series labels from
schema.fields[i].labels (e.g. { model: "claude-3.5" })
- Transform to Recharts:
{ time: number, [seriesName]: number }[]
Template Variables
$CodegenMode → default quality-v4
$AIModel → regex pattern, default .* (all)
$environment → cloud or cloud-v2, default cloud
- Variables interpolated via string replace before querying
Key Prometheus Metrics
- Codegen:
vcpcodegen_completion_total, vcpcodegen_completion_latency_bucket, vcpcodegen_feedback_total, vcpcodegen_error_total
- LLM:
llm_completion_cost_total, llm_input_tokens_total, llm_output_tokens_total, llm_latency_bucket, llm_failures_total, llm_completions_total
- Projects:
projects_proposed_config_total, projects_status_total, projects_remote_machine_*, projects_start_duration_bucket
- API/Runtime:
api_request_total, with_span_duration_bucket, memory_heap_usage_percent_bucket
- Fly.io:
fly_endpoint_total, fly_machine_*
- GitHub:
builderbot_pr_created_total, builderbot_pr_closed_total
Key Patterns & Gotchas
queryDatasource constructs body with datasource UID in each query target; timestamps as string ms
getAlertRules flattens nested rule groups from unified alerting API
getAlertInstances handles both array and wrapped object response shapes
- POST helper caches by default (fine for idempotent endpoints) but
queryDatasource skips cache
- Heatmap panels render as multi-series line/area (Recharts has no native heatmap; PromQL
histogram_quantile() computes quantiles)
Incident Investigation Pattern
For production issues, query Grafana/Prometheus FIRST:
- LLM latency by model (
llm_latency_bucket)
- Request rates (
api_request_total)
- Error rates (
llm_failures_total)
- Instance counts (via Cloud Monitoring)
Then check Agent-Native Analytics Monitoring -> Errors for captured application
issues, external Sentry when connected, and Cloud Logging for raw logs.
1---2name: grafana3description: >4 Query Grafana dashboards and Prometheus metrics for service health, LLM latency, and infrastructure monitoring.5 Use this skill when the user asks about service health, monitoring, alerts, or performance metrics.6metadata:7 source: https://skillmd.com/skills/builderio/grafana8---910# Grafana Integration1112## Connection1314- **Base URL**: `$GRAFANA_URL` (e.g. `https://your-org.grafana.net`)15- **Auth**: `Authorization: Bearer $GRAFANA_API_TOKEN` (service account token)16- **Credentials**: `GRAFANA_URL`, `GRAFANA_API_TOKEN` from Settings → Data sources17- **Caching**: 10-minute cache for metadata; query results are NOT cached (time-sensitive)18- **Key datasource**: Prometheus UID `grafanacloud-prom`1920## Server Lib & API Routes2122- **File**: `server/lib/grafana.ts`2324### Exported Functions2526| Function | Description |27| --------------------------------------------- | --------------------------------------- |28| `listDashboards(query?)` | Search dashboards by query |29| `getDashboard(uid)` | Full dashboard JSON with panels |30| `getDatasources()` | List all datasources |31| `getAlertRules()` | All alert rules (flattened from groups) |32| `getAlertInstances()` | Currently firing alert instances |33| `queryDatasource(uid, queries[], from?, to?)` | Proxy to Grafana's `/api/ds/query` |3435### API Routes3637| Route | Description |38| ------------------------------------ | ----------------------------------------- |39| `GET /api/grafana/dashboards` | Search dashboards |40| `GET /api/grafana/dashboard?uid=...` | Full dashboard JSON |41| `GET /api/grafana/datasources` | List datasources |42| `GET /api/grafana/alerts` | Alert rules and firing instances |43| `POST /api/grafana/query` | Query datasource (Prometheus, Loki, etc.) |4445### Agent Action4647Use `grafana` for agent-facing Grafana work. Do not call `/api/grafana/*`48directly from the agent.4950| Mode | Args | Description |51| ------------- | ---------------------------------------- | -------------------------------- |52| `dashboards` | `search` | Search dashboards |53| `dashboard` | `uid` | Full dashboard JSON |54| `datasources` | | List datasources |55| `alerts` | | Alert rules and firing instances |56| `query` | `datasourceUid`, `queries`, `from`, `to` | Query a datasource |5758### Dashboard5960- `/adhoc/engineering` — Engineering dashboard (mirrors a Grafana dashboard by UID)6162## Grafana Response Frame Format6364```65{ results: { [refId]: { frames: [{ schema: { fields }, data: { values } }] } } }66```6768- `values[0]` = timestamps (epoch ms), `values[1+]` = metric values69- Series labels from `schema.fields[i].labels` (e.g. `{ model: "claude-3.5" }`)70- Transform to Recharts: `{ time: number, [seriesName]: number }[]`7172## Template Variables7374- `$CodegenMode` → default `quality-v4`75- `$AIModel` → regex pattern, default `.*` (all)76- `$environment` → `cloud` or `cloud-v2`, default `cloud`77- Variables interpolated via string replace before querying7879## Key Prometheus Metrics8081- **Codegen**: `vcpcodegen_completion_total`, `vcpcodegen_completion_latency_bucket`, `vcpcodegen_feedback_total`, `vcpcodegen_error_total`82- **LLM**: `llm_completion_cost_total`, `llm_input_tokens_total`, `llm_output_tokens_total`, `llm_latency_bucket`, `llm_failures_total`, `llm_completions_total`83- **Projects**: `projects_proposed_config_total`, `projects_status_total`, `projects_remote_machine_*`, `projects_start_duration_bucket`84- **API/Runtime**: `api_request_total`, `with_span_duration_bucket`, `memory_heap_usage_percent_bucket`85- **Fly.io**: `fly_endpoint_total`, `fly_machine_*`86- **GitHub**: `builderbot_pr_created_total`, `builderbot_pr_closed_total`8788## Key Patterns & Gotchas8990- `queryDatasource` constructs body with datasource UID in each query target; timestamps as string ms91- `getAlertRules` flattens nested rule groups from unified alerting API92- `getAlertInstances` handles both array and wrapped object response shapes93- POST helper caches by default (fine for idempotent endpoints) but `queryDatasource` skips cache94- Heatmap panels render as multi-series line/area (Recharts has no native heatmap; PromQL `histogram_quantile()` computes quantiles)9596## Incident Investigation Pattern9798For production issues, query Grafana/Prometheus FIRST:991001. LLM latency by model (`llm_latency_bucket`)1012. Request rates (`api_request_total`)1023. Error rates (`llm_failures_total`)1034. Instance counts (via Cloud Monitoring)104105Then check Agent-Native Analytics Monitoring -> Errors for captured application106issues, external Sentry when connected, and Cloud Logging for raw logs.