Grafana Integration
Connection
- Base URL:
$GRAFANA_URL (e.g. https://your-org.grafana.net)
- Auth:
Authorization: Bearer $GRAFANA_API_TOKEN (service account token)
- Credentials:
GRAFANA_URL, GRAFANA_API_TOKEN from Settings → Data sources
- Caching: 10-minute cache for metadata; query results are NOT cached (time-sensitive)
- Key datasource: Prometheus UID
grafanacloud-prom
Server Lib & API Routes
- File:
server/lib/grafana.ts
Exported Functions
| Function |
Description |
listDashboards(query?) |
Search dashboards by query |
getDashboard(uid) |
Full dashboard JSON with panels |
getDatasources() |
List all datasources |
getAlertRules() |
All alert rules (flattened from groups) |
getAlertInstances() |
Currently firing alert instances |
queryDatasource(uid, queries[], from?, to?) |
Proxy to Grafana's /api/ds/query |
API Routes
| Route |
Description |
GET /api/grafana/dashboards |
Search dashboards |
GET /api/grafana/dashboard?uid=... |
Full dashboard JSON |
GET /api/grafana/datasources |
List datasources |
GET /api/grafana/alerts |
Alert rules and firing instances |
POST /api/grafana/query |
Query datasource (Prometheus, Loki, etc.) |
Agent Action
Use grafana for agent-facing Grafana work. Do not call /api/grafana/*
directly from the agent.
| Mode |
Args |
Description |
dashboards |
search |
Search dashboards |
dashboard |
uid |
Full dashboard JSON |
datasources |
|
List datasources |
alerts |
|
Alert rules and firing instances |
query |
datasourceUid, queries, from, to |
Query a datasource |
Dashboard
/adhoc/engineering — Engineering dashboard (mirrors a Grafana dashboard by UID)
Grafana Response Frame Format
{ results: { [refId]: { frames: [{ schema: { fields }, data: { values } }] } } }
values[0] = timestamps (epoch ms), values[1+] = metric values- Series labels from
schema.fields[i].labels (e.g. { model: "claude-3.5" })
- Transform to Recharts:
{ time: number, [seriesName]: number }[]
Template Variables
$CodegenMode → default quality-v4$AIModel → regex pattern, default .* (all)$environment → cloud or cloud-v2, default cloud- Variables interpolated via string replace before querying
Key Prometheus Metrics
- Codegen:
vcpcodegen_completion_total, vcpcodegen_completion_latency_bucket, vcpcodegen_feedback_total, vcpcodegen_error_total
- LLM:
llm_completion_cost_total, llm_input_tokens_total, llm_output_tokens_total, llm_latency_bucket, llm_failures_total, llm_completions_total
- Projects:
projects_proposed_config_total, projects_status_total, projects_remote_machine_*, projects_start_duration_bucket
- API/Runtime:
api_request_total, with_span_duration_bucket, memory_heap_usage_percent_bucket
- Fly.io:
fly_endpoint_total, fly_machine_*
- GitHub:
builderbot_pr_created_total, builderbot_pr_closed_total
Key Patterns & Gotchas
queryDatasource constructs body with datasource UID in each query target; timestamps as string msgetAlertRules flattens nested rule groups from unified alerting APIgetAlertInstances handles both array and wrapped object response shapes- POST helper caches by default (fine for idempotent endpoints) but
queryDatasource skips cache
- Heatmap panels render as multi-series line/area (Recharts has no native heatmap; PromQL
histogram_quantile() computes quantiles)
Incident Investigation Pattern
For production issues, query Grafana/Prometheus FIRST:
- LLM latency by model (
llm_latency_bucket)
- Request rates (
api_request_total)
- Error rates (
llm_failures_total)
- Instance counts (via Cloud Monitoring)
Then check Sentry for application errors, Cloud Logging for raw logs.
1---2name: grafana3description: Query Grafana dashboards and Prometheus metrics for service health, LLM latency, and infrastructure monitoring. Use this skill when the user asks about service health, monitoring, alerts, or performance metrics.4---56# Grafana Integration78## Connection910- **Base URL**: `$GRAFANA_URL` (e.g. `https://your-org.grafana.net`)11- **Auth**: `Authorization: Bearer $GRAFANA_API_TOKEN` (service account token)12- **Credentials**: `GRAFANA_URL`, `GRAFANA_API_TOKEN` from Settings → Data sources13- **Caching**: 10-minute cache for metadata; query results are NOT cached (time-sensitive)14- **Key datasource**: Prometheus UID `grafanacloud-prom`1516## Server Lib & API Routes1718- **File**: `server/lib/grafana.ts`1920### Exported Functions2122| Function | Description |23| --------------------------------------------- | --------------------------------------- |24| `listDashboards(query?)` | Search dashboards by query |25| `getDashboard(uid)` | Full dashboard JSON with panels |26| `getDatasources()` | List all datasources |27| `getAlertRules()` | All alert rules (flattened from groups) |28| `getAlertInstances()` | Currently firing alert instances |29| `queryDatasource(uid, queries[], from?, to?)` | Proxy to Grafana's `/api/ds/query` |3031### API Routes3233| Route | Description |34| ------------------------------------ | ----------------------------------------- |35| `GET /api/grafana/dashboards` | Search dashboards |36| `GET /api/grafana/dashboard?uid=...` | Full dashboard JSON |37| `GET /api/grafana/datasources` | List datasources |38| `GET /api/grafana/alerts` | Alert rules and firing instances |39| `POST /api/grafana/query` | Query datasource (Prometheus, Loki, etc.) |4041### Agent Action4243Use `grafana` for agent-facing Grafana work. Do not call `/api/grafana/*`44directly from the agent.4546| Mode | Args | Description |47| ------------- | ---------------------------------------- | -------------------------------- |48| `dashboards` | `search` | Search dashboards |49| `dashboard` | `uid` | Full dashboard JSON |50| `datasources` | | List datasources |51| `alerts` | | Alert rules and firing instances |52| `query` | `datasourceUid`, `queries`, `from`, `to` | Query a datasource |5354### Dashboard5556- `/adhoc/engineering` — Engineering dashboard (mirrors a Grafana dashboard by UID)5758## Grafana Response Frame Format5960```61{ results: { [refId]: { frames: [{ schema: { fields }, data: { values } }] } } }62```6364- `values[0]` = timestamps (epoch ms), `values[1+]` = metric values65- Series labels from `schema.fields[i].labels` (e.g. `{ model: "claude-3.5" }`)66- Transform to Recharts: `{ time: number, [seriesName]: number }[]`6768## Template Variables6970- `$CodegenMode` → default `quality-v4`71- `$AIModel` → regex pattern, default `.*` (all)72- `$environment` → `cloud` or `cloud-v2`, default `cloud`73- Variables interpolated via string replace before querying7475## Key Prometheus Metrics7677- **Codegen**: `vcpcodegen_completion_total`, `vcpcodegen_completion_latency_bucket`, `vcpcodegen_feedback_total`, `vcpcodegen_error_total`78- **LLM**: `llm_completion_cost_total`, `llm_input_tokens_total`, `llm_output_tokens_total`, `llm_latency_bucket`, `llm_failures_total`, `llm_completions_total`79- **Projects**: `projects_proposed_config_total`, `projects_status_total`, `projects_remote_machine_*`, `projects_start_duration_bucket`80- **API/Runtime**: `api_request_total`, `with_span_duration_bucket`, `memory_heap_usage_percent_bucket`81- **Fly.io**: `fly_endpoint_total`, `fly_machine_*`82- **GitHub**: `builderbot_pr_created_total`, `builderbot_pr_closed_total`8384## Key Patterns & Gotchas8586- `queryDatasource` constructs body with datasource UID in each query target; timestamps as string ms87- `getAlertRules` flattens nested rule groups from unified alerting API88- `getAlertInstances` handles both array and wrapped object response shapes89- POST helper caches by default (fine for idempotent endpoints) but `queryDatasource` skips cache90- Heatmap panels render as multi-series line/area (Recharts has no native heatmap; PromQL `histogram_quantile()` computes quantiles)9192## Incident Investigation Pattern9394For production issues, query Grafana/Prometheus FIRST:95961. LLM latency by model (`llm_latency_bucket`)972. Request rates (`api_request_total`)983. Error rates (`llm_failures_total`)994. Instance counts (via Cloud Monitoring)100101Then check Sentry for application errors, Cloud Logging for raw logs.
Builder.io