# Archestra Dev Observability

> Use when changing Archestra tracing, metrics, OpenTelemetry, Tempo, Grafana, Prometheus, LLM/MCP spans, observability labels, or local observability setup.

- Skill: `archestra-ai/archestra-dev-observability` (Agent Skill)
- Install (CLI): `npx skillmds@latest add archestra-ai/archestra-dev-observability`
- Raw SKILL.md: https://api.skillmd.com/api/skills/archestra-ai/archestra-dev-observability/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: archestra-ai (https://skillmd.com/u/archestra-ai)
- Updated: 2026-09-10
- Page: https://skillmd.com/skills/archestra-ai/archestra-dev-observability

---


# Archestra Observability

Use this skill before changing tracing, metrics, span naming, metric labels, or local observability setup.

Run commands from `platform/` unless specifically instructed otherwise.

## Naming new attributes and metrics

Before introducing any new span attribute or metric name, look it up — do not coin a name from intuition.

- **Span attributes**: search the OTEL semantic-convention registry and use the existing attribute verbatim if one fits. Registry: https://opentelemetry.io/docs/specs/semconv/registry/attributes/gen-ai/ (wider set: https://opentelemetry.io/docs/specs/semconv/registry/attributes/). Example: prompt-cache tokens are `gen_ai.usage.cache_read.input_tokens` and `gen_ai.usage.cache_creation.input_tokens`, not a custom `archestra.usage.*`. Only use an `archestra.*` name when nothing in the registry fits, and say why in a comment.
- **"Not yet stable" is not a reason to avoid a standard name.** The whole `gen_ai.*` namespace is Development-stability, including the `gen_ai.usage.*` attributes already emitted here — match that bar, don't custom-namespace to dodge it.
- **Metrics**: match the existing `llm_*` / prom-client family and label names in `metrics/`; don't introduce a new metric style. Add a label value to an existing metric only if it won't change what current aggregates mean — otherwise add a dedicated metric (cache tokens use a separate `llm_cache_tokens_total`, not new `type` values on `llm_tokens_total`).

## Local setup

```bash
tilt trigger observability
docker compose -f dev/docker-compose.observability.yml up -d
```

Both commands are equivalent — the tilt resource wraps the same compose file — and start the full observability stack with pre-configured datasources: Tempo, Loki, OTEL Collector, Prometheus, and Grafana.

## Local URLs

- Tempo API: `http://localhost:3200/`.
- Grafana: `http://localhost:3002/`.
- Prometheus: `http://localhost:9090/`.
- Backend metrics: `http://localhost:9050/metrics`.

## Tracing

- Follow OTEL GenAI Semantic Conventions (see "Naming new attributes and metrics" — check the registry before adding any attribute): https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-agent-spans/.
- LLM spans use `gen_ai.agent.id`, `gen_ai.agent.name`, `gen_ai.provider.name`, `gen_ai.request.model`, `gen_ai.operation.name`, and `archestra.agent.label.<key>` for dynamic agent labels.
- MCP spans use `gen_ai.tool.name` and `mcp.server.name`.
- Team metadata uses the custom `archestra.<scope>.team.*` namespace (no OTEL registry equivalent), where scope is the principal the teams belong to — `agent` (the executing agent's teams) or `user` (the requesting user's teams). `archestra.<scope>.team.ids` / `.names` are array-valued (a principal can belong to multiple teams), and `archestra.<scope>.team.label.<key>` carries team labels merged per key across the principal's teams. Set via `setTeamAttributes(span, teams, scope)` in `observability/tracing/attributes.ts`; agent teams come from `AgentTeamModel.getTeamLabelInfoForAgent` and user teams from `TeamModel.getTeamLabelInfoForUser`, resolved once per request.
- Session tracking uses `gen_ai.conversation.id` from the `X-Archestra-Session-Id` header.
- Span names are `chat {model}`, `generate_content {model}`, and `execute_tool {tool_name}`.
- Agent label keys are fetched from the database on startup and used as dynamic Prometheus metric label dimensions (see Metrics); the tracing SDK's resource carries only `service.name`/`service.version`. On spans, agent labels are set per-request via `setAgentAttributes`.
- Traces are stored in Grafana Tempo.
- User identity is tracked with `archestra.user.id`, `archestra.user.email`, and `archestra.user.name` when available.
- LLM spans include `archestra.cost` in USD and `gen_ai.usage.total_tokens`.

## Metrics

- Prometheus metrics `llm_request_duration_seconds` and `llm_tokens_total` include `provider`, `model`, `agent_id`, `agent_name`, `agent_type`, `source`, and dynamic agent labels as dimensions — deliberately NOT `external_agent_id`, which is client-supplied and unbounded and would explode series cardinality. Do not add it back.
- `agent_id` is internal.
- `external_agent_id` comes from the client-provided `X-Archestra-Agent-Id` header and is a label only on `agent_runs_total`.
- MCP metrics include `agent_id`, `agent_name`, and `agent_type`.
- Metrics are reinitialized on startup with current label keys from the database.