Archestra Observability
Use this skill before changing tracing, metrics, span naming, metric labels, or local observability setup.
Run commands from platform/ unless specifically instructed otherwise.
Naming new attributes and metrics
Before introducing any new span attribute or metric name, look it up — do not coin a name from intuition.
- Span attributes: search the OTEL semantic-convention registry and use the existing attribute verbatim if one fits. Registry: https://opentelemetry.io/docs/specs/semconv/registry/attributes/gen-ai/ (wider set: https://opentelemetry.io/docs/specs/semconv/registry/attributes/). Example: prompt-cache tokens are
gen_ai.usage.cache_read.input_tokens and gen_ai.usage.cache_creation.input_tokens, not a custom archestra.usage.*. Only use an archestra.* name when nothing in the registry fits, and say why in a comment.
- "Not yet stable" is not a reason to avoid a standard name. The whole
gen_ai.* namespace is Development-stability, including the gen_ai.usage.* attributes already emitted here — match that bar, don't custom-namespace to dodge it.
- Metrics: match the existing
llm_* / prom-client family and label names in metrics/; don't introduce a new metric style. Add a label value to an existing metric only if it won't change what current aggregates mean — otherwise add a dedicated metric (cache tokens use a separate llm_cache_tokens_total, not new type values on llm_tokens_total).
Local setup
tilt trigger observability
docker compose -f dev/docker-compose.observability.yml up -d
Both commands are equivalent — the tilt resource wraps the same compose file — and start the full observability stack with pre-configured datasources: Tempo, Loki, OTEL Collector, Prometheus, and Grafana.
Local URLs
- Tempo API:
http://localhost:3200/.
- Grafana:
http://localhost:3002/.
- Prometheus:
http://localhost:9090/.
- Backend metrics:
http://localhost:9050/metrics.
Tracing
- Follow OTEL GenAI Semantic Conventions (see "Naming new attributes and metrics" — check the registry before adding any attribute): https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-agent-spans/.
- LLM spans use
gen_ai.agent.id, gen_ai.agent.name, gen_ai.provider.name, gen_ai.request.model, gen_ai.operation.name, and archestra.agent.label.<key> for dynamic agent labels.
- MCP spans use
gen_ai.tool.name and mcp.server.name.
- Team metadata uses the custom
archestra.<scope>.team.* namespace (no OTEL registry equivalent), where scope is the principal the teams belong to — agent (the executing agent's teams) or user (the requesting user's teams). archestra.<scope>.team.ids / .names are array-valued (a principal can belong to multiple teams), and archestra.<scope>.team.label.<key> carries team labels merged per key across the principal's teams. Set via setTeamAttributes(span, teams, scope) in observability/tracing/attributes.ts; agent teams come from AgentTeamModel.getTeamLabelInfoForAgent and user teams from TeamModel.getTeamLabelInfoForUser, resolved once per request.
- Session tracking uses
gen_ai.conversation.id from the X-Archestra-Session-Id header.
- Span names are
chat {model}, generate_content {model}, and execute_tool {tool_name}.
- Agent label keys are fetched from the database on startup and used as dynamic Prometheus metric label dimensions (see Metrics); the tracing SDK's resource carries only
service.name/service.version. On spans, agent labels are set per-request via setAgentAttributes.
- Traces are stored in Grafana Tempo.
- User identity is tracked with
archestra.user.id, archestra.user.email, and archestra.user.name when available.
- LLM spans include
archestra.cost in USD and gen_ai.usage.total_tokens.
Metrics
- Prometheus metrics
llm_request_duration_seconds and llm_tokens_total include provider, model, agent_id, agent_name, agent_type, source, and dynamic agent labels as dimensions — deliberately NOT external_agent_id, which is client-supplied and unbounded and would explode series cardinality. Do not add it back.
agent_id is internal.
external_agent_id comes from the client-provided X-Archestra-Agent-Id header and is a label only on agent_runs_total.
- MCP metrics include
agent_id, agent_name, and agent_type.
- Metrics are reinitialized on startup with current label keys from the database.
1---2name: archestra-dev-observability3description: Use when changing Archestra tracing, metrics, OpenTelemetry, Tempo, Grafana, Prometheus, LLM/MCP spans, observability labels, or local observability setup.4metadata:5 source: https://skillmd.com/skills/archestra-ai/archestra-dev-observability6---78# Archestra Observability910Use this skill before changing tracing, metrics, span naming, metric labels, or local observability setup.1112Run commands from `platform/` unless specifically instructed otherwise.1314## Naming new attributes and metrics1516Before introducing any new span attribute or metric name, look it up — do not coin a name from intuition.1718- **Span attributes**: search the OTEL semantic-convention registry and use the existing attribute verbatim if one fits. Registry: https://opentelemetry.io/docs/specs/semconv/registry/attributes/gen-ai/ (wider set: https://opentelemetry.io/docs/specs/semconv/registry/attributes/). Example: prompt-cache tokens are `gen_ai.usage.cache_read.input_tokens` and `gen_ai.usage.cache_creation.input_tokens`, not a custom `archestra.usage.*`. Only use an `archestra.*` name when nothing in the registry fits, and say why in a comment.19- **"Not yet stable" is not a reason to avoid a standard name.** The whole `gen_ai.*` namespace is Development-stability, including the `gen_ai.usage.*` attributes already emitted here — match that bar, don't custom-namespace to dodge it.20- **Metrics**: match the existing `llm_*` / prom-client family and label names in `metrics/`; don't introduce a new metric style. Add a label value to an existing metric only if it won't change what current aggregates mean — otherwise add a dedicated metric (cache tokens use a separate `llm_cache_tokens_total`, not new `type` values on `llm_tokens_total`).2122## Local setup2324```bash25tilt trigger observability26docker compose -f dev/docker-compose.observability.yml up -d27```2829Both commands are equivalent — the tilt resource wraps the same compose file — and start the full observability stack with pre-configured datasources: Tempo, Loki, OTEL Collector, Prometheus, and Grafana.3031## Local URLs3233- Tempo API: `http://localhost:3200/`.34- Grafana: `http://localhost:3002/`.35- Prometheus: `http://localhost:9090/`.36- Backend metrics: `http://localhost:9050/metrics`.3738## Tracing3940- Follow OTEL GenAI Semantic Conventions (see "Naming new attributes and metrics" — check the registry before adding any attribute): https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-agent-spans/.41- LLM spans use `gen_ai.agent.id`, `gen_ai.agent.name`, `gen_ai.provider.name`, `gen_ai.request.model`, `gen_ai.operation.name`, and `archestra.agent.label.<key>` for dynamic agent labels.42- MCP spans use `gen_ai.tool.name` and `mcp.server.name`.43- Team metadata uses the custom `archestra.<scope>.team.*` namespace (no OTEL registry equivalent), where scope is the principal the teams belong to — `agent` (the executing agent's teams) or `user` (the requesting user's teams). `archestra.<scope>.team.ids` / `.names` are array-valued (a principal can belong to multiple teams), and `archestra.<scope>.team.label.<key>` carries team labels merged per key across the principal's teams. Set via `setTeamAttributes(span, teams, scope)` in `observability/tracing/attributes.ts`; agent teams come from `AgentTeamModel.getTeamLabelInfoForAgent` and user teams from `TeamModel.getTeamLabelInfoForUser`, resolved once per request.44- Session tracking uses `gen_ai.conversation.id` from the `X-Archestra-Session-Id` header.45- Span names are `chat {model}`, `generate_content {model}`, and `execute_tool {tool_name}`.46- Agent label keys are fetched from the database on startup and used as dynamic Prometheus metric label dimensions (see Metrics); the tracing SDK's resource carries only `service.name`/`service.version`. On spans, agent labels are set per-request via `setAgentAttributes`.47- Traces are stored in Grafana Tempo.48- User identity is tracked with `archestra.user.id`, `archestra.user.email`, and `archestra.user.name` when available.49- LLM spans include `archestra.cost` in USD and `gen_ai.usage.total_tokens`.5051## Metrics5253- Prometheus metrics `llm_request_duration_seconds` and `llm_tokens_total` include `provider`, `model`, `agent_id`, `agent_name`, `agent_type`, `source`, and dynamic agent labels as dimensions — deliberately NOT `external_agent_id`, which is client-supplied and unbounded and would explode series cardinality. Do not add it back.54- `agent_id` is internal.55- `external_agent_id` comes from the client-provided `X-Archestra-Agent-Id` header and is a label only on `agent_runs_total`.56- MCP metrics include `agent_id`, `agent_name`, and `agent_type`.57- Metrics are reinitialized on startup with current label keys from the database.