Monitoring & Observability
A wrap around Prometheus, Grafana, OpenTelemetry and Langfuse, not a re-teaching of them. This
skill carries OrchestKit's delta (version floors, house decisions, scars) and points at the
vendor for everything else. Start at references/ork-delta.md.
Upstream coverage (do not restate)
These topics are fully covered first-party. Read the source, do not add a local copy.
| Topic |
First-party source |
| Prometheus metric types, RED method, cardinality, PromQL |
https://prometheus.io/docs/practices/ |
| Alertmanager grouping, inhibition, escalation, runbooks |
https://prometheus.io/docs/alerting/latest/configuration/ |
| Grafana dashboards, Loki and LogQL, Promtail |
https://grafana.com/docs/ |
| OpenTelemetry spans, sampling, context propagation |
https://opentelemetry.io/docs/ |
Langfuse Python SDK (@observe, as_type, score_current_span, should_export_span, LangfuseMedia) |
https://langfuse.com/docs/sdk/python |
| Langfuse v2 to v4 Python and v3 to v5 JS migration paths |
https://langfuse.com/docs/sdk/python/v4-migration |
| Langfuse self-hosting (ClickHouse, Redis, S3, Helm) |
https://langfuse.com/docs/deployment/self-host |
| Langfuse cost tracking, model pricing, Metrics API v2 |
https://langfuse.com/docs/model-usage-and-cost |
| Langfuse scores, online evaluators, annotation queues, prompt management |
https://langfuse.com/docs/scores/overview |
| Langfuse framework integrations (LangChain, LangGraph, CrewAI, Pydantic AI, Bedrock, LiveKit) |
https://langfuse.com/docs/integrations |
| Agent Graphs, observation types, rendered tool calls |
https://langfuse.com/docs/tracing-features/agent-graphs |
| PSI, KS test, KL and JS divergence, Wasserstein, embedding drift |
https://www.evidentlyai.com/blog/data-drift-detection-large-datasets |
| EWMA control charts |
https://www.itl.nist.gov/div898/handbook/pmc/section3/pmc324.htm |
| structlog, Winston, correlation IDs, log sampling |
https://www.structlog.org/en/stable/ |
Quick Reference
Total: 5 rules across 3 categories. Drift detection, cost tracking, eval scoring, Prometheus
instrumentation and alert-rule authoring moved to the upstream sources listed above.
Quick Start
# Langfuse v4 LLM tracing: semantic as_type plus inline scoring
from langfuse import observe, get_client
@observe(as_type="generation", name="analyze_content")
async def analyze_content(content: str):
get_client().update_current_trace(
user_id="user_123", session_id="session_abc",
tags=["production", "orchestkit"],
)
result = await llm.generate(content)
get_client().score_current_span(name="response_quality", value=0.85)
return result
# Prometheus RED method, wired the way this repo expects (bounded labels only)
from prometheus_client import Counter, Histogram
http_requests = Counter('http_requests_total', 'Total requests', ['method', 'endpoint', 'status'])
http_duration = Histogram('http_request_duration_seconds', 'Request latency',
buckets=[0.01, 0.05, 0.1, 0.5, 1, 2, 5])
Infrastructure Monitoring
Dashboard and health-check patterns. Metric instrumentation and alert-rule syntax are upstream.
| Rule |
File |
Key Pattern |
| Grafana Dashboards |
rules/monitoring-grafana.md |
Golden Signals, SLO/SLI, health checks |
CC 2.1.161 — OTEL resource attributes as metric labels: OTEL_RESOURCE_ATTRIBUTES values are now attached as labels on metric datapoints, so usage metrics can be sliced by custom dimensions (team, repo, environment). Add label selectors to dashboards for multi-tenant / per-team cost and usage tracking.
LLM Observability
Langfuse-based tracing for LLM applications. Cost tracking, scoring and drift statistics are
upstream; what stays here is how this repo wires traces.
| Rule |
File |
Key Pattern |
| Langfuse Traces |
rules/llm-langfuse-traces.md |
@observe decorator, OTEL spans, agent graphs |
Silent Failures
Detection and alerting for silent failures in LLM agents.
| Rule |
File |
Key Pattern |
| Tool Skipping |
rules/silent-tool-skipping.md |
Expected vs actual tool calls, Langfuse traces |
| Quality Degradation |
rules/silent-degraded-quality.md |
Heuristics + LLM-as-judge, z-score baselines |
| Silent Alerting |
rules/silent-alerting.md |
Loop detection, token spikes, escalation workflow |
CC 2.1.169 — OTEL client-cert paths require trust: untrusted project settings can no longer set OTEL client-certificate paths without a trust confirmation. If your OTEL exporter uses client certs configured in project .claude/settings.json, expect a one-time trust prompt on first use in an untrusted project — telemetry silently not flowing after 2.1.169 is usually this gate, not the collector.
Key Decisions
| Decision |
Recommendation |
Rationale |
| Metric methodology |
RED method (Rate, Errors, Duration) |
Industry standard, covers essential service health |
| Log format |
Structured JSON |
Machine-parseable, supports log aggregation |
| Tracing |
OpenTelemetry |
Vendor-neutral, auto-instrumentation, broad ecosystem |
| LLM observability |
Langfuse (not LangSmith) |
Open-source, self-hosted, built-in prompt management |
| LLM tracing API |
@observe(as_type=...) + score_current_span() |
v4: semantic types, inline scoring, span filtering |
| Langfuse APIs |
Observations API v2 + Metrics API v2 |
v4 (Mar 2026): faster querying, aggregations at scale |
| Hook telemetry transport |
JSONL under ~/.claude/analytics/, never an SDK in-process |
Hooks are per-event processes; SDK init would be paid on every spawn (references/ork-delta.md) |
Detailed Documentation
| Resource |
Description |
references/ork-delta.md |
Start here. Floors, house decisions and scars that upstream docs do not carry |
references/langfuse-js-v5.md |
JS/TS SDK v5 delta from Python 4.x: package map, phantom packages, SpanProcessor vs exporter. Read before writing any JS Langfuse code |
references/experiments-api.md |
Langfuse experiments and dataset runs as this repo uses them |
references/evaluation-scores.md |
Score shapes and scoring pipeline wiring |
references/session-tracking.md |
Session and user grouping across multi-step workflows |
references/metrics-collection.md |
Claude Code OTEL metric inventory and collector-side joins |
references/dashboards.md |
Dashboard layout conventions |
references/structured-logging.md |
Structured log field conventions |
references/dev-agent-lens.md |
LiteLLM proxy layer for API-boundary observability |
examples/orchestkit-monitoring-dashboard.md |
Worked monitoring dashboard example |
scripts/ |
Templates: Prometheus, OpenTelemetry, health checks, Langfuse |
Related Skills
defense-in-depth - Layer 8 observability as part of security architecture
devops-deployment - Observability integration with CI/CD and Kubernetes
resilience-patterns - Monitoring circuit breakers and failure scenarios
llm-evaluation - Evaluation patterns that integrate with Langfuse scoring
caching - Caching strategies that reduce costs tracked by Langfuse
1---
2name: monitoring-observability
3license: MIT
4compatibility: "Claude Code 2.1.251+."
5description: Monitoring and observability patterns for Prometheus metrics, Grafana dashboards, Langfuse v4 LLM tracing (as_type, score_current_span, should_export_span, LangfuseMedia), and drift detection. Use when adding logging, metrics, distributed tracing, LLM cost tracking, or quality drift monitoring.
6tags: [monitoring, observability, prometheus, grafana, langfuse, tracing, metrics, drift-detection, logging]
7context: fork
8version: 3.0.0
9author: OrchestKit
10user-invocable: false
11disable-model-invocation: true
12complexity: medium
13persuasion-type: reference
14targets:
15 # Python SDK. The JS/TS SDK is a separate line at 5.x (@langfuse/* packages) — see
16 # references/langfuse-js-v5.md. The self-hosted platform is a third axis (v3) and is not an
17 # SDK version.
18 - library: langfuse
19 version: ">=4.0.0"
20upstream-version-tested: "4.15.1"
21metadata:
22 source: https://skillmd.com/skills/yonatangross/monitoring-observability
23 category: document-asset-creation
24allowed-tools:
25 - Read
26 - Glob
27 - Grep
28 - WebFetch
29 - WebSearch
30path_patterns: ["**/metrics/**", "**/tracing/**", "prometheus.*", "grafana/**"]
31---
32
33# Monitoring & Observability
34
35A wrap around Prometheus, Grafana, OpenTelemetry and Langfuse, not a re-teaching of them. This
36skill carries OrchestKit's delta (version floors, house decisions, scars) and points at the
37vendor for everything else. Start at `references/ork-delta.md`.
38
39## Upstream coverage (do not restate)
40
41These topics are fully covered first-party. Read the source, do not add a local copy.
42
43| Topic | First-party source |
44|-------|--------------------|
45| Prometheus metric types, RED method, cardinality, PromQL | <https://prometheus.io/docs/practices/> |
46| Alertmanager grouping, inhibition, escalation, runbooks | <https://prometheus.io/docs/alerting/latest/configuration/> |
47| Grafana dashboards, Loki and LogQL, Promtail | <https://grafana.com/docs/> |
48| OpenTelemetry spans, sampling, context propagation | <https://opentelemetry.io/docs/> |
49| Langfuse Python SDK (`@observe`, `as_type`, `score_current_span`, `should_export_span`, `LangfuseMedia`) | <https://langfuse.com/docs/sdk/python> |
50| Langfuse v2 to v4 Python and v3 to v5 JS migration paths | <https://langfuse.com/docs/sdk/python/v4-migration> |
51| Langfuse self-hosting (ClickHouse, Redis, S3, Helm) | <https://langfuse.com/docs/deployment/self-host> |
52| Langfuse cost tracking, model pricing, Metrics API v2 | <https://langfuse.com/docs/model-usage-and-cost> |
53| Langfuse scores, online evaluators, annotation queues, prompt management | <https://langfuse.com/docs/scores/overview> |
54| Langfuse framework integrations (LangChain, LangGraph, CrewAI, Pydantic AI, Bedrock, LiveKit) | <https://langfuse.com/docs/integrations> |
55| Agent Graphs, observation types, rendered tool calls | <https://langfuse.com/docs/tracing-features/agent-graphs> |
56| PSI, KS test, KL and JS divergence, Wasserstein, embedding drift | <https://www.evidentlyai.com/blog/data-drift-detection-large-datasets> |
57| EWMA control charts | <https://www.itl.nist.gov/div898/handbook/pmc/section3/pmc324.htm> |
58| structlog, Winston, correlation IDs, log sampling | <https://www.structlog.org/en/stable/> |
59
60## Quick Reference
61
62| Category | Rules | Impact | When to Use |
63|----------|-------|--------|-------------|
64| [Infrastructure Monitoring](#infrastructure-monitoring) | 1 | CRITICAL | Grafana dashboards, Golden Signals, SLO/SLI |
65| [LLM Observability](#llm-observability) | 1 | HIGH | Langfuse tracing, observation types, agent graphs |
66| [Silent Failures](#silent-failures) | 3 | HIGH | Tool skipping, quality degradation, loop/token spike alerting |
67
68**Total: 5 rules across 3 categories.** Drift detection, cost tracking, eval scoring, Prometheus
69instrumentation and alert-rule authoring moved to the upstream sources listed above.
70
71## Quick Start
72
73```python
74# Langfuse v4 LLM tracing: semantic as_type plus inline scoring
75from langfuse import observe, get_client
76
77@observe(as_type="generation", name="analyze_content")
78async def analyze_content(content: str):
79 get_client().update_current_trace(
80 user_id="user_123", session_id="session_abc",
81 tags=["production", "orchestkit"],
82 )
83 result = await llm.generate(content)
84 get_client().score_current_span(name="response_quality", value=0.85)
85 return result
86```
87
88```python
89# Prometheus RED method, wired the way this repo expects (bounded labels only)
90from prometheus_client import Counter, Histogram
91
92http_requests = Counter('http_requests_total', 'Total requests', ['method', 'endpoint', 'status'])
93http_duration = Histogram('http_request_duration_seconds', 'Request latency',
94 buckets=[0.01, 0.05, 0.1, 0.5, 1, 2, 5])
95```
96
97## Infrastructure Monitoring
98
99Dashboard and health-check patterns. Metric instrumentation and alert-rule syntax are upstream.
100
101| Rule | File | Key Pattern |
102|------|------|-------------|
103| Grafana Dashboards | `rules/monitoring-grafana.md` | Golden Signals, SLO/SLI, health checks |
104
105> **CC 2.1.161 — OTEL resource attributes as metric labels:** `OTEL_RESOURCE_ATTRIBUTES` values are now attached as labels on metric datapoints, so usage metrics can be sliced by custom dimensions (team, repo, environment). Add label selectors to dashboards for multi-tenant / per-team cost and usage tracking.
106
107## LLM Observability
108
109Langfuse-based tracing for LLM applications. Cost tracking, scoring and drift statistics are
110upstream; what stays here is how this repo wires traces.
111
112| Rule | File | Key Pattern |
113|------|------|-------------|
114| Langfuse Traces | `rules/llm-langfuse-traces.md` | @observe decorator, OTEL spans, agent graphs |
115
116## Silent Failures
117
118Detection and alerting for silent failures in LLM agents.
119
120| Rule | File | Key Pattern |
121|------|------|-------------|
122| Tool Skipping | `rules/silent-tool-skipping.md` | Expected vs actual tool calls, Langfuse traces |
123| Quality Degradation | `rules/silent-degraded-quality.md` | Heuristics + LLM-as-judge, z-score baselines |
124| Silent Alerting | `rules/silent-alerting.md` | Loop detection, token spikes, escalation workflow |
125
126> **CC 2.1.169 — OTEL client-cert paths require trust:** untrusted project settings can no longer set OTEL client-certificate paths without a trust confirmation. If your OTEL exporter uses client certs configured in project `.claude/settings.json`, expect a one-time trust prompt on first use in an untrusted project — telemetry silently not flowing after 2.1.169 is usually this gate, not the collector.
127
128## Key Decisions
129
130| Decision | Recommendation | Rationale |
131|----------|----------------|-----------|
132| Metric methodology | RED method (Rate, Errors, Duration) | Industry standard, covers essential service health |
133| Log format | Structured JSON | Machine-parseable, supports log aggregation |
134| Tracing | OpenTelemetry | Vendor-neutral, auto-instrumentation, broad ecosystem |
135| LLM observability | Langfuse (not LangSmith) | Open-source, self-hosted, built-in prompt management |
136| LLM tracing API | `@observe(as_type=...)` + `score_current_span()` | v4: semantic types, inline scoring, span filtering |
137| Langfuse APIs | Observations API v2 + Metrics API v2 | v4 (Mar 2026): faster querying, aggregations at scale |
138| Hook telemetry transport | JSONL under `~/.claude/analytics/`, never an SDK in-process | Hooks are per-event processes; SDK init would be paid on every spawn (`references/ork-delta.md`) |
139
140## Detailed Documentation
141
142| Resource | Description |
143|----------|-------------|
144| `references/ork-delta.md` | **Start here.** Floors, house decisions and scars that upstream docs do not carry |
145| `references/langfuse-js-v5.md` | **JS/TS SDK v5** delta from Python 4.x: package map, phantom packages, SpanProcessor vs exporter. Read before writing any JS Langfuse code |
146| `references/experiments-api.md` | Langfuse experiments and dataset runs as this repo uses them |
147| `references/evaluation-scores.md` | Score shapes and scoring pipeline wiring |
148| `references/session-tracking.md` | Session and user grouping across multi-step workflows |
149| `references/metrics-collection.md` | Claude Code OTEL metric inventory and collector-side joins |
150| `references/dashboards.md` | Dashboard layout conventions |
151| `references/structured-logging.md` | Structured log field conventions |
152| `references/dev-agent-lens.md` | LiteLLM proxy layer for API-boundary observability |
153| `examples/orchestkit-monitoring-dashboard.md` | Worked monitoring dashboard example |
154| `scripts/` | Templates: Prometheus, OpenTelemetry, health checks, Langfuse |
155
156## Related Skills
157
158- `defense-in-depth` - Layer 8 observability as part of security architecture
159- `devops-deployment` - Observability integration with CI/CD and Kubernetes
160- `resilience-patterns` - Monitoring circuit breakers and failure scenarios
161- `llm-evaluation` - Evaluation patterns that integrate with Langfuse scoring
162- `caching` - Caching strategies that reduce costs tracked by Langfuse