Langfuse Monitoring
This skill covers operating the data once it's in Langfuse: visualizing it (dashboards),
extracting it (metrics API), analyzing eval scores (score analytics), and alerting. It does not
cover emitting the data (that's instrumentation — vendored langfuse skill) or designing the scores
(that's the langfuse-evaluation skill).
Operating principles
- Distill judgment, fetch facts. This skill carries what to monitor and how to structure it. For
exact UI steps and API schemas, fetch live docs by appending
.md to the page URL (e.g.
https://langfuse.com/docs/metrics/features/metrics-api.md).
- You can only monitor what you instrumented. Cost, latency, userId, tags, release/version must
be on the traces to slice by them. If a needed dimension is missing, the fix is upstream
(instrumentation → vendored
langfuse skill), not here.
Workflow
1. Frame the monitoring goal
Identify the metric family — cost, latency, quality, or volume — and the dimensions
to slice by (trace name/feature, user, model, tags, release). See references/dashboards.md for the
metric/dimension model.
2. Pick the surface
- Visual, shareable, exploratory → dashboards. Start with curated (Latency/Cost/Usage), then
build custom widgets.
references/dashboards.md.
- Programmatic / scheduled / billing / embed elsewhere →
references/metrics-api.md (use v2).
- Eval scores specifically (distributions, trends, judge-vs-human agreement) →
references/score-analytics.md.
3. Build the standard dashboards
For a new production app, stand up the three durable dashboards (production health, cost
optimization, quality/UX) from references/dashboards.md.
4. Set up alerting
Disambiguate first: Spend Alerts = your Langfuse Cloud bill, not app cost. For app-level
cost/latency/quality alerts, build a Metrics-API-driven check. See references/alerting.md.
Bundled resources
references/dashboards.md — metrics & dimensions, curated vs custom dashboards, the widget
model, and the three standard dashboards to build (health / cost / quality).references/metrics-api.md — programmatic metrics: use v2, the query model
(view/metrics/dimensions/filters/timeDimension), and v1→v2 migration gotchas.references/score-analytics.md — zero-config eval-score analysis: distributions, trends,
and judge-vs-human agreement metrics (MAE/RMSE, Cohen's Kappa/F1).references/alerting.md — Spend Alerts (Cloud billing) vs application-level alerting (Metrics
API + your own check); how to alert on cost/latency/quality.
Hand-off map
| Need |
Where |
| Dashboards, metrics extraction, score analytics, alerting |
this skill |
| Emitting cost/latency/userId/tags on traces (instrumentation) |
vendored langfuse skill |
| Designing the scores/evaluators being monitored |
langfuse-evaluation skill |
| Onboarding-time spend-alert setup / production-readiness checklist |
langfuse-setup skill |
| Formal judge calibration (vs lightweight score analytics) |
vendored langfuse skill judge-calibration.md |
| Exact dashboard UI / metrics API schema |
live docs (.md-append) |
1---2name: langfuse-monitoring3description: Monitors and analyzes LLM application data already in Langfuse — dashboards, metrics, and alerting for cost, latency, quality, and volume. Use whenever the user wants to observe or report on production Langfuse data: "monitor my LLM app", "build a Langfuse dashboard", "track cost / latency / quality over time", "Langfuse metrics API", "score analytics", "set up a spend alert", "alert me when costs spike", "dashboard for production monitoring", or interpreting usage/cost/quality trends. Owns operating-the-data (dashboards/metrics/alerting); defers instrumentation to the vendored `langfuse` skill and score/evaluator design to the `langfuse-evaluation` skill.4---56# Langfuse Monitoring78This skill covers *operating the data* once it's in Langfuse: visualizing it (dashboards),9extracting it (metrics API), analyzing eval scores (score analytics), and alerting. It does not10cover emitting the data (that's instrumentation — vendored `langfuse` skill) or designing the scores11(that's the `langfuse-evaluation` skill).1213## Operating principles14151. **Distill judgment, fetch facts.** This skill carries what to monitor and how to structure it. For16 exact UI steps and API schemas, fetch live docs by appending `.md` to the page URL (e.g.17 `https://langfuse.com/docs/metrics/features/metrics-api.md`).182. **You can only monitor what you instrumented.** Cost, latency, userId, tags, release/version must19 be on the traces to slice by them. If a needed dimension is missing, the fix is upstream20 (instrumentation → vendored `langfuse` skill), not here.2122## Workflow2324### 1. Frame the monitoring goal25Identify the metric family — **cost**, **latency**, **quality**, or **volume** — and the dimensions26to slice by (trace name/feature, user, model, tags, release). See `references/dashboards.md` for the27metric/dimension model.2829### 2. Pick the surface30- **Visual, shareable, exploratory** → dashboards. Start with **curated** (Latency/Cost/Usage), then31 build **custom** widgets. `references/dashboards.md`.32- **Programmatic / scheduled / billing / embed elsewhere** → `references/metrics-api.md` (use v2).33- **Eval scores specifically** (distributions, trends, judge-vs-human agreement) →34 `references/score-analytics.md`.3536### 3. Build the standard dashboards37For a new production app, stand up the three durable dashboards (production health, cost38optimization, quality/UX) from `references/dashboards.md`.3940### 4. Set up alerting41Disambiguate first: **Spend Alerts = your Langfuse Cloud bill**, *not* app cost. For app-level42cost/latency/quality alerts, build a Metrics-API-driven check. See `references/alerting.md`.4344## Bundled resources4546- **`references/dashboards.md`** — metrics & dimensions, curated vs custom dashboards, the widget47 model, and the three standard dashboards to build (health / cost / quality).48- **`references/metrics-api.md`** — programmatic metrics: use v2, the query model49 (view/metrics/dimensions/filters/timeDimension), and v1→v2 migration gotchas.50- **`references/score-analytics.md`** — zero-config eval-score analysis: distributions, trends,51 and judge-vs-human agreement metrics (MAE/RMSE, Cohen's Kappa/F1).52- **`references/alerting.md`** — Spend Alerts (Cloud billing) vs application-level alerting (Metrics53 API + your own check); how to alert on cost/latency/quality.5455## Hand-off map5657| Need | Where |58|------|-------|59| Dashboards, metrics extraction, score analytics, alerting | this skill |60| Emitting cost/latency/userId/tags on traces (instrumentation) | vendored `langfuse` skill |61| Designing the scores/evaluators being monitored | `langfuse-evaluation` skill |62| Onboarding-time spend-alert setup / production-readiness checklist | `langfuse-setup` skill |63| Formal judge calibration (vs lightweight score analytics) | vendored `langfuse` skill `judge-calibration.md` |64| Exact dashboard UI / metrics API schema | live docs (`.md`-append) |
jbaham2