sboghossian Langfuse Trace Inspector
What it does
Langfuse is an open-source LLM observability platform. Every LLM request in the legal AI product emits a trace to Langfuse, capturing the full prompt, the model's response, token counts, latency, scores, and metadata. The trace inspector is the procedure and tooling for using Langfuse traces to:
- Debug why a skill produced an unexpected or incorrect output.
- Understand the prompt that was sent to the model (including assembled system prompt, skill content, and matter context).
- Measure skill-level latency and identify bottlenecks.
- Review human or automated quality scores against individual traces.
- Correlate traces with audit log events for compliance investigations.
Trace structure
In the legal AI product, each request produces a Langfuse trace with this span hierarchy:
Trace: [request_id]
├── span: router.skill-selection
│ input: user_message, session_context
│ output: selected_skill_id, confidence
│ latency: Xms
│
├── span: context.assembly
│ input: skill_id, matter_id, user_id
│ output: assembled_prompt_token_count
│ metadata: cache_hit_l1, cache_hit_l2, cache_hit_l3
│ latency: Xms
│
├── generation: llm.invoke
│ model: claude-sonnet-4-6
│ input: full prompt (system + skill + matter + user)
│ output: model response
│ usage: {input_tokens, output_tokens, cached_tokens}
│ latency: {ttfb, total}
│ cost: {usd}
│
├── span: output.formatter
│ input: raw model response
│ output: formatted response for UI
│ latency: Xms
│
└── scores: [quality_score, accuracy_score, user_feedback]
Key metadata to emit per trace
At the trace level, always include:
langfuse.trace(
name=f"skill.{skill_id}",
user_id=user_id,
session_id=session_id,
metadata={
"org_id": org_id,
"matter_id": matter_id,
"skill_id": skill_id,
"skill_version": skill_version,
"model_id": model_id,
"cache_hit_l1": bool,
"cache_hit_l2": bool,
"cache_hit_l3": bool,
"fallback_used": bool,
"fallback_reason": str | None
},
tags=[skill_id, org_id, matter_type, "production"]
)
Tags enable filtering in the Langfuse UI without querying metadata.
Inspection workflow
Step 1: Locate the trace
Filter by one or more of:
session_id(from user complaint or incident report)user_id(user-specific investigation)org_id+ date range (org-level quality review)skill_id+ score < threshold (quality regression hunt)tag: production+model_id+ date (model behavior after upgrade)
Step 2: Inspect the generation span
The generation span reveals exactly what was sent to the model and what the model returned. Key things to check:
- System prompt: was the correct system prompt version loaded?
- Skill content: is the correct skill version present in the context?
- Matter context: was the matter-specific context correctly assembled?
- User message: was the user input correctly parsed and forwarded?
- Model output: does the raw output match the formatted output? Any post-processing issues?
In a legal product, the full prompt can be large (100K+ tokens with matter context). Use Langfuse's diff view to compare prompt versions across two traces to identify what changed between a good and bad output.
Step 3: Review scores
Four types of scores may be attached to a trace:
| Score type | Source | Range |
|---|---|---|
quality |
Judge model (automated eval) | 0–1 |
accuracy |
Legal accuracy check (automated or human) | 0–1 |
user_feedback |
Thumbs up/down from the UI | -1, 0, 1 |
latency_slo |
Pass/fail against SLO | 0, 1 |
Cross-reference: low quality + low accuracy score on the same trace = likely content issue. Low quality + high accuracy + negative user_feedback = likely formatting or communication issue.
Step 4: Identify the root cause
Common root causes in legal AI products:
| Symptom | Likely cause | Investigation |
|---|---|---|
| Wrong conflict check result | Incorrect matter context; stale KB data | Check matter_id in trace; verify matter context assembly |
| Engagement letter with wrong jurisdiction | Wrong template loaded | Check skill content in generation span; verify template selection logic |
| Slow response on a specific skill | Context too large; cache miss | Check context.assembly latency; cache_hit_l3 flag |
| Hallucinated statute number | Model inference error; no relevant KB doc retrieved | Check retrieval results in context; add a KB document covering that statute |
| Fallback model used unexpectedly | Primary model overloaded | Check fallback_used flag; correlate with provider status page |
Sampling and retention
Tracing every production request at full detail is expensive. Use a sampling strategy:
| Traffic segment | Sampling rate | Full prompt stored? |
|---|---|---|
| P0 skills (conflict, engagement) | 100% | Yes |
| P1 skills | 20% | Yes |
| P2 skills | 5% | Metadata only |
| Failed requests | 100% | Yes |
| Low-score requests (score < 0.7) | 100% | Yes |
Langfuse traces containing full prompts include user-inputted content (potentially privileged). Apply the same data-retention and access-control policies as the audit log: org-scoped, role-restricted, minimum 1 year for P0 traces.
Privacy and privilege considerations
Langfuse traces contain:
- User messages (which may include client-identifying details)
- Matter context (potentially privileged)
- Model responses (potentially privileged attorney work product)
Configure Langfuse in your own infrastructure (self-hosted) for any firm that requires data residency or has restrictions on third-party data processors. Do not use the Langfuse cloud service for a firm with strict data governance requirements without a signed DPA and data residency confirmation.
Related skills
- [[eng-langfuse-eval-runner]]
- [[eng-audit-log-schema]]
- [[eng-latency-slo-by-skill]]
- [[eng-cost-per-message-tracker]]