Trulens Diagnosis

TruLens Diagnosis and Improvement

After running evaluations, use this skill to diagnose why scores are low and what to change — closing the loop from eval results to concrete fixes.

Prerequisites

Before diagnosing, ensure you have:

1. Run evaluations with at least one feedback function (see running-evaluations skill)
2. Access to the session to retrieve records and feedback results

Instructions

Step 1: Triage — identify failing metrics and patterns

Pull records with feedback scores and filter to failing cases:

import pandas as pd
from trulens.core import TruSession

session = TruSession()

# Get all records with feedback columns
records_df, feedback_cols = session.get_records_and_feedback()

# Identify feedback functions present
print("Feedback functions:", list(feedback_cols))

# Filter to low-scoring records (score < 0.7)
FAIL_THRESHOLD = 0.7
failing = {}
for col in feedback_cols:
    if col in records_df.columns:
        low = records_df[records_df[col] < FAIL_THRESHOLD]
        if not low.empty:
            failing[col] = low
            print(
                f"\n{col}: {len(low)} failing records out of {len(records_df)}"
            )
            print(low[["input", "output", col]].head(5))

Look for patterns:

• Which feedback functions fail most often?
• Are failures clustered around specific queries or query types?
• Do failures correlate with specific app versions?

Step 2: Root cause analysis — trace failures to their span

For each failing metric, inspect the OTEL trace to find the problematic span:

import json

# Get the record_id of a failing record
failing_record_id = failing["Context Relevance"].iloc[0]["record_id"]

# Fetch the full trace for that record
records = session.get_records_and_feedback()[0]
record = records[records["record_id"] == failing_record_id].iloc[0]

# Inspect the raw trace JSON
trace = json.loads(record["record_json"])
print(json.dumps(trace, indent=2))

Use the dashboard to inspect individual span attributes:

from trulens.dashboard import run_dashboard

run_dashboard(session)
# Navigate to Records → click a failing record → expand spans

Span-to-metric mapping:

Step 3: Apply targeted fixes

Use the failure pattern and root-cause span to select the right fix:

Low Context Relevance (retrieved contexts don't match query)

# Symptom: RETRIEVAL span query_text doesn't match retrieved_contexts

# Fix A: Increase retrieval k
vector_store.query(query_texts=query, n_results=5)  # was 2

# Fix B: Improve chunking — reduce chunk size, add overlap
from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(chunk_size=256, chunk_overlap=50)

# Fix C: Switch embedding model
from langchain_openai import OpenAIEmbeddings

embeddings = OpenAIEmbeddings(model="text-embedding-3-large")  # was small


# Fix D: Add query rewriting before retrieval
@instrument(span_type=SpanAttributes.SpanType.RETRIEVAL)
def retrieve(self, query: str) -> list:
    rewritten = rewrite_query(query)  # expand/clarify query
    return vector_store.query(rewritten)

Low Groundedness (response not grounded in retrieved context)

# Symptom: GENERATION span output contains facts not in retrieved contexts

# Fix A: Strengthen system prompt to cite sources
SYSTEM_PROMPT = """Answer ONLY using the provided context.
If the context doesn't contain the answer, say 'I don't know.'
Do not add information from your training data."""


# Fix B: Filter low-relevance chunks before passing to LLM
def filter_contexts(contexts: list, query: str, threshold: float = 0.7) -> list:
    # Re-rank and filter by relevance score
    return [c for c, score in zip(contexts, scores) if score >= threshold]


# Fix C: Reduce temperature to minimize hallucination
response = client.chat.completions.create(
    model="gpt-4o-mini",
    temperature=0.0,  # was 0.7
    messages=[...],
)

Low Answer Relevance (response doesn't address the question)

# Symptom: RECORD_ROOT output doesn't align with input question

# Fix A: Add chain-of-thought prompting
SYSTEM_PROMPT = """First, identify what the user is asking.
Then answer that specific question directly.
Question: {query}"""

# Fix B: Add output format instructions
SYSTEM_PROMPT += "\nStart your response by restating the question, then answer."

# Fix C: Use a more capable model for complex questions
model = "gpt-4o"  # upgrade from gpt-4o-mini for complex reasoning

Low Tool Selection (agent picks wrong tools)

# Symptom: TOOL span tool_name doesn't match the subtask goal

# Fix A: Improve tool descriptions — be explicit about when to use each tool
tools = [
    {
        "name": "search_documents",
        "description": (
            "Search the knowledge base for relevant documents. "
            "Use this for factual questions about stored content. "
            "Do NOT use for real-time data or calculations."
        ),
    },
    ...,
]

# Fix B: Add few-shot examples to the agent prompt showing correct tool use
AGENT_PROMPT = """You have these tools: {tools}

Examples of correct tool selection:
- For 'What does the policy say about X?' → use search_documents
- For 'Calculate the total cost' → use calculator
- For 'Get today's weather' → use web_search
"""

Low Tool Calling (agent forms bad tool arguments)

# Symptom: TOOL span input_arguments are invalid or semantically off

# Fix A: Add argument validation examples to the tool description
# Fix B: Use structured output for tool argument generation
response = client.chat.completions.create(
    model="gpt-4o",
    tools=tools,
    tool_choice="required",
    # Add a system message explaining each argument
)

# Fix C: Add error handling and retry logic
try:
    result = call_tool(tool_name, args)
except ToolArgumentError as e:
    # Re-prompt with the error message
    corrected_args = correct_arguments(args, error=str(e))
    result = call_tool(tool_name, corrected_args)

Low Plan Adherence (agent doesn't follow its own plan)

# Symptom: AGENT spans don't correspond to planned steps

# Fix A: Simplify the plan — fewer, clearer steps
# Fix B: Add step validation checkpoints
# Fix C: Use structured plan format the model follows more reliably
PLAN_PROMPT = """Create a plan with numbered steps.
After each step, verify it was completed before proceeding to the next."""

Low Logical Consistency (reasoning contains contradictions)

# Symptom: AGENT/GENERATION spans contain unsupported assertions

# Fix A: Reduce temperature
# Fix B: Add explicit reasoning instructions
PROMPT += "\nThink step by step. Each claim must be traceable to the context."

# Fix C: Add a self-review step
PROMPT += "\nAfter your answer, review it for logical consistency."

Step 4: Re-evaluate — compare before and after

Run the same test set under a new app version and compare:

# New version with fixes applied
tru_v2 = YourWrapper(
    fixed_app,
    app_name="MyApp",
    app_version="v2",  # bump version
    feedbacks=feedbacks,
)

with tru_v2 as recording:
    for query in test_queries:
        fixed_app.query(query)

recording.retrieve_feedback_results(timeout=300)

# Compare on leaderboard
session.get_leaderboard()

Step 5: Regression check — verify fixes didn't hurt other metrics

records_v2, _ = session.get_records_and_feedback(app_versions=["v2"])

for col in feedback_cols:
    if col in records_v2.columns:
        v1_mean = records_df[col].mean()
        v2_mean = records_v2[col].mean()
        delta = v2_mean - v1_mean
        status = "✅" if delta >= 0 else "⚠️"
        print(f"{status} {col}: {v1_mean:.3f} → {v2_mean:.3f} ({delta:+.3f})")

Flag any metric that regressed by more than 0.05 — investigate whether the fix for metric A introduced a failure in metric B.

Troubleshooting