# Trulens Evaluation Setup

> Configure feedback functions and selectors for TruLens evaluations

- Skill: `truera/trulens-evaluation-setup` (Agent Skill)
- Install (CLI): `npx skillmds@latest add truera/trulens-evaluation-setup`
- Raw SKILL.md: https://api.skillmd.com/api/skills/truera/trulens-evaluation-setup/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: truera (https://skillmd.com/u/truera)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/truera/trulens-evaluation-setup

---


# TruLens Evaluation Setup

Configure feedback functions to evaluate your LLM app's quality.

## Interactive Evaluation Selection

**Before proceeding, let's determine the right evaluations for your app.**

### Question 1: What type of app are you building?

**Option A: RAG (Retrieval-Augmented Generation)**
- Your app retrieves documents/chunks from a knowledge base
- It generates responses grounded in the retrieved context
- Examples: Q&A systems, document search, knowledge assistants

→ **Recommended: RAG Triad metrics**
- Context Relevance
- Groundedness
- Answer Relevance

**Option B: Agent**
- Your app uses tools to accomplish tasks
- It may involve multi-step reasoning or planning
- Examples: research agents, coding assistants, task automation

→ **Recommended: Agent GPA metrics** (continue to Question 2)

---

### Question 2 (Agents only): Does your agent do explicit planning?

**Yes, my agent creates plans before executing:**
- Agent outputs a plan/strategy before taking actions
- Agent references its plan during execution

→ **Use all Agent GPA metrics:**
- Logical Consistency
- Plan Quality
- Plan Adherence
- Execution Efficiency
- Tool Selection
- Tool Calling
- Tool Quality

**No, my agent acts without explicit planning:**
- Agent takes actions directly without stating a plan
- Agent uses reactive decision-making

→ **Use Agent GPA metrics (excluding plan metrics):**
- Logical Consistency
- Execution Efficiency
- Tool Selection
- Tool Calling
- Tool Quality

---

### Question 3: Do you want to add any additional evaluations?

Consider adding these based on your needs:

| Evaluation | Use Case |
|------------|----------|
| **Coherence** | Check if output is well-structured and readable |
| **Conciseness** | Ensure responses aren't unnecessarily verbose |
| **Harmlessness** | Detect potentially harmful content |
| **Sentiment** | Analyze emotional tone of responses |
| **Custom metrics** | Domain-specific evaluations (see below) |

#### Creating Custom Metrics

If you need domain-specific evaluations, describe what you want to measure:

**What aspect of your app do you want to evaluate?**

Examples:
- "Check if the response follows our brand voice guidelines"
- "Verify the output contains required legal disclaimers"
- "Measure technical accuracy for code generation"
- "Evaluate if customer support responses show empathy"

**Template for custom metrics:**

```python
def my_custom_metric(input_text: str, output_text: str) -> float:
    """
    Describe what this metric evaluates.

    Returns:
        float: Score between 0.0 (worst) and 1.0 (best)
    """
    # Option 1: Rule-based logic
    # score = 1.0 if "required phrase" in output_text else 0.0

    # Option 2: Use LLM-as-judge
    # provider = OpenAI()
    # response = provider.client.chat.completions.create(
    #     model="gpt-4o",
    #     messages=[{
    #         "role": "user",
    #         "content": f"Rate this response on [YOUR CRITERIA]. Input: {input_text} Output: {output_text}. Return only a number 0-10."
    #     }]
    # )
    # score = float(response.choices[0].message.content) / 10.0

    return score


f_custom = Metric(
    implementation=my_custom_metric,
    name="My Custom Metric",
    selectors={
        "input_text": Selector.select_record_input(),
        "output_text": Selector.select_record_output(),
    },
)
```

**Custom metric with context:**

```python
def custom_with_context(query: str, context: str, response: str) -> float:
    """Evaluate using query, retrieved context, and response."""
    # Your evaluation logic
    return score


f_custom_context = Metric(
    implementation=custom_with_context,
    name="Custom Context Metric",
    selectors={
        "query": Selector.select_record_input(),
        "context": Selector.select_context(collect_list=True),
        "response": Selector.select_record_output(),
    },
)
```

**Tell me what you want to evaluate and I'll help you create the metric!**

---

## Overview

Feedback functions evaluate specific aspects of your app by:

1. Selecting data from instrumented spans (inputs, outputs, retrieved contexts)
2. Applying evaluation logic (LLM-as-judge, similarity metrics, etc.)
3. Returning scores between 0.0 and 1.0

## Prerequisites

```bash
pip install trulens trulens-providers-openai
```

## Instructions

### Step 1: Initialize a Feedback Provider

```python
from trulens.providers.openai import OpenAI

provider = OpenAI(model_engine="gpt-4o")
```

### Step 2: Create Feedback Functions with Selector Shortcuts

TruLens provides shortcuts for common selection patterns:

```python
from trulens.core import Metric, Selector

# Answer relevance: input → output
f_answer_relevance = Metric(
    implementation=provider.relevance_with_cot_reasons,
    name="Answer Relevance",
    selectors={
        "prompt": Selector.select_record_input(),
        "response": Selector.select_record_output(),
    },
)

# Context relevance: input → each context chunk
f_context_relevance = Metric(
    implementation=provider.context_relevance_with_cot_reasons,
    name="Context Relevance",
    selectors={
        "question": Selector.select_record_input(),
        "context": Selector.select_context(collect_list=False),
    },
)

# Groundedness: all contexts → output
f_groundedness = Metric(
    implementation=provider.groundedness_measure_with_cot_reasons,
    name="Groundedness",
    selectors={
        "source": Selector.select_context(collect_list=True),
        "statement": Selector.select_record_output(),
    },
)
```

**Shortcut Reference:**

| Shortcut | Selects | Required Span Type |
|----------|---------|-------------------|
| `on_input()` | App input | `RECORD_ROOT` |
| `on_output()` | App output | `RECORD_ROOT` |
| `on_context()` | Retrieved contexts | `RETRIEVAL` |

**⚠️ IMPORTANT: `.on_input()` and `.on_output()` require `RECORD_ROOT` spans!**

These shortcuts look for spans with `span_type=SpanAttributes.SpanType.RECORD_ROOT`. If you use manual instrumentation with a different span type (like `AGENT`), the shortcuts will not find any data.

**Solutions:**
- Use framework wrappers (`TruGraph`, `TruChain`, `TruLlama`) which create `RECORD_ROOT` automatically
- Use explicit `@instrument(span_type=SpanAttributes.SpanType.RECORD_ROOT, ...)` on your entry point
- Use explicit `Selector` objects instead of shortcuts (see Step 3)

### Step 3: Using Explicit Selectors

For more control, use `Selector` to target specific span attributes:

```python
from trulens.core import Metric
from trulens.core.feedback.selector import Selector
from trulens.otel.semconv.trace import SpanAttributes

f_answer_relevance = Metric(
    implementation=provider.relevance_with_cot_reasons,
    name="Answer Relevance",
    selectors={
        "prompt": Selector(
            span_type=SpanAttributes.SpanType.RECORD_ROOT,
            span_attribute=SpanAttributes.RECORD_ROOT.INPUT,
        ),
        "response": Selector(
            span_type=SpanAttributes.SpanType.RECORD_ROOT,
            span_attribute=SpanAttributes.RECORD_ROOT.OUTPUT,
        ),
    },
)
```

### Step 4: Understanding collect_list

The `collect_list` parameter controls how multiple values are handled:

| Setting | Behavior | Use Case |
|---------|----------|----------|
| `collect_list=False` | Evaluate each value individually | Context relevance (score each chunk) |
| `collect_list=True` | Concatenate all values | Groundedness (check against all context) |

```python
# Evaluate each retrieved context individually (returns multiple scores)
f_context_relevance = Metric(
    implementation=provider.context_relevance_with_cot_reasons,
    name="Context Relevance",
    selectors={
        "question": Selector.select_record_input(),
        "context": Selector(
            span_type=SpanAttributes.SpanType.RETRIEVAL,
            span_attribute=SpanAttributes.RETRIEVAL.RETRIEVED_CONTEXTS,
            collect_list=False,
        ),
    },
)

# Evaluate against all contexts combined (returns single score)
f_groundedness = Metric(
    implementation=provider.groundedness_measure_with_cot_reasons,
    name="Groundedness",
    selectors={
        "source": Selector(
            span_type=SpanAttributes.SpanType.RETRIEVAL,
            span_attribute=SpanAttributes.RETRIEVAL.RETRIEVED_CONTEXTS,
            collect_list=True,
        ),
        "statement": Selector.select_record_output(),
    },
)
```

### Step 5: Aggregating Multiple Scores

When `collect_list=False` produces multiple scores, aggregate them:

```python
import numpy as np

f_context_relevance = Metric(
    implementation=provider.context_relevance_with_cot_reasons,
    name="Context Relevance",
    selectors={
        "question": Selector.select_record_input(),
        "context": Selector.select_context(collect_list=False),
    },
    agg=np.mean,
)
```

Common aggregation functions:
- `np.mean` - Average score
- `np.min` - Worst score (conservative)
- `np.max` - Best score (optimistic)

## Common Patterns

### RAG Triad Setup

```python
import numpy as np
from trulens.core import Metric, Selector
from trulens.providers.openai import OpenAI

provider = OpenAI()

# Context Relevance: Is each retrieved chunk relevant to the query?
f_context_relevance = Metric(
    implementation=provider.context_relevance_with_cot_reasons,
    name="Context Relevance",
    selectors={
        "question": Selector.select_record_input(),
        "context": Selector.select_context(collect_list=False),
    },
    agg=np.mean,
)

# Groundedness: Is the response grounded in the retrieved context?
f_groundedness = Metric(
    implementation=provider.groundedness_measure_with_cot_reasons,
    name="Groundedness",
    selectors={
        "source": Selector.select_context(collect_list=True),
        "statement": Selector.select_record_output(),
    },
)

# Answer Relevance: Does the response answer the original question?
f_answer_relevance = Metric(
    implementation=provider.relevance_with_cot_reasons,
    name="Answer Relevance",
    selectors={
        "prompt": Selector.select_record_input(),
        "response": Selector.select_record_output(),
    },
)

rag_feedbacks = [f_context_relevance, f_groundedness, f_answer_relevance]
```

### Agent GPA Setup

```python
from trulens.core import Metric, Selector
from trulens.providers.openai import OpenAI

provider = OpenAI()

# Logical Consistency
f_logical_consistency = Metric(
    implementation=provider.logical_consistency_with_cot_reasons,
    name="Logical Consistency",
    selectors={"trace": Selector(trace_level=True)},
)

# Plan Quality (exclude if agent doesn't do explicit planning)
f_plan_quality = Metric(
    implementation=provider.plan_quality_with_cot_reasons,
    name="Plan Quality",
    selectors={"trace": Selector(trace_level=True)},
)

# Plan Adherence (exclude if agent doesn't do explicit planning)
f_plan_adherence = Metric(
    implementation=provider.plan_adherence_with_cot_reasons,
    name="Plan Adherence",
    selectors={"trace": Selector(trace_level=True)},
)

# Execution Efficiency
f_execution_efficiency = Metric(
    implementation=provider.execution_efficiency_with_cot_reasons,
    name="Execution Efficiency",
    selectors={"trace": Selector(trace_level=True)},
)

# Tool Selection
f_tool_selection = Metric(
    implementation=provider.tool_selection_with_cot_reasons,
    name="Tool Selection",
    selectors={"trace": Selector(trace_level=True)},
)

# Tool Calling
f_tool_calling = Metric(
    implementation=provider.tool_calling_with_cot_reasons,
    name="Tool Calling",
    selectors={"trace": Selector(trace_level=True)},
)

# Tool Quality
f_tool_quality = Metric(
    implementation=provider.tool_quality_with_cot_reasons,
    name="Tool Quality",
    selectors={"trace": Selector(trace_level=True)},
)

# Use all for agents with planning
agent_feedbacks_with_planning = [
    f_logical_consistency,
    f_plan_quality,
    f_plan_adherence,
    f_execution_efficiency,
    f_tool_selection,
    f_tool_calling,
    f_tool_quality,
]

# For agents without explicit planning, exclude plan metrics
agent_feedbacks_no_planning = [
    f_logical_consistency,
    f_execution_efficiency,
    f_tool_selection,
    f_tool_calling,
    f_tool_quality,
]
```

### Custom Feedback Function

```python
def my_custom_metric(input_text: str, output_text: str) -> float:
    """Custom evaluation returning score between 0.0 and 1.0."""
    # Your evaluation logic here
    score = len(output_text) / (len(input_text) + len(output_text))
    return min(max(score, 0.0), 1.0)


f_custom = Metric(
    implementation=my_custom_metric,
    name="Custom Metric",
    selectors={
        "input_text": Selector.select_record_input(),
        "output_text": Selector.select_record_output(),
    },
)
```

## Troubleshooting

- **Selector not finding data**: Ensure the span attribute was set during instrumentation
- **Empty context**: Verify `RETRIEVAL.RETRIEVED_CONTEXTS` is mapped in your `@instrument()` decorator
- **Aggregation errors**: Check that `collect_list=False` is set when using `.aggregate()`
- **`.on_input()/.on_output()` returning no data**: These shortcuts require `RECORD_ROOT` span type. Use framework wrappers or explicit `@instrument(span_type=SpanAttributes.SpanType.RECORD_ROOT, ...)`. See the instrumentation skill for details.
- **Feedback columns show empty/null**: Verify your instrumentation creates `RECORD_ROOT` spans with `INPUT` and `OUTPUT` attributes