# Evaluate Multimodal

> Evaluate multimodal AI agents that process images, audio, PDFs, or other files. Sets up evaluations using LangWatch's LLM-as-judge with image inputs, Scenario's multimodal testing, and document parsing evaluation patterns. Use when your agent handles non-text inputs.

- Skill: `langwatch/evaluate-multimodal` (Agent Skill)
- Install (CLI): `npx skillmds@latest add langwatch/evaluate-multimodal`
- Raw SKILL.md: https://api.skillmd.com/api/skills/langwatch/evaluate-multimodal/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- License: MIT
- Author: langwatch (https://skillmd.com/u/langwatch)
- Updated: 2026-09-10
- Page: https://skillmd.com/skills/langwatch/evaluate-multimodal

---


# Evaluate Your Multimodal Agent

This recipe helps you evaluate agents that process images, audio, PDFs, or other non-text inputs.

## Step 1: Identify Modalities

Read the codebase to understand what your agent processes:

- **Images**: classification, analysis, generation, OCR
- **Audio**: transcription, voice agents, audio Q\&A
- **PDFs/Documents**: parsing, extraction, summarization
- **Mixed**: multiple input types in one pipeline

## Step 2: Read the Relevant Docs

Use the `langwatch` CLI to fetch the right pages:

```bash
langwatch scenario-docs                            # Index: locate multimodal pages
langwatch scenario-docs multimodal/audio-to-text   # Audio testing patterns
langwatch scenario-docs multimodal/multimodal-files # Generic file analysis patterns
langwatch docs                                     # LangWatch docs index
langwatch docs evaluations/experiments/sdk         # Experiment SDK basics
langwatch docs evaluations/evaluators/list         # Browse evaluator types
```

For PDF evaluation specifically, reference the pattern from `sdks/python/examples/pdf_parsing_evaluation.ipynb`:

- Download/load documents
- Define extraction pipeline
- Use LangWatch experiment SDK to evaluate extraction accuracy

## Step 3: Set Up Evaluation by Modality

### Image Evaluation

LangWatch's LLM-as-judge evaluators can accept images. Create an evaluation that:

1. Loads test images
2. Runs the agent on each image
3. Uses an LLM-as-judge evaluator to assess output quality

```python
import langwatch

experiment = langwatch.experiment.init("image-eval")

for idx, entry in experiment.loop(enumerate(image_dataset)):
    result = my_agent(image=entry["image_path"])
    experiment.evaluate(
        "llm_boolean",
        index=idx,
        data={
            "input": entry["image_path"],  # LLM-as-judge can view images
            "output": result,
        },
        settings={
            "model": "openai/gpt-5-mini",
            "prompt": "Does the agent correctly describe/classify this image?",
        },
    )
```

### Audio Evaluation

Use Scenario's audio testing patterns:

- Audio-to-text: verify transcription accuracy
- Audio-to-audio: verify voice agent responses

Read the dedicated guide:

```bash
langwatch scenario-docs multimodal/audio-to-text
```

### PDF/Document Evaluation

Follow the pattern from the PDF parsing evaluation example:

1. Load documents (PDFs, CSVs, etc.)
2. Define extraction/parsing pipeline
3. Evaluate extraction accuracy against expected fields
4. Use structured evaluation (exact match for fields, LLM judge for summaries)

### File Analysis

For agents that process arbitrary files, read the file analysis guide:

```bash
langwatch scenario-docs multimodal/multimodal-files
```

## Step 4: Generate Domain-Specific Test Data

For each modality, generate or collect test data that matches the agent's actual use case:

- If it's a medical imaging agent → use relevant medical image samples
- If it's a document parser → use real document types the agent encounters
- If it's a voice assistant → record realistic voice prompts

## Step 5: Run and Iterate

Run the evaluation, review results, fix issues, re-run until quality is acceptable.

## Common Mistakes

- Do NOT evaluate multimodal agents with text-only metrics. Use image-aware judges
- Do NOT skip testing with real file formats. Synthetic descriptions aren't enough
- Do NOT forget to handle file loading errors in evaluations
- Do NOT use generic test images. Use domain-specific ones matching the agent's purpose
- Always read the relevant `langwatch scenario-docs ...` page for the modality before writing code; multimodal patterns differ a lot from text-only ones