# Read Eval Logs

> View and analyse Inspect evaluation log files using the Python API. Trigger whenever you need to look at a .eval file yourself without using pre-written scripts.

- Skill: `ukgovernmentbeis/read-eval-logs` (Agent Skill)
- Install (CLI): `npx skillmds@latest add ukgovernmentbeis/read-eval-logs`
- Raw SKILL.md: https://api.skillmd.com/api/skills/ukgovernmentbeis/read-eval-logs/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Integrations & APIs
- Author: UKGovernmentBEIS (https://skillmd.com/u/ukgovernmentbeis)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/ukgovernmentbeis/read-eval-logs

---


# Analysing Eval Log Files

This skill covers how to view and analyse Inspect evaluation log files using the Python API and CLI commands.

## Quick Reference

### CLI Commands

```bash
# List all logs in the default log directory (./logs or INSPECT_LOG_DIR)
uv run inspect log list --json

# List logs with specific status
uv run inspect log list --json --status success
uv run inspect log list --json --status error

# List retryable logs (error/cancelled without subsequent success)
uv run inspect log list --json --retryable

# Dump a log file as JSON (works with any format: .eval or .json)
uv run inspect log dump <log_file_path>

# Convert between log formats
uv run inspect log convert source.json --to eval --output-dir log-output
uv run inspect log convert logs/ --to eval --output-dir logs-eval

# Get JSON schema for log files
uv run inspect log schema
```

### Interactive Log Viewer

```bash
# Start the interactive log viewer (updates automatically)
uv run inspect view
```

## Python API

### Key Imports

```python
from inspect_ai.log import (
    # Listing and reading
    list_eval_logs,
    read_eval_log,
    read_eval_log_sample,
    read_eval_log_samples,
    read_eval_log_sample_summaries,
    
    # Writing
    write_eval_log,
    
    # Utilities
    retryable_eval_logs,
    recompute_metrics,
    resolve_sample_attachments,
    
    # Types
    EvalLog,
    EvalLogInfo,
    EvalSample,
    EvalSampleSummary,
)
```

### Listing Logs

```python
# List all logs in default directory
logs = list_eval_logs()

# List logs in a specific directory
logs = list_eval_logs(log_dir="./experiment-logs")

# Filter by format
logs = list_eval_logs(formats=["eval"])  # Only .eval files

# Filter with a custom function (receives header-only EvalLog)
logs = list_eval_logs(filter=lambda log: log.status == "success")

# Non-recursive listing
logs = list_eval_logs(recursive=False)
```

### Reading Logs

```python
# Read full log
log = read_eval_log("path/to/logfile.eval")

# Read header only (fast, excludes samples)
log = read_eval_log("path/to/logfile.eval", header_only=True)

# Read with attachments resolved
log = read_eval_log("path/to/logfile.eval", resolve_attachments=True)
```

### Reading Samples

```python
# Read a single sample by ID and epoch
sample = read_eval_log_sample("path/to/logfile.eval", id=42, epoch=1)

# Read a single sample by UUID
sample = read_eval_log_sample("path/to/logfile.eval", uuid="sample-uuid")

# Stream all samples (memory efficient - one at a time)
for sample in read_eval_log_samples("path/to/logfile.eval"):
    process(sample)

# Stream samples from incomplete logs
for sample in read_eval_log_samples("path/to/logfile.eval", all_samples_required=False):
    process(sample)

# Read sample summaries (fast, includes scoring info)
summaries = read_eval_log_sample_summaries("path/to/logfile.eval")
```

### Filtering Samples

```python
# Read only samples with errors using summaries for filtering
errors = []
for summary in read_eval_log_sample_summaries(log_file):
    if summary.error is not None:
        errors.append(
            read_eval_log_sample(log_file, summary.id, summary.epoch)
        )
```

## EvalLog Structure

The `EvalLog` object contains:

| Field        | Type                         | Description                                           |
| ------------ | ---------------------------- | ----------------------------------------------------- |
| `version`    | `int`                        | File format version (currently 2)                     |
| `status`     | `str`                        | `"started"`, `"success"`, `"cancelled"`, or `"error"` |
| `eval`       | `EvalSpec`                   | Task, model, creation time, config                    |
| `plan`       | `EvalPlan`                   | Solvers and generation config                         |
| `results`    | `EvalResults`                | Aggregate scores and metrics                          |
| `stats`      | `EvalStats`                  | Runtime, model usage statistics                       |
| `error`      | `EvalError`                  | Error info if `status == "error"`                     |
| `samples`    | `list[EvalSample]`           | Individual samples (if not header_only)               |
| `reductions` | `list[EvalSampleReductions]` | Multi-epoch reductions                                |
| `location`   | `str`                        | URI where log was read from                           |

### Always Check Status

```python
log = read_eval_log("path/to/logfile.eval")
if log.status == "success":
    # Safe to analyse results
    for score in log.results.scores:
        print(f"{score.name}: {score.metrics}")
```

## EvalSample Structure

Each sample contains:

| Field         | Type                       | Description               |
| ------------- | -------------------------- | ------------------------- |
| `id`          | `int \| str`               | Unique sample ID          |
| `epoch`       | `int`                      | Epoch number              |
| `input`       | `str \| list[ChatMessage]` | Sample input              |
| `target`      | `str \| list[str]`         | Expected target(s)        |
| `messages`    | `list[ChatMessage]`        | Full conversation history |
| `output`      | `ModelOutput`              | Model's output            |
| `scores`      | `dict[str, Score]`         | Scores from scorers       |
| `metadata`    | `dict[str, Any]`           | Sample metadata           |
| `store`       | `dict[str, Any]`           | State at end of execution |
| `events`      | `list[Event]`              | Transcript events         |
| `error`       | `EvalError`                | Error if sample failed    |
| `total_time`  | `float`                    | Total sample runtime      |
| `model_usage` | `dict[str, ModelUsage]`    | Token usage               |

## Common Analysis Patterns

### Get Aggregate Metrics

```python
log = read_eval_log(log_file, header_only=True)
if log.results:
    for score in log.results.scores:
        print(f"Scorer: {score.name}")
        for metric_name, metric in score.metrics.items():
            print(f"  {metric_name}: {metric.value}")
```

### Analyse Failed Samples

```python
log = read_eval_log(log_file)
if log.samples:
    failed = [s for s in log.samples if s.error is not None]
    for sample in failed:
        print(f"Sample {sample.id}: {sample.error.message}")
```

### Extract Model Usage

```python
log = read_eval_log(log_file, header_only=True)
for model, usage in log.stats.model_usage.items():
    print(f"{model}: {usage.input_tokens} in, {usage.output_tokens} out")
```

### Compare Multiple Runs

```python
logs = list_eval_logs(filter=lambda l: l.eval.task == "my_task")
for log_info in logs:
    log = read_eval_log(log_info, header_only=True)
    if log.results and log.results.scores:
        accuracy = log.results.scores[0].metrics.get("accuracy")
        print(f"{log.eval.model}: {accuracy.value if accuracy else 'N/A'}")
```

### Find Retryable Logs

```python
all_logs = list_eval_logs()
retryable = retryable_eval_logs(all_logs)
for log_info in retryable:
    print(f"Can retry: {log_info.name}")
```

## Log File Formats

| Type    | Description                                               |
| ------- | --------------------------------------------------------- |
| `.eval` | Binary format, ~1/8 size of JSON, fast incremental access |
| `.json` | Text format, human-readable, slower for large files       |

Both formats are fully supported by the API and can be intermixed.

## Working with Large Logs

For logs too large to fit in memory:

1. **Use `.eval` format** - supports compression and incremental access
2. **Read header only** - `read_eval_log(log_file, header_only=True)`
3. **Stream samples** - `read_eval_log_samples()` yields one at a time
4. **Use summaries** - `read_eval_log_sample_summaries()` for quick overview

## Modifying Logs

```python
# Read, modify, and write back
log = read_eval_log(log_file)
# ... modify log ...
write_eval_log(log)  # Uses log.location automatically

# Write to a new location
write_eval_log(log, location="new_path.eval")

# Recompute metrics after score edits
recompute_metrics(log)
write_eval_log(log)
```

## Environment Variables

| Variable                        | Description                                        |
| ------------------------------- | -------------------------------------------------- |
| `INSPECT_LOG_DIR`               | Default log directory (default: `./logs`)          |
| `INSPECT_EVAL_LOG_FILE_PATTERN` | Log filename pattern (e.g., `{task}_{model}_{id}`) |

## Related Tools

- **[Inspect Scout](https://meridianlabs-ai.github.io/inspect_scout/)** - Transcript analysis
- **[Inspect Viz](https://meridianlabs-ai.github.io/inspect_viz/)** - Data visualization
- **Log Dataframes** - Extract pandas DataFrames from logs (see `inspect_ai.analysis`)