# Analyzing Logs

> Use whenever the user wants to analyze or understand what happened in an Inspect AI eval, sample, or set of logs. Routes to the right tool (`inspect_ai.log` for single-log work, `inspect_ai.analysis` for cross-log dataframes, Inspect Scout for transcript pattern detection) and covers the `inspect_ai.analysis` surface for ad-hoc analysis. Triggers on questions like "what happened in this eval/sample", "how did model X compare to Y", "show me outliers across these runs", "did any sample do X", or any follow-up where the user is analyzing logs (not just reading them).

- Skill: `meridianlabs-ai/analyzing-logs` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add meridianlabs-ai/analyzing-logs`
- Raw SKILL.md: https://api.skillmd.com/api/skills/meridianlabs-ai/analyzing-logs/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: meridianlabs-ai (https://skillmd.com/u/meridianlabs-ai)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/meridianlabs-ai/analyzing-logs

---


# Analyzing Inspect eval logs

Use this skill when the user asks **questions about what happened** in one or more eval logs. The first job is to pick the right tool for the question; the second is to apply it.

## Pick the right tool

| Question shape | Tool | Skill |
|---|---|---|
| About **one** log header, one sample's content, one sample's error history | `inspect_ai.log` | `reading-logs` skill |
| About **many** samples or logs at once: aggregates, score distributions, comparisons, outliers, dataframes | `inspect_ai.analysis` | this skill, below |
| **Pattern detection** inside transcripts: "did any sample refuse / reward-hack / repeat / get stuck", "find samples where X happened in the messages" | **Inspect Scout** | [Inspect Scout docs](https://meridianlabs-ai.github.io/inspect_scout/llms.txt) |

The boundary cues:

- **Single log or sample → `reading-logs`.** A question answerable by reading one log's header or one sample's `.messages` belongs there, not here. Even if the question is "about transcript content" for one specific sample, it's still a single-sample read. **When you drop down to single-sample work mid-analysis, invoke the `reading-logs` skill explicitly** — its decision tree covers the right `read_eval_log_sample` / `read_eval_log_samples` patterns plus the `all_samples_required=False` caveat for non-success logs.
- **Cross-sample numerical aggregates → this skill (`inspect_ai.analysis`).** Score distributions, model comparisons, token-usage rollups, outliers, anything that wants a table with one row per sample or per run.
- **Pattern mining inside transcripts → Scout.** "Did any sample do X" / "find samples where Y" questions, where the answer requires scanning the *content* of many transcripts.

## Cross-log / cross-sample analysis: `inspect_ai.analysis`

When the question is "how do these samples or runs compare?", reach for `inspect_ai.analysis`. It builds **pandas dataframes** at four granularities; pick the cheapest one that answers the question:

| Builder | Output | Use when |
|---|---|---|
| `evals_df(logs)` | 1 row per log file | per-run rollups: which model scored highest, how long did each run take |
| `samples_df(logs)` | 1 row per sample | per-sample analysis across runs: score distributions, outliers, comparisons |
| `messages_df(logs)` | 1 row per message | message-content slicing: filter by role/function, count tool calls |
| `events_df(logs)` | 1 row per event | event-level timing or transcript reconstruction |

```python
from inspect_ai.analysis import samples_df, EvalModel, SampleSummary, SampleScores
df = samples_df("logs/", columns=EvalModel + SampleSummary + SampleScores)
```

**Composable columns.** Pick the granularity (one of the four `*_df` builders) and the columns you need from the pre-built groups (composed via `+`). Pre-built groups cover most needs:

- Eval-level: `EvalInfo`, `EvalTask`, `EvalModel`, `EvalDataset`, `EvalConfiguration`, `EvalResults`, `EvalScores`
- Sample-level: `SampleSummary`, `SampleScores`, `SampleMessages`
- Message-level: `MessageContent`, `MessageToolCalls`
- Event-level: `EventInfo`, `EventTiming`, `ModelEventColumns`, `ToolEventColumns`

Custom columns inherit from `EvalColumn` / `SampleColumn` / `MessageColumn` / `EventColumn`.

**Common args.** `parallel=True` for full-content reads (capped at 8 workers; not available on `evals_df`). `strict=False` returns `(df, errors)` so partial failures don't raise; always unpack the tuple: `df, errors = samples_df(..., strict=False)`.

**`prepare(df, [...])`** chains post-build operations: `model_info` (add org, display name, release date), `task_info` (task display names), `log_viewer` (local paths to URLs), `frontier` (mark top performers per task), `score_to_float` (score column conversion). Reach for it when enriching dataframes for plots or leaderboards.

**Three traps to know about up front:**

- **Score columns: single-scorer vs multi-scorer.** `samples_df` explodes scores into one column per scorer (`score_<scorer-name>`). For a one-scorer eval, group by that single column and aggregate. For multi-scorer / cross-scorer comparisons, prefer **`evals_df` with `EvalScores`** (gives you `score_headline_value` per log) and accept that you're averaging across scorers. Values can be strings (`'C'`/`'I'`), numbers, or nested dicts depending on the scorer; use `prepare(df, [score_to_float("score_headline_value")])` for eval headline scores, or pass the concrete sample score column(s) such as `score_<scorer-name>` when normalizing `samples_df`.
- **pyarrow dtypes bite scalar comparisons.** `samples_df` returns columns with pyarrow-backed dtypes (`double[pyarrow]`, `large_string[pyarrow]`). Plain `df[col].isna()` works, but scalar `==` on `<NA>` raises `TypeError: boolean value of NA is ambiguous`. Use `pd.to_numeric(df[col], errors='coerce')` or `df[col].fillna(...)` before comparing.
- **Status field canonical values.** `EvalLog.status ∈ {success, error, cancelled, started}`. `started` means the run crashed mid-execution (see `reading-logs` for `recoverable_eval_logs` recovery). Filter on these explicitly rather than guessing.

**ALWAYS fetch and read https://inspect.aisi.org.uk/dataframe.html.md before doing any non-trivial analysis work.** The column system has more depth than this section covers, and the docs are the source of truth.

## Transcript pattern detection: Inspect Scout

For questions like "did any sample reward-hack", "find samples where the agent repeated the same tool call", "which transcripts show refusals": that's **Inspect Scout** territory. Don't try to express these as `messages_df` filter expressions; Scout's scanners are purpose-built for it and the result is more reliable. Point the user at https://meridianlabs-ai.github.io/inspect_scout/llms.txt for the surface.

## Reading the right log dir

Most analysis starts with a directory of logs. Default lookups:

- The user's working `logs/` directory if they're inside an Inspect project.
- `INSPECT_LOG_DIR` if set.
- The `log_location` from `ctl tasks --json` if the user is asking about a still-running eval (see `babysitting-evals`).

If ambiguous, ask. Don't guess.

**Multiple logdirs**: the dataframe builders take `LogPaths | EvalLog | Sequence[EvalLog]`, so you can pass a list of paths to analyze several logdirs in one call, e.g. `samples_df(["runs/2026-06/", "runs/2026-05/"])`.

## Keeping logs in memory across questions

When the user is going to ask several follow-up questions about the same log (or a set of logs), don't re-read on every turn. Set up **persistent state** via a Python REPL MCP, i.e. a tool that keeps a Python interpreter alive across calls so variables you define in one call are still there in the next.

**When to set this up:**

- The user explicitly asks for an analysis session, or signals follow-ups are coming ("walk me through this", "now show me X about the same log").
- The log is large enough that re-reading hurts: roughly, multi-GB on disk, or hundreds of samples you'd otherwise stream on each turn.
- You're using `samples_df` (or any full-content read); the deserialized dataframe is expensive to rebuild from scratch every turn.

For a one-shot question on a small log, skip this; reload is cheaper than setup.

**How to do it.** This plugin bundles a persistent Python REPL MCP server ([`posit-dev/mcp-repl`](https://github.com/posit-dev/mcp-repl)). The server name depends on the agent: on Claude Code it's `plugin:inspect-skills:py-repl` (namespaced because plugin-bundled MCPs use a `plugin:<plugin-name>:<server-name>` prefix); on Codex it's registered flat as `py-repl` (no plugin prefix, as shown by `codex mcp list`) and the tool is invoked by its bare name. One tool:

- `repl`, args: `input` (the Python source to run, as a string) and optional `timeout_ms`. State persists across calls. Multi-line input runs as a complete cell, so `def` / `for` / `if` blocks execute as-is with no special termination. Plots from `matplotlib` etc. are returned as inline images when the client model is vision-capable; otherwise as file paths. Smart-echo suppresses redundant stdout. The session runs in an OS-level sandbox at the `workspace-write` policy: reads broadly, writes only inside the workspace, network restricted to allowlisted PyPI domains (`pypi.org`, `files.pythonhosted.org`) so the dep-bootstrap can `pip install` but the REPL can't otherwise reach the network.

**Resetting the session.** There is no separate reset tool: send a `repl` call whose `input` starts with the EOF character `\x04` (Ctrl-D) and the server restarts the backend session, clearing all in-memory state (it replies "new session started"). **Don't do this mid-session** — it wipes the dataframes you've been building up. Reach for it only if the user explicitly wants a clean slate, or if the worker has wedged and you need a fresh start.

**Bootstrap dependencies on the first call.** `posit-mcp-repl` uses whatever Python interpreter it discovers (nearest `.venv` walking up from cwd, then `python3` on PATH). It does not manage its own package environment. So your first `repl` call should ensure `inspect-ai`, `pandas`, and `pyarrow` are importable, and `pip install` them if they're not:

```python
import subprocess, sys
for pkg, mod in [("inspect-ai==0.3.241", "inspect_ai"),
                 ("pandas==2.3.3", "pandas"),
                 ("pyarrow==24.0.0", "pyarrow")]:
    try:
        __import__(mod)
    except ImportError:
        subprocess.check_call([sys.executable, "-m", "pip", "install", "--quiet", pkg])
```

If `inspect_ai` is already on the user's Python, the loop is a no-op (each `__import__` succeeds, no install). Otherwise it installs the three pinned versions into whatever environment Posit discovered. This only runs once per session — subsequent `repl` calls reuse the loaded modules.

- **Claude Code**: the MCP starts automatically when the plugin is enabled. Use it directly.
- **Codex**: the MCP is bundled but may be either disabled in config or lazy-loaded rather than listed up front. Try the `tool_search` discovery step below first; if discovery still doesn't expose the tool, point the user at the README for the enable step.
- **Other agents installed via `npx skills add`** (Cursor, GitHub Copilot, Windsurf, Cline): the MCP isn't auto-registered. Point the user at the plugin README's per-agent JSON snippets (the one they paste depends on the agent — VS Code Copilot's schema is different from the rest). They restart their agent to pick it up.

If the tool still isn't visible after discovery, fall back to the stateless pattern below; don't block the analysis on getting the MCP set up. Common reasons the tool won't be visible:

- The user is on an agent we don't have a recipe for, or they prefer a different REPL MCP, or the bundled MCP just isn't installed in their config.
- **You're running as a subagent in Claude Code with a restricted `tools:` allowlist.** The bundled REPL reaches subagents only through the deferred-tool channel, loaded on demand via `ToolSearch`. Subagents with a wildcard (`tools: *`) or exclusion-style `tools:` declaration inherit it fine. But an agent defined with an *enumerated-include* `tools:` list that omits `ToolSearch` gets no deferred-tools list and no `ToolSearch` at all, so there's no channel to load the REPL schema and the MCP is unreachable. If that's you, either add `ToolSearch` (or use `tools: *`) to the agent definition, or fall back to the stateless reads below. If the parent needs persistent state, it can also just do the analysis itself rather than delegating to a tools-restricted subagent.

The pattern with the MCP available (after the dep-bootstrap above):

```python
# First call: load once
from inspect_ai.log import read_eval_log, read_eval_log_samples
header = read_eval_log("logs/run.eval", header_only=True)
# all_samples_required=False so partial / errored / cancelled logs don't raise.
samples = list(read_eval_log_samples("logs/run.eval", all_samples_required=False))

# Subsequent calls: just reference `header` and `samples`. State persists.
[s for s in samples if s.error is not None]
```

## Capturing the analysis to a runnable file (opt-in)

Exploratory analysis can leave the user with no durable artifact when the session ends. Offer an opt-in record they can re-run from scratch and keep.

For any **multi-turn or exploratory log-analysis session**, surface the capture option before or alongside the first substantive analysis step. This applies whether you use a persistent REPL MCP or fall back to stateless shell/Python commands. Do not skip this just because the REPL tool is unavailable.

**Ask via your agent's structured-question tool** (e.g., `AskUserQuestion` in Claude Code) when available. Don't bury the offer in a planning message, and don't pair the question with your own decision in the same message — phrasing like "I'll leave it off for now and continue" makes the user think the choice is already made and they often won't react. State the question on its own; if your agent can't pause for a reply, begin the analysis without capture (capture is off by default until the user explicitly opts in — they can say yes any later turn). If your agent doesn't expose a structured-question tool, ask a standalone, clearly-formatted question instead.

The question:

> "Want me to capture this analysis to a file you can keep and re-run? I can write a **Python script** (`.py`, runs end-to-end with `python …`), a **Jupyter notebook** (`.ipynb`, opens in JupyterLab with inline output), or skip it."

Three options:
- **Python script (`.py`)** *(default)* — a cleaned-up, linear script named `inspect-analysis-<timestamp>.py` that runs top-to-bottom from a fresh interpreter. Imports at the top, one commented section per keeper step, plots as `plt.savefig(...)`. Diffs cleanly in git, needs no Jupyter, and re-running it regenerates every result. Prefer this unless the user asks for a notebook.
- **Jupyter notebook (`.ipynb`)** — cells with captured stdout and inline plots, for users who live in JupyterLab.
- **No file** — skip capture; any persistent REPL state still works for follow-up questions during this session.

Whichever format they pick, two principles hold for both:

- **Keepers only.** Skip throwaway exploration (`print(dir(thing))` to figure out an API, error tracebacks you worked around, type-probe calls). Keep the steps that move the user's question forward.
- **Don't claim to have written something without actually running the tool call.** The most common failure mode is writing "captured that step" or "appended this as a cell" in your message *without* the underlying file-write / `append` call. If you state something landed, the matching tool call must appear in the same turn. If you're not sure whether it ran, read the file back before saying "done."

### If they pick a notebook (`.ipynb`)

**Find the helper script** (shipped inside this skill's directory). If multiple copies exist across plugin caches, pick the newest by mtime:

```bash
APPEND=$(find ~/.claude/plugins ~/.codex/plugins ~/.agents/skills -name append_to_notebook.py 2>/dev/null \
  | python3 -c "import sys, os; ps=[l.strip() for l in sys.stdin if l.strip()]; print(max(ps, key=os.path.getmtime, default=''))")
```

This searches all three install layouts (Claude Code plugin cache, Codex plugin cache, `npx skills add` under `~/.agents/skills/analyzing-logs/scripts/`) and picks the most recently updated copy so stale caches don't shadow the current version. If `$APPEND` is empty, tell the user and skip the capture; don't try to reconstruct the script inline.

**Create the notebook** (auto-named, prints the path):

```bash
NB=$(python3 "$APPEND" init)
```

**Append at natural pause points, in batches.** End of a logical step, end of a turn, before you summarise a finding — that's when you append. One Bash with several `cat <<EOF` + `append` invocations is cleaner than a separate Bash after every `repl` call, and it lets you pick keepers with hindsight (you'll know which exploratory blocks led somewhere useful).

**Verify cells actually landed.** Per the keepers-only / don't-claim-without-running rule above, the `append` here is the tool call that must appear in the same turn as any "appended this as a cell" claim. If you're not sure whether it ran, list the notebook cells (read the `.ipynb` with your file-read tool, or `python3 -c 'import json; print(len(json.load(open("'"$NB"'"))["cells"]))'`) before saying "done."

Use `mktemp -d` for the temp files so they're per-session (no name collisions across analysis sessions on the same machine, and not world-readable in shared `/tmp`):

```bash
TMP=$(mktemp -d)
cat > "$TMP/c1.py" <<'CODE'
<keeper code 1>
CODE
cat > "$TMP/c1.out" <<'OUT'
<captured stdout 1>
OUT
python3 "$APPEND" append "$NB" --code-file "$TMP/c1.py" --output-file "$TMP/c1.out"

cat > "$TMP/c2.py" <<'CODE'
<keeper code 2>
CODE
# ... etc

rm -rf "$TMP"   # clean up at end of batch
```

**Capture plots inline.** When a `repl` call produces a plot, save the figure to a PNG inside your `repl` code before calling `plt.show()`, then pass it to `append` via `--image-file`. Two small habits matter:

- **Suppress the noisy return value** with `_ =` or a trailing `;`. `plt.hist(...)` returns a `(counts, bins, patches)` tuple that can trip the oversized-output handler. We already configure `--oversized-output files` so it spills to disk instead of getting stuck in a pager, but suppressing is still cleaner.
- **Save to disk as the source of truth.** The inline render returned by Posit can come up empty on the very first `matplotlib` call in a session (font cache build races with the capture). `plt.savefig` always works, so write the file first and rely on `--image-file` for the captured notebook output. If you need to verify the plot before continuing, read the saved PNG via your file-read tool.

```python
import matplotlib.pyplot as plt
_ = plt.hist(tokens, bins=50)
plt.title("Token distribution")
plt.savefig("/tmp/cell-plot.png", dpi=80, bbox_inches="tight")
plt.show()  # also renders inline for vision-capable clients (best-effort)
```

then on the host:

```bash
python3 "$APPEND" append "$NB" --code-file "$TMP/cN.py" --output-file "$TMP/cN.out" --image-file /tmp/cell-plot.png
```

The PNG is embedded as a `display_data` output so the plot renders the moment the user opens the notebook in JupyterLab — they don't need to Run All to see it. The code cell still runs cleanly from a fresh kernel because `plt.savefig` regenerates the image. Use a fresh `--image-file` path per cell (e.g., `$TMP/cN.png`) if you're appending several plot cells in one batch.

**Cells must be self-contained.** The REPL state is for *exploration* — variables built across many back-and-forth calls. The notebook is the *curated record* — someone should be able to Run All from a fresh kernel and reproduce. So when appending a cell, inline the imports it needs, the variables it references, and the paths it reads. If a piece of state was built across five exploratory blocks in the REPL, collapse it into one self-contained cell in the notebook.

**What is and isn't captured.** Code, stdout, and saved PNG plots are captured. For plots, save the figure and pass the PNG via `--image-file`; the helper embeds it as `display_data` so it renders when the notebook opens. Not captured today: rich `display(df)` HTML or unsaved inline renders returned only by the REPL.

---

Last reviewed: 2026-06-26 against inspect_ai v0.3.241