# Analyses Snapshot Testing

> Conventions for the analyses snapshot testing framework in analyses-snapshot-testing/. Use when working with protocol analysis snapshots, adding protocols, updating snapshots, or running snapshot tests.

- Skill: `opentrons/analyses-snapshot-testing` (Agent Skill)
- Install (CLI): `npx skillmds@latest add opentrons/analyses-snapshot-testing`
- Raw SKILL.md: https://api.skillmd.com/api/skills/opentrons/analyses-snapshot-testing/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: opentrons (https://skillmd.com/u/opentrons)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/opentrons/analyses-snapshot-testing

---


# Analyses Snapshot Testing Instructions

## Overview

The `analyses-snapshot-testing` directory validates that protocol analysis output remains consistent across code changes by comparing JSON results against committed snapshots.

- **Regression testing**: Detect unintended changes in protocol analysis behavior
- **Protocol validation**: Ensure protocols analyze correctly for OT-2
- **CI/CD**: Automated testing in GitHub Actions with matrix-based parallel execution
- **Snapshot management**: Track expected output and flag deviations

## Architecture

### Core Components

1. **Protocol Files** (`files/protocols/`) — testing protocols organized by source (standard, Protocol Designer exports, Protocol Library imports, generators)
2. **Analysis Engine** (`automation/analyze.py`) — runs protocol analysis in subprocess, 120-second timeout, JSON output
3. **Snapshot Storage** (`tests/__snapshots__/`) — committed JSON snapshots managed by `syrupy` with custom JSON extension
4. **Test Suite** (`tests/`) — `analyses_snapshot_test.py` (main), `audit_snapshot_test.py` (audit), `custom_json_snapshot_extension.py` (serialization)
5. **Audit** (`automation/audit_snapshots.py`) — validates snapshots: `OT2_S` protocols must have zero errors; `OT2_X` protocols are expected to have errors
6. **Protocol Registry** (`automation/data/`) — `protocols.py` (auto-generated), `protocols_with_overrides.py` (manual), `protocol_registry.py` (combined)
7. **CI/CD** (`citools/`, `.github/workflows/`) — matrix-based parallel execution via Docker

## Protocol Naming Convention

```markdown
{Robot}_{Status}_{Version}_{Source}_{Pipettes}_{Modules}_{Overrides}\_{Description}
```

- **Robot**: `OT2`
- **Status**: `S` (Success) or `X` (Failure expected)
- **Version**: API version (e.g., `v2_19`) or `PD` (Protocol Designer)
- **Source** (optional): `PL_` (Protocol Library) or `MPL_` (Manual Protocol Library)
- **Pipettes**: e.g., `P1000M_P50M`
- **Modules**: `GRIP`, `HS`, `MM`, `MB`, `TC`, `TM`
- **Overrides**: `Overrides` if protocol has parameter overrides
- **Description**: Max 25 characters

Example: `OT2_S_v2_19_P300M_P20S_HS_TC_TM_SmokeTestV3.py`

## Development Workflow

### Adding New Protocols

1. Create protocol file in `files/protocols/` following naming convention
2. Run `make prep` (auto-updates `automation/data/protocols.py`, displays snapshot command)
3. Generate snapshots: `make snapshot-test-update PROTOCOL_NAMES=YourProtocolName OVERRIDE_PROTOCOL_NAMES=none`
4. Commit protocol file + snapshot + updated `protocols.py`

### Adding Override Protocols

1. Create generator in `files/protocols/generators/`
2. Add entry to `automation/data/protocols_with_overrides.py`
3. Generate snapshots: `make snapshot-test-update PROTOCOL_NAMES=none OVERRIDE_PROTOCOL_NAMES=YourOverrideProtocol`
4. Commit generator + snapshot + updated registry

### Updating Snapshots

```bash
make snapshot-test-update                                          # All
make snapshot-test-update PROTOCOL_NAMES="P1,P2" OVERRIDE_PROTOCOL_NAMES=none  # Specific
```

### Running Tests Locally

```bash
make snapshot-test                                                   # All (reduced verbosity)
make snapshot-test PROTOCOL_NAMES=OT2_S_v2_7_P20S_None_Walkthrough OVERRIDE_PROTOCOL_NAMES=none  # Specific
uv run python -m pytest -k analyses_snapshot_test -vv --tb=short    # Verbose debugging
```

## Code Modification Guidelines

### When Modifying Analysis Logic (`api/`, `shared-data/`)

1. Run snapshot tests to detect changes
2. Review diffs carefully
3. Update snapshots only if changes are intentional
4. Document breaking changes in commit messages

### Protocol Registry (`automation/data/protocols.py`)

- **DO NOT** manually edit — auto-generated by `make prep`
- Regenerate after adding/removing protocols

### Snapshot Extension (`tests/custom_json_snapshot_extension.py`)

- Normalizes timestamps, IDs, file paths for stable comparisons
- Add new normalization rules to `replacement_patterns`
- Add new ID-like fields to `id_keys_to_replace`

## Environment Setup

**Prerequisites**: Python 3.12, **uv**, Node.js/pnpm (for prettier)

```bash
cd analyses-snapshot-testing
make setup    # Creates uv venv and installs dependencies
```

The setup script (`bootstrap_uv_env.py`) installs `api` and `shared-data` as editable packages so analysis uses local code.

## Common Commands

```bash
# Setup
make setup                    # Initial environment setup
make teardown                 # Remove virtual environment

# Testing
make snapshot-test            # Run all snapshot tests
make snapshot-test-update     # Update all snapshots
make snapshot-audit-test      # Validate snapshot audit metadata

# Protocol Management
make prep                     # Regenerate protocol registry
make generate-protocols       # Generate override protocols

# Formatting
make format                   # Format Python + Markdown
make ruff                     # Format and lint Python

# CI Simulation
make gen-chunks               # Generate protocol chunks
make analyze-chunk CHUNK=chunk_0.json  # Analyze specific chunk
```

## CI/CD Integration

Workflow (`analyses-snapshot-test.yaml`) triggers on:

- PRs affecting `api/`, `shared-data/`, or this directory
- Scheduled daily at 7:26 AM UTC
- Manual dispatch

PR label `gen-analyses-snapshot-pr` auto-opens a PR with updated snapshots on failure.

## Troubleshooting

- **Snapshot mismatch**: Review diff, update if intentional (`make snapshot-test-update PROTOCOL_NAMES=...`), fix code if not
- **Protocol not found**: Check registry, run `make prep`, verify file exists and follows naming convention
- **Analysis timeout** (120s): Simplify protocol or increase timeout in `automation/analyze.py`
- **Environment issues**: Run `make setup`, ensure `uv` is installed