# Inspect AI Pr Contribution

> Find, claim, implement, validate, and shepherd focused GitHub PRs for UKGovernmentBEIS/inspect_ai. Use when the user wants to contribute to Inspect AI, work on inspect_ai issues, fix an Inspect AI PR, sweep Inspect AI review feedback or CI, shares the Inspect AI author PR-list URL https://github.com/UKGovernmentBEIS/inspect_ai/pulls/deepujain, or mentions Inspect AI framework contribution work. Also use the one-word trigger "sweep" when the active repo/thread context identifies Inspect AI.

- Skill: `deepujain/inspect-ai-pr-contribution` (Agent Skill, multi-file: 3 files)
- Install (CLI): `npx skillmds@latest add deepujain/inspect-ai-pr-contribution`
- Raw SKILL.md: https://api.skillmd.com/api/skills/deepujain/inspect-ai-pr-contribution/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Docs & Writing
- Author: deepujain (https://skillmd.com/u/deepujain)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/deepujain/inspect-ai-pr-contribution

---


# Inspect AI PR Contribution Recipe

Also apply the shared OSS contribution quality protocol in
[../../references/contribution-quality.md](../../references/contribution-quality.md).
Project-specific instructions below override the shared protocol when they
conflict.

Use this recipe for [UKGovernmentBEIS/inspect_ai](https://github.com/UKGovernmentBEIS/inspect_ai), the Inspect AI evaluation framework. Follow the upstream contributor guide closely: agree direction in an issue when required, claim accepted work, branch, code, run `make check`, run `make test`, then open a ready-for-review PR.

## Project facts

- Local checkout: `/Users/dejain/nvidia/oss/worktrees/inspect-ai/inspect_ai`
- Upstream repo: `UKGovernmentBEIS/inspect_ai`
- Default branch: `main`
- Fork/head owner: `deepujain`
- Python: 3.10+
- Dev setup: `pip install -e ".[dev]"`
- Optional hooks: `make hooks`
- Full validation: `make check` and `make test`
- Docs: https://inspect.aisi.org.uk/
- CI: ruff, mypy, pytest, package build on Python 3.10 and 3.11
- Viewer/frontend CI: `src/inspect_ai/_view/ts-mono` pnpm build, generated schema/types, checked-in `dist`
- Sandbox-tools CI: special gates for `src/inspect_sandbox_tools`, `src/inspect_ai/tool/**`, and `sandbox_tools_version.txt`
- Current contribution policy: check `AGENTS.md`, `CONTRIBUTING.md`, and `.github/scripts/pr_gate.py` live. Qualified contributors are listed by account id; every other contributor needs an `accepted` issue before coding except trivial docs-only fixes. Upstream commit `cf70b2404` removed the former per-author open-PR cap because accepted issues now establish maintainer-approved demand; the Skippy queue target remains an independent local target, not an upstream maximum. Draft PRs are not reviewed, and unrequested core functionality should become an extension unless maintainers accept it for core.

## Required live reconnaissance

Before creating a new contribution or updating this skill, inspect the live repo state:

```bash
gh issue list --repo UKGovernmentBEIS/inspect_ai --state open --limit 100
gh pr list --repo UKGovernmentBEIS/inspect_ai --state open --limit 100
gh pr list --repo UKGovernmentBEIS/inspect_ai --author deepujain --state open
```

Treat [deepujain's Inspect AI PR list](https://github.com/UKGovernmentBEIS/inspect_ai/pulls/deepujain)
as the canonical starting point for "my Inspect AI PRs", open-PR sweeps, CI
maintenance, and review follow-up. When this URL is supplied, scope the task to
Inspect AI and inspect every open PR before deciding that no action is needed.

For open-PR sweeps, fetch structured state before editing:

First reconcile the previous or recent authored PR set with the current open
set. For every PR that disappeared, query its exact state and merge/close
timestamps, then inspect final comments, reviews, timeline, linked issue,
overlapping PRs, and any replacement commit. Record merged PRs as merged. For a
PR closed without merge, establish whether it was duplicate, superseded, out of
scope, policy-blocked, abandoned, or unresolved, and say whether the
contribution survived in another PR. If the closure yields a reusable testing,
design, review, or workflow lesson, add the smallest durable rule at the correct
place in this skill, validate it, and commit/push the skill repository. Do not
overfit unexplained closures; report `no skill change needed` when there is no
reusable lesson. Include a departed-PR table before the open-PR table whenever
anything merged or closed since the previous sweep.

Use this sweep report shape every time:

- **Departed PRs** - include when any recent authored PR is no longer open.
- **Open PR Sweep** - include every current authored open PR, or `none open`.
- **Lessons Learned** - always include a table with `Lesson`, `Evidence`,
  `Skill Update`, and `Action` columns.

In `Lessons Learned`, distinguish actual reusable process lessons from one-off
events. When a lesson is reusable, update the correct section of this project
skill in the same run, validate the diff, commit, and push only the skill
change. When no rule changed, include `no skill change needed` with the reason.
Do not append loose notes to the end of the skill; put each rule where future
Inspect AI work will naturally use it.

### Sweep replenishment

Treat `sweep` as both PR maintenance and controlled replenishment, whether it
is invoked by the user or by a scheduled task.

- On every sweep, after inspecting all authored open PRs, evaluate whether the
  repo's contribution gates allow a new issue PR. Do this even when no authored
  PR merged since the previous sweep.
- Respect the live gate in this skill: non-qualified contributors need an
  `accepted` issue before coding except for trivial docs-only fixes. Do not
  infer an upstream open-PR cap from the configured Skippy queue target.
- For each slot below the configured target that contributor policy allows,
  pick a well-scoped accepted issue using the normal Inspect AI issue-selection
  rules in this skill and run the full new-PR recipe in the same sweep.
- A failing/rerunning CI job, unresolved review, merge conflict, stale base, or
  unclear maintainer acceptance on one authored PR is a maintenance workstream
  for that PR, not a replenishment stop. Maintain it, then keep screening and
  publishing independent qualified slots. Stop below the target only for a
  verified maximum, a shared policy/publication restriction, or no qualified
  non-overlapping accepted issue after the complete screen.
- Report the outcome explicitly in the sweep output: `opened new PR`,
  `issue selected, PR in progress`, or `replenishment skipped` with the exact
  blocker such as the configured queue target reached, issue not accepted, a shared
  publication restriction, duplicate risk, or no strong candidate.

```bash
gh pr list --repo UKGovernmentBEIS/inspect_ai --author deepujain --state open --limit 100 \
  --json number,title,url,headRefName,isDraft,mergeable,reviewDecision,statusCheckRollup,updatedAt
```

Then search overlap for the issue and affected subsystem:

```bash
gh search prs --repo UKGovernmentBEIS/inspect_ai '<issue-number> in:title,body' --state open
gh search prs --repo UKGovernmentBEIS/inspect_ai '<subsystem keyword or error text>' --state open
```

Also inspect current tooling files before deciding validation:

- `CONTRIBUTING.md`
- `AGENTS.md`
- `.github/qualified.yml`
- `CLAUDE.md`
- `Makefile`
- `pyproject.toml`
- `CHANGELOG.md`
- `.github/workflows/build.yml`
- `.github/workflows/log_viewer.yml` when touching viewer or schema files
- `src/inspect_sandbox_tools/CLAUDE.md` when touching sandbox tools

As of 2026-05-07, active open-issue themes included structured/plain log output, eval run config files, OpenAI computer-tool model gating, score/metric semantics, log/sample memory filtering, cancelled eval aggregation, Bedrock/Anthropic provider behavior, HF and CSV/JSON dataset retry/caching, docs link fixes, and prompt logprobs. Several already had active PRs, including MMLU docs (#3834), aggregate metrics (#3735), HF retry (#3733), cancelled eval results (#3750), and Bedrock Claude sampling params (#3766). Refresh before acting and avoid duplicate PRs.

Current open PR overlap has been heavy around model events/providers, `_view` OpenAPI/schema/frontend, `tests/conftest.py`, extension tests, S3 log writing, scorer metrics, dataset sources, docs/evals generation, and sandbox/local tools. Treat these as hot areas: search open PRs and changed files before editing them.

When calibrating contribution style, also scan recent merged PRs from other contributors. Extract reusable patterns only: PR descriptions that state current behavior, new behavior, compatibility, design choices, and validation; file-level conventions such as changelog entries, generated viewer assets, or sandbox tool version bumps; and review comments that reveal reviewer preferences. Do not copy empty maintainer-only templates or tool-generated footers.

## Contribution loop

1. **Pick and claim the issue.**
   - Read the current `AGENTS.md` and `CONTRIBUTING.md` contribution gates before opening or reopening PRs.
   - Determine contributor policy live: compare `gh api users/deepujain --jq .id` with `.github/qualified.yml`. If the account is not listed, do not code or open a PR until the issue is labeled `accepted`, except for trivial docs-only fixes under the upstream size limit; prior merged PRs do not bypass this gate.
   - Treat the configured healthy-open target as Skippy queue policy, not an upstream cap. Refresh `AGENTS.md`, `CONTRIBUTING.md`, and `.github/scripts/pr_gate.py` before relying on any contributor maximum.
   - Prefer `good first issue` or clearly scoped bugs/docs gaps unless the user asks for a larger change.
   - Treat `good first issue` as accepted, matching the upstream guide.
   - Before coding, re-evaluate value: require a demonstrated problem, reproduction, failing test, or maintainer-accepted direction. If the need is speculative, file or update an issue with evidence instead.
   - Prefer issues with a concrete failure mode, a narrow compatibility-preserving fix, and a regression test that can fail on the old behavior.
   - Good targets include provider request/argument precedence, CLI type or option mismatches, scorer/log observability fixes, retry/recovery edge cases, docs gaps tied to existing APIs, and narrowly scoped test stability fixes.
   - For new functionality in providers, tools, scorers, metrics, solvers, storage backends, or example evals, default to an extension package unless an accepted issue explicitly asks for an Inspect core change.
   - Read the full issue, comments, linked PRs, and affected code.
   - Search open PRs by issue number, title keywords, stack trace text, and subsystem names.
   - Comment on the issue to claim it before investing real implementation time, matching the project contributor guidance.
   - If there is no issue for the intended change, open or ask for an issue first unless the user explicitly says to proceed with a tiny docs/test-only fix.

2. **Sync before changing files.**

   ```bash
   cd /Users/dejain/nvidia/oss/worktrees/inspect-ai/inspect_ai
   git fetch origin main
   git switch main
   git pull --ff-only origin main
   git switch -c issue-NN-short-topic
   ```

   If the local checkout is missing, clone upstream first and add the `deepujain` fork remote before pushing.

3. **Implement narrowly.**
   - Follow `CONTRIBUTING.md`, `Makefile`, `pyproject.toml`, and nearby code patterns.
   - Main package code lives under `src/inspect_ai`; tests live under `tests`.
   - Read `CLAUDE.md` before coding. It requires strict typing, Google-style public API docstrings, context-rich errors, and repo-specific async/file-path patterns.
   - Async tests run under anyio's asyncio and trio backends. Do not use `@pytest.mark.asyncio`; use `anyio.sleep`, `anyio.Event`, and `inspect_ai._util._async.tg_collect()` rather than raw asyncio helpers unless the local pattern says otherwise.
   - File-path code must support `s3://`, `file://`, and plain local paths; use `filesystem()` and `local_path()` from `inspect_ai._util.file`.
   - Public APIs, CLI behavior, solver/scorer/model interfaces, log formats, sandbox/tooling behavior, and docs examples need tests or explicit manual validation.
   - Keep compatibility in mind: Inspect AI is a framework used by downstream eval projects, so avoid broad behavior changes unless the issue calls for them.
   - Preserve existing precedence rules. New env vars, defaults, headers, or config fallbacks should not override explicit user/model args.
   - For persistence or filesystem fixes, use atomic writes, cleanup-on-exception, bounded growth, and recovery from corrupt optional state where practical.
   - For larger internal refactors, split mechanical extraction from behavior changes when possible and call out stacked-base or dependency context in the PR body.
   - Add a concise `CHANGELOG.md` entry for user-facing fixes, provider changes, docs-visible features, CLI changes, viewer changes, and sandbox tool changes. Write changelog entries outcome-first: describe the user-visible behavior and affected tools/APIs, not the internal mechanism. Skip changelog only for pure tests/internal maintenance and say why in the PR body if it might be questioned.
   - Put new changelog entries under the first `## Unreleased` section. If a fresh upstream release removed that section, recreate `## Unreleased` at the top of `CHANGELOG.md` and move the PR's entry there before pushing; the `entries-under-unreleased` CI check fails otherwise.
   - Avoid private tool-generated references in code, commit messages, PR titles, PR bodies, comments, branch names, or any other GitHub-visible text.
   - Do not add any source-agent, tool-name, "generated by", automation, or bracketed provenance labels anywhere in contributor-facing artifacts.
   - If touching `_view`, regenerate schema/types and build/check the frontend output. CI will fail if `inspect-openapi.json`, generated types, or `src/inspect_ai/_view/dist` are stale.
   - For viewer source fixes that live in `src/inspect_ai/_view/ts-mono`, open and maintain the ts-mono PR first. Do not open an Inspect AI submodule/dist bump that points at an unmerged ts-mono commit unless a maintainer explicitly asks for pre-merge staging; wait for the ts-mono commit to land on main, or let maintainers open the Inspect AI bump.
   - For sandbox fallback fixes, preserve loud provider-level errors unless maintainers ask otherwise. Do not turn sandbox capability limits, such as unsupported `user=` exec, into quiet tool-output feedback. In the PR body, state whether rootless behavior, user switching, tool visibility, and injectable-versioning are changed or intentionally left unchanged.
   - If touching sandbox tools or injectable support, check whether `src/inspect_ai/tool/_sandbox_tools_utils/sandbox_tools_version.txt` must be bumped and run the relevant slow tool tests when practical. Put `_sandbox_tools_utils` regression tests under `tests/tools/sandbox_tools_utils/`, and prefer outcome assertions over exact ordered sandbox `exec()` call sequences unless the sequence itself is the contract.

4. **Validate the feature specifically.**
   - First run the narrow test or command that proves the issue requirement.
   - Add or update tests for behavior changes.
   - When practical, verify the targeted regression fails before the fix and passes after it; mention that in the PR body.
   - Run the adjacent subsystem test file or focused test group, not only the single new test, when behavior touches shared scorer, provider, log, CLI, viewer, or sandbox paths.
   - For CLI work, run the relevant `inspect ...` command or help path.
   - For docs examples, verify imports and snippets where practical.
   - In the PR body, do not list only `make check` and `make test`; also explain which targeted validation proves the requested behavior.

5. **Run checks.**

   ```bash
   pytest tests/path/to/test_file.py::test_name -v
   make check
   make test
   ```

   Run `uv run ruff format --check` (or `ruff format` then commit) on touched Python
   files before pushing. CI's `pre-commit` and `ruff` jobs fail on formatting-only
   drift such as missing blank lines before new test functions (#5181, 2026-09-02).

   `make check` runs ruff and mypy. `make test` runs pytest. If the full suite is blocked by environment or unrelated failures, report the exact blocker and preserve focused feature evidence.

   CI uses `uv` even though the contributor guide also documents `pip install -e ".[dev]"`. To mirror CI more closely in a fresh checkout, use:

   ```bash
   uv venv
   uv pip install .[dev]
   uv run pytest tests/path/to/test_file.py::test_name -v
   ```

   Additional validation by area:

   - Before validating changes to model providers, sandbox/tool code, agents, or async plumbing, read the live `.claude/skills/slow-tests/SKILL.md` and run the applicable gated classes (`slow`, live provider API, flaky, and/or trio). Plain `pytest` skips these classes, and PR CI does not run live provider tests. Add a `### Slow tests` section under "Other information" with each exact command, passed and skipped counts by class/provider, and every environment limit such as missing keys, Docker, model access, or a local server. Never count a skip as an executed test.
   - Docs: render or at least inspect the affected Quarto page and generated data path.
   - CLI: run the exact `inspect ...` command or `--help` path affected by the issue.
   - Model/provider: test request construction, parsing, header merging, default/fallback precedence, and opt-out behavior without requiring secrets; include provider-specific tests.
   - Scorer/log/recovery: test observability fields, event/sample reconstruction, retry carry-forward, legacy log compatibility, and dataframe/export behavior when schema changes.
   - Viewer: run `python src/inspect_ai/_view/schema.py`, build/check `src/inspect_ai/_view/ts-mono`, and include changed `inspect-openapi.json`, generated types/submodule updates, and `src/inspect_ai/_view/dist` when frontend output can change.
   - For ts-mono viewer PR maintenance, rerun `pnpm install --frozen-lockfile` after fetching or rebasing over dependency updates before trusting typecheck or lint failures; stale dependencies can produce false markdown/typescript errors.
   - ACP/TUI/interactivity: add unit tests plus E2E-style tests for protocol/session routing and textual widgets when UI state or live agent behavior changes.
   - Sandbox tools: bump `sandbox_tools_version.txt` when injectable code changes and run `uv run pytest --runslow -m slow tests/tools/ -x` or the narrower sandbox-tool tests when the local environment can support it.

6. **Commit and push.**
   - Commit only files needed for this issue.
   - Use the author `Deepak Jain <deepujain@gmail.com>`.
   - Use a clear maintainer-facing commit message, for example `fix: handle empty task list in CLI`.
   - Do not add generated or local-only artifacts unless the repo expects them.

7. **Open the PR.**

   ```bash
   gh pr create \
     --repo UKGovernmentBEIS/inspect_ai \
     --base main \
     --head deepujain:issue-NN-short-topic
   ```

   Open a PR only when it is intended for review: linked to accepted work when required, locally validated, current with `main`, and within the configured Skippy queue target unless the user changes it. Do not open draft PRs for work that should be reviewed; maintainers do not review draft PRs, and repeated draft rebases consume CI without advancing mergeability. If work is not ready, keep it local or in the fork without an upstream PR.

   Inspect AI requires contributor tooling disclosure in PR descriptions. If the user's public-text rule forbids naming or describing tooling, stop before opening the PR and ask for approved wording or do not open the PR. Never omit a required disclosure, invent a false disclosure, or publish private tool names against the user's instruction.

   Use a clean maintainer-facing PR title with a repository-style public prefix and no private provenance prefix or attribution.

   Prefer the repository PR template, filled in with concrete maintainer-facing detail. The strongest Inspect AI PRs spell out the previous behavior, the new behavior, whether the change is breaking, and the exact validation that proves the issue is fixed. Do not leave template headings empty or invent extra template categories unless the upstream template changes.

   A concise custom structure is also acceptable when it is clearer than the template: Summary, Background/root cause, Changes, Design choices, Breaking changes, and Validation. This is useful for sandbox-tool, recovery, or larger feature work where reviewers need the failure mechanism and tradeoffs.

   PR body shape:

   ```markdown
   ## This PR contains:
   - [ ] New features
   - [ ] Changes to dev-tools e.g. CI config / github tooling
   - [ ] Docs
   - [ ] Bug fixes
   - [ ] Code refactor

   ### What is the current behavior? (You can also link to an open issue here)

   ...

   ### What is the new behavior?

   - ...

   ### Does this PR introduce a breaking change? (What changes might users need to make in their application due to this PR?)

   No. ...

   ### Other information:

   Validation:
   - Targeted:
     - `uv run pytest ... -v`
       - Explain the regression or behavior this proves.
   - Regression:
     - `uv run make check`
     - `uv run make test`

   CI/CD coverage expected:
   - State which standard workflows should cover the change, and note any area-specific workflow such as log viewer or sandbox tools when touched.

   Closes #NN.
   ```

   Quality bar for titles and descriptions:
   - Use a short maintainer-facing title with Inspect AI's visible conventions, such as `fix: ...`, `feat: ...`, `docs: ...`, or `Provider: ...`.
   - Avoid private provenance attribution and process notes.
   - Mention the changelog entry when the change is user-facing, or explicitly say "No changelog entry: test-only/internal-only" when that is the honest reason.
   - For docs-only PRs, describe the implementation/API checks used to verify the examples instead of pretending there is a new runtime test.
   - For bug fixes, include a targeted regression test when practical and say whether it fails on `main` if you verified that.
   - For provider changes, validate request construction/parsing without secrets where possible, and mention any live or optional-provider tests separately.
   - For viewer changes, mention schema/type regeneration and `pnpm --filter inspect build`/dist validation when applicable.
   - For sandbox-tool changes, mention the version bump gate and slow tool tests when applicable.
   - For sandbox self-check changes (`src/inspect_ai/util/_sandbox/self_check.py`
     or `tests/tools/test_sandbox_docker_and_local.py`), expect
     `slow-tool-tests-dev` when CI selects `tests/tools/`. Poll it to terminal
     state. Treat the self-check as a downstream provider conformance contract:
     do not tighten it until the linked issue is `accepted`, the intended
     provider-side behavior is settled, and affected provider maintainers can
     validate it. Do not mask a real product bug with a strict xfail, especially
     one tied to an unpinned platform image; fix the provider first, then make
     the conformance check enforce the fixed behavior without an xfail.
   - When a PR includes agent-review disclosure, include the reviewer model,
     number of passes, and concrete findings (including zero findings). A bare
     tool name or generic claim of AI review is not useful review evidence.

8. **Shepherd the PR.**
   - Inspect CI, review comments, and requested changes after pushing.
   - If a sweep finds an otherwise healthy PR has become `CONFLICTING`/`DIRTY`,
     rebase it promptly on upstream `main`, preserve existing top-of-file
     changelog entries plus the PR's `## Unreleased` entry, rerun focused tests
     and `uv run make check`, force-push with lease, then re-check CI.
   - Do not keep closed, unaccepted, over-limit, or draft PRs alive with routine rebase churn. If a PR is not eligible for review, stop CI-triggering pushes and move the discussion back to the issue with concrete evidence.
   - If a maintainer says draft PRs are not reviewed, convert only genuinely review-ready, policy-compliant PRs to ready-for-review. Otherwise leave them closed/local and prepare the accepted-issue path first.
   - If an accepted issue came from a maintainer-closed draft PR and GitHub refuses to reopen it, recreate a ready-for-review PR from a fresh rebased branch rather than fighting the closed PR.
   - If maintainers close a batch for policy reasons, do not reopen by default. Classify it as policy-blocked, record linked issue state and any replacement PRs, then select at most 4 issues to pursue through the accepted-issue path.
   - Treat Build and Build Log Viewer jobs as expected PR gates: ruff, mypy, pre-commit, package inspection, Python 3.10/3.11 tests, schema/type checks, submodule-on-main, and dist validation. Do not ignore a required failure just because unrelated deploy-style checks can fail outside the PR gate.
   - Treat `submodule-on-main` as a blocker for Inspect AI viewer bump PRs, not as a standing expected failure. If it fails because the ts-mono source PR is unmerged, keep the source PR moving and wait for the merged commit before reopening or recreating the Inspect AI bump.
   - If GitHub refuses to rerun failed jobs because the account lacks admin rights, and the failure is clearly unrelated or flaky after log inspection, refresh CI with a content-preserving `git commit --amend --no-edit` plus `git push --force-with-lease` rather than adding a throwaway commit. State the failed test and why no code change was made.
   - Check top-level comments, reviews, and inline review comments. Recent merged PRs often had little public discussion, but actionable inline comments focused on small correctness/style details such as temp-file cleanup, exception safety, and keeping docstrings/descriptions intact.
   - Fix actionable feedback on the current head, rerun relevant validation, and leave a short factual status comment.
   - Prefer applying reviewer suggestions directly when they are low-risk, then add a follow-up commit with the exact validation rerun. If a maintainer says they will make minor changes before merge, still keep the branch green and avoid further churn.
   - If maintainer feedback changes the design direction, follow it instead of defending the first implementation.

## Issue selection preferences

- Trigger phrase: **"sweep"** when the active repo/thread context identifies Inspect AI.

Good Inspect AI contribution targets include small CLI bugs, docs corrections, model/provider integration fixes, scorer/solver behavior gaps, logging/viewer regressions, sandbox/tool edge cases, and narrowly scoped tests for existing behavior.

Avoid large API redesigns, new provider integrations requiring secrets, eval semantics that need maintainer product judgment, broad viewer/frontend churn without generated artifacts, and changes that could silently alter downstream evaluation results without a clear migration story.