Inspect AI Pr Contribution

Inspect AI PR Contribution Recipe

Also apply the shared OSS contribution quality protocol in ../../references/contribution-quality.md. Project-specific instructions below override the shared protocol when they conflict.

Use this recipe for UKGovernmentBEIS/inspect_ai, the Inspect AI evaluation framework. Follow the upstream contributor guide closely: agree direction in an issue when required, claim accepted work, branch, code, run make check, run make test, then open a ready-for-review PR.

Project facts

• Local checkout: /Users/dejain/nvidia/oss/worktrees/inspect-ai/inspect_ai
• Upstream repo: UKGovernmentBEIS/inspect_ai
• Default branch: main
• Fork/head owner: deepujain
• Python: 3.10+
• Dev setup: pip install -e ".[dev]"
• Optional hooks: make hooks
• Full validation: make check and make test
• Docs: https://inspect.aisi.org.uk/
• CI: ruff, mypy, pytest, package build on Python 3.10 and 3.11
• Viewer/frontend CI: src/inspect_ai/_view/ts-mono pnpm build, generated schema/types, checked-in dist
• Sandbox-tools CI: special gates for src/inspect_sandbox_tools, src/inspect_ai/tool/**, and sandbox_tools_version.txt
• Current contribution policy: check AGENTS.md, CONTRIBUTING.md, and .github/scripts/pr_gate.py live. Qualified contributors are listed by account id; every other contributor needs an accepted issue before coding except trivial docs-only fixes. Upstream commit cf70b2404 removed the former per-author open-PR cap because accepted issues now establish maintainer-approved demand; the Skippy queue target remains an independent local target, not an upstream maximum. Draft PRs are not reviewed, and unrequested core functionality should become an extension unless maintainers accept it for core.

Required live reconnaissance

Before creating a new contribution or updating this skill, inspect the live repo state:

gh issue list --repo UKGovernmentBEIS/inspect_ai --state open --limit 100
gh pr list --repo UKGovernmentBEIS/inspect_ai --state open --limit 100
gh pr list --repo UKGovernmentBEIS/inspect_ai --author deepujain --state open

Treat deepujain's Inspect AI PR list as the canonical starting point for "my Inspect AI PRs", open-PR sweeps, CI maintenance, and review follow-up. When this URL is supplied, scope the task to Inspect AI and inspect every open PR before deciding that no action is needed.

For open-PR sweeps, fetch structured state before editing:

First reconcile the previous or recent authored PR set with the current open set. For every PR that disappeared, query its exact state and merge/close timestamps, then inspect final comments, reviews, timeline, linked issue, overlapping PRs, and any replacement commit. Record merged PRs as merged. For a PR closed without merge, establish whether it was duplicate, superseded, out of scope, policy-blocked, abandoned, or unresolved, and say whether the contribution survived in another PR. If the closure yields a reusable testing, design, review, or workflow lesson, add the smallest durable rule at the correct place in this skill, validate it, and commit/push the skill repository. Do not overfit unexplained closures; report no skill change needed when there is no reusable lesson. Include a departed-PR table before the open-PR table whenever anything merged or closed since the previous sweep.

Use this sweep report shape every time:

• Departed PRs - include when any recent authored PR is no longer open.
• Open PR Sweep - include every current authored open PR, or none open.
• Lessons Learned - always include a table with Lesson, Evidence, Skill Update, and Action columns.

In Lessons Learned, distinguish actual reusable process lessons from one-off events. When a lesson is reusable, update the correct section of this project skill in the same run, validate the diff, commit, and push only the skill change. When no rule changed, include no skill change needed with the reason. Do not append loose notes to the end of the skill; put each rule where future Inspect AI work will naturally use it.

Sweep replenishment

Treat sweep as both PR maintenance and controlled replenishment, whether it is invoked by the user or by a scheduled task.

• On every sweep, after inspecting all authored open PRs, evaluate whether the repo's contribution gates allow a new issue PR. Do this even when no authored PR merged since the previous sweep.
• Respect the live gate in this skill: non-qualified contributors need an accepted issue before coding except for trivial docs-only fixes. Do not infer an upstream open-PR cap from the configured Skippy queue target.
• For each slot below the configured target that contributor policy allows, pick a well-scoped accepted issue using the normal Inspect AI issue-selection rules in this skill and run the full new-PR recipe in the same sweep.
• A failing/rerunning CI job, unresolved review, merge conflict, stale base, or unclear maintainer acceptance on one authored PR is a maintenance workstream for that PR, not a replenishment stop. Maintain it, then keep screening and publishing independent qualified slots. Stop below the target only for a verified maximum, a shared policy/publication restriction, or no qualified non-overlapping accepted issue after the complete screen.
• Report the outcome explicitly in the sweep output: opened new PR, issue selected, PR in progress, or replenishment skipped with the exact blocker such as the configured queue target reached, issue not accepted, a shared publication restriction, duplicate risk, or no strong candidate.

gh pr list --repo UKGovernmentBEIS/inspect_ai --author deepujain --state open --limit 100 \
  --json number,title,url,headRefName,isDraft,mergeable,reviewDecision,statusCheckRollup,updatedAt

Then search overlap for the issue and affected subsystem:

gh search prs --repo UKGovernmentBEIS/inspect_ai '<issue-number> in:title,body' --state open
gh search prs --repo UKGovernmentBEIS/inspect_ai '<subsystem keyword or error text>' --state open

Also inspect current tooling files before deciding validation:

• CONTRIBUTING.md
• AGENTS.md
• .github/qualified.yml
• CLAUDE.md
• Makefile
• pyproject.toml
• CHANGELOG.md
• .github/workflows/build.yml
• .github/workflows/log_viewer.yml when touching viewer or schema files
• src/inspect_sandbox_tools/CLAUDE.md when touching sandbox tools

As of 2026-05-07, active open-issue themes included structured/plain log output, eval run config files, OpenAI computer-tool model gating, score/metric semantics, log/sample memory filtering, cancelled eval aggregation, Bedrock/Anthropic provider behavior, HF and CSV/JSON dataset retry/caching, docs link fixes, and prompt logprobs. Several already had active PRs, including MMLU docs (#3834), aggregate metrics (#3735), HF retry (#3733), cancelled eval results (#3750), and Bedrock Claude sampling params (#3766). Refresh before acting and avoid duplicate PRs.

Current open PR overlap has been heavy around model events/providers, _view OpenAPI/schema/frontend, tests/conftest.py, extension tests, S3 log writing, scorer metrics, dataset sources, docs/evals generation, and sandbox/local tools. Treat these as hot areas: search open PRs and changed files before editing them.

When calibrating contribution style, also scan recent merged PRs from other contributors. Extract reusable patterns only: PR descriptions that state current behavior, new behavior, compatibility, design choices, and validation; file-level conventions such as changelog entries, generated viewer assets, or sandbox tool version bumps; and review comments that reveal reviewer preferences. Do not copy empty maintainer-only templates or tool-generated footers.

Contribution loop

1. Pick and claim the issue.

   • Read the current AGENTS.md and CONTRIBUTING.md contribution gates before opening or reopening PRs.
   • Determine contributor policy live: compare gh api users/deepujain --jq .id with .github/qualified.yml. If the account is not listed, do not code or open a PR until the issue is labeled accepted, except for trivial docs-only fixes under the upstream size limit; prior merged PRs do not bypass this gate.
   • Treat the configured healthy-open target as Skippy queue policy, not an upstream cap. Refresh AGENTS.md, CONTRIBUTING.md, and .github/scripts/pr_gate.py before relying on any contributor maximum.
   • Prefer good first issue or clearly scoped bugs/docs gaps unless the user asks for a larger change.
   • Treat good first issue as accepted, matching the upstream guide.
   • Before coding, re-evaluate value: require a demonstrated problem, reproduction, failing test, or maintainer-accepted direction. If the need is speculative, file or update an issue with evidence instead.
   • Prefer issues with a concrete failure mode, a narrow compatibility-preserving fix, and a regression test that can fail on the old behavior.
   • Good targets include provider request/argument precedence, CLI type or option mismatches, scorer/log observability fixes, retry/recovery edge cases, docs gaps tied to existing APIs, and narrowly scoped test stability fixes.
   • For new functionality in providers, tools, scorers, metrics, solvers, storage backends, or example evals, default to an extension package unless an accepted issue explicitly asks for an Inspect core change.
   • Read the full issue, comments, linked PRs, and affected code.
   • Search open PRs by issue number, title keywords, stack trace text, and subsystem names.
   • Comment on the issue to claim it before investing real implementation time, matching the project contributor guidance.
   • If there is no issue for the intended change, open or ask for an issue first unless the user explicitly says to proceed with a tiny docs/test-only fix.

2. Sync before changing files.

   cd /Users/dejain/nvidia/oss/worktrees/inspect-ai/inspect_ai
   git fetch origin main
   git switch main
   git pull --ff-only origin main
   git switch -c issue-NN-short-topic
   

   If the local checkout is missing, clone upstream first and add the deepujain fork remote before pushing.

3. Implement narrowly.

   • Follow CONTRIBUTING.md, Makefile, pyproject.toml, and nearby code patterns.
   • Main package code lives under src/inspect_ai; tests live under tests.
   • Read CLAUDE.md before coding. It requires strict typing, Google-style public API docstrings, context-rich errors, and repo-specific async/file-path patterns.
   • Async tests run under anyio's asyncio and trio backends. Do not use @pytest.mark.asyncio; use anyio.sleep, anyio.Event, and inspect_ai._util._async.tg_collect() rather than raw asyncio helpers unless the local pattern says otherwise.
   • File-path code must support s3://, file://, and plain local paths; use filesystem() and local_path() from inspect_ai._util.file.
   • Public APIs, CLI behavior, solver/scorer/model interfaces, log formats, sandbox/tooling behavior, and docs examples need tests or explicit manual validation.
   • Keep compatibility in mind: Inspect AI is a framework used by downstream eval projects, so avoid broad behavior changes unless the issue calls for them.
   • Preserve existing precedence rules. New env vars, defaults, headers, or config fallbacks should not override explicit user/model args.
   • For persistence or filesystem fixes, use atomic writes, cleanup-on-exception, bounded growth, and recovery from corrupt optional state where practical.
   • For larger internal refactors, split mechanical extraction from behavior changes when possible and call out stacked-base or dependency context in the PR body.
   • Add a concise CHANGELOG.md entry for user-facing fixes, provider changes, docs-visible features, CLI changes, viewer changes, and sandbox tool changes. Write changelog entries outcome-first: describe the user-visible behavior and affected tools/APIs, not the internal mechanism. Skip changelog only for pure tests/internal maintenance and say why in the PR body if it might be questioned.
   • Put new changelog entries under the first ## Unreleased section. If a fresh upstream release removed that section, recreate ## Unreleased at the top of CHANGELOG.md and move the PR's entry there before pushing; the entries-under-unreleased CI check fails otherwise.
   • Avoid private tool-generated references in code, commit messages, PR titles, PR bodies, comments, branch names, or any other GitHub-visible text.
   • Do not add any source-agent, tool-name, "generated by", automation, or bracketed provenance labels anywhere in contributor-facing artifacts.
   • If touching _view, regenerate schema/types and build/check the frontend output. CI will fail if inspect-openapi.json, generated types, or src/inspect_ai/_view/dist are stale.
   • For viewer source fixes that live in src/inspect_ai/_view/ts-mono, open and maintain the ts-mono PR first. Do not open an Inspect AI submodule/dist bump that points at an unmerged ts-mono commit unless a maintainer explicitly asks for pre-merge staging; wait for the ts-mono commit to land on main, or let maintainers open the Inspect AI bump.
   • For sandbox fallback fixes, preserve loud provider-level errors unless maintainers ask otherwise. Do not turn sandbox capability limits, such as unsupported user= exec, into quiet tool-output feedback. In the PR body, state whether rootless behavior, user switching, tool visibility, and injectable-versioning are changed or intentionally left unchanged.
   • If touching sandbox tools or injectable support, check whether src/inspect_ai/tool/_sandbox_tools_utils/sandbox_tools_version.txt must be bumped and run the relevant slow tool tests when practical. Put _sandbox_tools_utils regression tests under tests/tools/sandbox_tools_utils/, and prefer outcome assertions over exact ordered sandbox exec() call sequences unless the sequence itself is the contract.

4. Validate the feature specifically.

   • First run the narrow test or command that proves the issue requirement.
   • Add or update tests for behavior changes.
   • When practical, verify the targeted regression fails before the fix and passes after it; mention that in the PR body.
   • Run the adjacent subsystem test file or focused test group, not only the single new test, when behavior touches shared scorer, provider, log, CLI, viewer, or sandbox paths.
   • For CLI work, run the relevant inspect ... command or help path.
   • For docs examples, verify imports and snippets where practical.
   • In the PR body, do not list only make check and make test; also explain which targeted validation proves the requested behavior.

5. Run checks.

   pytest tests/path/to/test_file.py::test_name -v
   make check
   make test
   

   Run uv run ruff format --check (or ruff format then commit) on touched Python files before pushing. CI's pre-commit and ruff jobs fail on formatting-only drift such as missing blank lines before new test functions (#5181, 2026-09-02).

   make check runs ruff and mypy. make test runs pytest. If the full suite is blocked by environment or unrelated failures, report the exact blocker and preserve focused feature evidence.

   CI uses uv even though the contributor guide also documents pip install -e ".[dev]". To mirror CI more closely in a fresh checkout, use:

   uv venv
   uv pip install .[dev]
   uv run pytest tests/path/to/test_file.py::test_name -v
   

   Additional validation by area:

   • Before validating changes to model providers, sandbox/tool code, agents, or async plumbing, read the live .claude/skills/slow-tests/SKILL.md and run the applicable gated classes (slow, live provider API, flaky, and/or trio). Plain pytest skips these classes, and PR CI does not run live provider tests. Add a ### Slow tests section under "Other information" with each exact command, passed and skipped counts by class/provider, and every environment limit such as missing keys, Docker, model access, or a local server. Never count a skip as an executed test.
   • Docs: render or at least inspect the affected Quarto page and generated data path.
   • CLI: run the exact inspect ... command or --help path affected by the issue.
   • Model/provider: test request construction, parsing, header merging, default/fallback precedence, and opt-out behavior without requiring secrets; include provider-specific tests.
   • Scorer/log/recovery: test observability fields, event/sample reconstruction, retry carry-forward, legacy log compatibility, and dataframe/export behavior when schema changes.
   • Viewer: run python src/inspect_ai/_view/schema.py, build/check src/inspect_ai/_view/ts-mono, and include changed inspect-openapi.json, generated types/submodule updates, and src/inspect_ai/_view/dist when frontend output can change.
   • For ts-mono viewer PR maintenance, rerun pnpm install --frozen-lockfile after fetching or rebasing over dependency updates before trusting typecheck or lint failures; stale dependencies can produce false markdown/typescript errors.
   • ACP/TUI/interactivity: add unit tests plus E2E-style tests for protocol/session routing and textual widgets when UI state or live agent behavior changes.
   • Sandbox tools: bump sandbox_tools_version.txt when injectable code changes and run uv run pytest --runslow -m slow tests/tools/ -x or the narrower sandbox-tool tests when the local environment can support it.

6. Commit and push.

   • Commit only files needed for this issue.
   • Use the author Deepak Jain <deepujain@gmail.com>.
   • Use a clear maintainer-facing commit message, for example fix: handle empty task list in CLI.
   • Do not add generated or local-only artifacts unless the repo expects them.

7. Open the PR.

   gh pr create \
     --repo UKGovernmentBEIS/inspect_ai \
     --base main \
     --head deepujain:issue-NN-short-topic
   

   Open a PR only when it is intended for review: linked to accepted work when required, locally validated, current with main, and within the configured Skippy queue target unless the user changes it. Do not open draft PRs for work that should be reviewed; maintainers do not review draft PRs, and repeated draft rebases consume CI without advancing mergeability. If work is not ready, keep it local or in the fork without an upstream PR.

   Inspect AI requires contributor tooling disclosure in PR descriptions. If the user's public-text rule forbids naming or describing tooling, stop before opening the PR and ask for approved wording or do not open the PR. Never omit a required disclosure, invent a false disclosure, or publish private tool names against the user's instruction.

   Use a clean maintainer-facing PR title with a repository-style public prefix and no private provenance prefix or attribution.

   Prefer the repository PR template, filled in with concrete maintainer-facing detail. The strongest Inspect AI PRs spell out the previous behavior, the new behavior, whether the change is breaking, and the exact validation that proves the issue is fixed. Do not leave template headings empty or invent extra template categories unless the upstream template changes.

   A concise custom structure is also acceptable when it is clearer than the template: Summary, Background/root cause, Changes, Design choices, Breaking changes, and Validation. This is useful for sandbox-tool, recovery, or larger feature work where reviewers need the failure mechanism and tradeoffs.

   PR body shape:

   ## This PR contains:
   - [ ] New features
   - [ ] Changes to dev-tools e.g. CI config / github tooling
   - [ ] Docs
   - [ ] Bug fixes
   - [ ] Code refactor
   
   ### What is the current behavior? (You can also link to an open issue here)
   
   ...
   
   ### What is the new behavior?
   
   - ...
   
   ### Does this PR introduce a breaking change? (What changes might users need to make in their application due to this PR?)
   
   No. ...
   
   ### Other information:
   
   Validation:
   - Targeted:
     - `uv run pytest ... -v`
       - Explain the regression or behavior this proves.
   - Regression:
     - `uv run make check`
     - `uv run make test`
   
   CI/CD coverage expected:
   - State which standard workflows should cover the change, and note any area-specific workflow such as log viewer or sandbox tools when touched.
   
   Closes #NN.
   

   Quality bar for titles and descriptions:

   • Use a short maintainer-facing title with Inspect AI's visible conventions, such as fix: ..., feat: ..., docs: ..., or Provider: ....
   • Avoid private provenance attribution and process notes.
   • Mention the changelog entry when the change is user-facing, or explicitly say "No changelog entry: test-only/internal-only" when that is the honest reason.
   • For docs-only PRs, describe the implementation/API checks used to verify the examples instead of pretending there is a new runtime test.
   • For bug fixes, include a targeted regression test when practical and say whether it fails on main if you verified that.
   • For provider changes, validate request construction/parsing without secrets where possible, and mention any live or optional-provider tests separately.
   • For viewer changes, mention schema/type regeneration and pnpm --filter inspect build/dist validation when applicable.
   • For sandbox-tool changes, mention the version bump gate and slow tool tests when applicable.
   • For sandbox self-check changes (src/inspect_ai/util/_sandbox/self_check.py or tests/tools/test_sandbox_docker_and_local.py), expect slow-tool-tests-dev when CI selects tests/tools/. Poll it to terminal state. Treat the self-check as a downstream provider conformance contract: do not tighten it until the linked issue is accepted, the intended provider-side behavior is settled, and affected provider maintainers can validate it. Do not mask a real product bug with a strict xfail, especially one tied to an unpinned platform image; fix the provider first, then make the conformance check enforce the fixed behavior without an xfail.
   • When a PR includes agent-review disclosure, include the reviewer model, number of passes, and concrete findings (including zero findings). A bare tool name or generic claim of AI review is not useful review evidence.

8. Shepherd the PR.

   • Inspect CI, review comments, and requested changes after pushing.
   • If a sweep finds an otherwise healthy PR has become CONFLICTING/DIRTY, rebase it promptly on upstream main, preserve existing top-of-file changelog entries plus the PR's ## Unreleased entry, rerun focused tests and uv run make check, force-push with lease, then re-check CI.
   • Do not keep closed, unaccepted, over-limit, or draft PRs alive with routine rebase churn. If a PR is not eligible for review, stop CI-triggering pushes and move the discussion back to the issue with concrete evidence.
   • If a maintainer says draft PRs are not reviewed, convert only genuinely review-ready, policy-compliant PRs to ready-for-review. Otherwise leave them closed/local and prepare the accepted-issue path first.
   • If an accepted issue came from a maintainer-closed draft PR and GitHub refuses to reopen it, recreate a ready-for-review PR from a fresh rebased branch rather than fighting the closed PR.
   • If maintainers close a batch for policy reasons, do not reopen by default. Classify it as policy-blocked, record linked issue state and any replacement PRs, then select at most 4 issues to pursue through the accepted-issue path.
   • Treat Build and Build Log Viewer jobs as expected PR gates: ruff, mypy, pre-commit, package inspection, Python 3.10/3.11 tests, schema/type checks, submodule-on-main, and dist validation. Do not ignore a required failure just because unrelated deploy-style checks can fail outside the PR gate.
   • Treat submodule-on-main as a blocker for Inspect AI viewer bump PRs, not as a standing expected failure. If it fails because the ts-mono source PR is unmerged, keep the source PR moving and wait for the merged commit before reopening or recreating the Inspect AI bump.
   • If GitHub refuses to rerun failed jobs because the account lacks admin rights, and the failure is clearly unrelated or flaky after log inspection, refresh CI with a content-preserving git commit --amend --no-edit plus git push --force-with-lease rather than adding a throwaway commit. State the failed test and why no code change was made.
   • Check top-level comments, reviews, and inline review comments. Recent merged PRs often had little public discussion, but actionable inline comments focused on small correctness/style details such as temp-file cleanup, exception safety, and keeping docstrings/descriptions intact.
   • Fix actionable feedback on the current head, rerun relevant validation, and leave a short factual status comment.
   • Prefer applying reviewer suggestions directly when they are low-risk, then add a follow-up commit with the exact validation rerun. If a maintainer says they will make minor changes before merge, still keep the branch green and avoid further churn.
   • If maintainer feedback changes the design direction, follow it instead of defending the first implementation.

Issue selection preferences

• Trigger phrase: "sweep" when the active repo/thread context identifies Inspect AI.

Good Inspect AI contribution targets include small CLI bugs, docs corrections, model/provider integration fixes, scorer/solver behavior gaps, logging/viewer regressions, sandbox/tool edge cases, and narrowly scoped tests for existing behavior.

Avoid large API redesigns, new provider integrations requiring secrets, eval semantics that need maintainer product judgment, broad viewer/frontend churn without generated artifacts, and changes that could silently alter downstream evaluation results without a clear migration story.