Offload Onboard

Onboard Repository to Offload

This skill walks through onboarding the current repository to use Offload — a parallel test runner that executes tests across Modal cloud sandboxes. Offload is installed as part of the procedure below.

Procedure

Follow these steps in order. Use the TodoWrite tool to track progress through each step.

Step 1: Detect Test Framework and Test Paths

Investigate how the repository runs its tests:

1. Look for pyproject.toml, setup.cfg, setup.py, Cargo.toml, package.json, go.mod, or similar project files
2. Look for existing CI workflows (.github/workflows/, .gitlab-ci.yml, etc.) to see how tests are currently invoked
3. Look for test directories: tests/, test/, **/test_*.py, **/*_test.go, src/**/tests/, etc.
4. Determine:
   • Framework: pytest, cargo, vitest, or default (generic)
   • Test paths: Where tests live (e.g. ["tests/"], ["src/"])
   • Python runner: If pytest, determine if the project uses uv, poetry, pip, or plain python
   • Extra args: Any special invocation needed (e.g. ["run", "--group", "test"] for uv with dependency groups)

Ask the user to confirm your detection if anything is ambiguous.

Step 2: Verify Prerequisites

Verify the following are installed and authenticated. Do not continue until all prerequisites are confirmed.

• uv (required — Offload uses uv to run the Modal sandbox script regardless of project language or package manager)
• modal CLI — must be installed (pip install modal) and authenticated. Run modal profile list to check. If not authenticated, tell the user to run modal token new (opens a browser, writes credentials to ~/.modal.toml). Wait for the user to confirm authentication before proceeding.
• rsync (required — Offload uses rsync for working directory snapshots during the full-build fallback path. Without it, if the thin-diff path fails, the entire job fails with no recovery)
• For pytest projects: the configured Python runner (uv, poetry, or python) and pytest must be available locally for test discovery
• For cargo projects: cargo-nextest must be installed (cargo install cargo-nextest)

Step 3: Find or Create a Dockerfile

Offload's Modal provider needs a Dockerfile to build sandbox images. Look for an existing one:

1. Check .devcontainer/Dockerfile, Dockerfile, docker/Dockerfile, or any Dockerfile referenced in CI
2. If one exists and is suitable (has the language runtime + package manager), note its path
3. If none exists, create .devcontainer/Dockerfile with the minimal base:

For Python projects:

FROM python:<version>-slim

RUN apt-get update && apt-get install -y --no-install-recommends rsync \
    && rm -rf /var/lib/apt/lists/*

# Only install git if tests or dependencies require it (not needed for thin diffs)
# RUN apt-get update && apt-get install -y --no-install-recommends git \
#     && rm -rf /var/lib/apt/lists/*

# Install the package manager used by the project (uv, poetry, etc.)
# For uv:
COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv
# For poetry:
# RUN pip install poetry

WORKDIR /app

For Rust projects:

FROM rust:<version>-slim

# Only install git if tests or dependencies require it (not needed for thin diffs)
RUN apt-get update && apt-get install -y --no-install-recommends pkg-config libssl-dev rsync \
    && rm -rf /var/lib/apt/lists/*

# Install cargo-nextest if the project uses it
# RUN cargo install cargo-nextest

WORKDIR /app

Key principles:

• Match the language/runtime version from the project's config (e.g. requires-python, rust-version)
• Include git if any tests create git repos or the project uses git-based dependencies
• Do NOT COPY . . — Offload overlays source via --copy-dir at image build time
• Keep it minimal — dependencies are installed at runtime inside the sandbox

Step 4: Create offload.toml

Create offload.toml at the project root using the Modal provider. In the templates below, values in <angle-brackets> are placeholders you must substitute with project-specific values.

[offload]
max_parallel = 3
test_timeout_secs = 120
sandbox_repo_root = "/app"

[provider]
type = "modal"
dockerfile = "<path-to-dockerfile>"
include_cwd = true

[framework]
type = "pytest"
paths = ["<test-paths>"]
command = "<pytest-command>"      # e.g. "uv run pytest", "poetry run pytest", "python -m pytest"

[groups.all]
retry_count = 0
filters = ""                    # pytest args for discovery filtering (e.g. "-m 'not slow'")

[report]
output_dir = "test-results"
junit = true
junit_file = "junit.xml"

For Cargo (Rust) projects, replace the [framework] section with:

[framework]
type = "cargo"

Optional: sandbox_init_cmd for build-time setup

Do not add sandbox_init_cmd initially. If you observe that every sandbox runs the same setup work (e.g. uv sync --all-packages, git apply, database migrations), move that work into sandbox_init_cmd so it runs once during image build instead of on every sandbox:

[offload]
max_parallel = 3
test_timeout_secs = 120
sandbox_repo_root = "/app"
sandbox_init_cmd = "uv sync --all-packages"   # runs once at image build time

This is especially useful for monorepo setups where dependency installation is slow.

Optional: post_patch_cmd for derived artifact regeneration

If the project generates code or builds assets that must stay in sync with source (e.g., API clients from OpenAPI schemas, frontend bundles, compiled protobufs), add post_patch_cmd to regenerate them after the thin-diff patch is applied:

[offload]
post_patch_cmd = "make generate-client"

Unlike sandbox_init_cmd (which runs once during base image builds), post_patch_cmd runs after every patch, ensuring derived artifacts match the patched source. The OFFLOAD_PATCH_FILE env var is set when a diff exists.

Optional: skip coverage tracing during discovery (pytest-cov)

If the project uses pytest-cov, coverage tracing runs even during pytest --collect-only, roughly doubling local discovery time. Set discovery_args under [framework] so Offload skips it during collection (discovery only — test-run coverage is unaffected):

[framework]
type = "pytest"
command = "uv run pytest"
discovery_args = "--no-cov"

Only set this when pytest-cov is installed; pytest --collect-only --no-cov is a usage error without the plugin. Keep it out of group filters, which are test selectors.

When to use type = "default" for the framework:

The built-in pytest and cargo frameworks cover straightforward setups. Fall back to type = "default" for the [framework] section when:

• Conflicting local config: The project's pyproject.toml or setup.cfg has addopts that conflict with Offload (e.g. xdist workers, coverage plugins) and you need to override them with -o addopts= or -p no:xdist
• Custom discovery pipeline: Standard collection doesn't work and you need shell pipelines (e.g. grep filtering, marker exclusions combined with workspace sync)
• Unsupported framework: Jest, Go, Mocha, or any framework not directly supported

Example — pytest in a monorepo with xdist conflict (still using the Modal provider):

[provider]
type = "modal"
dockerfile = "<path-to-dockerfile>"
include_cwd = true

[framework]
type = "default"
discover_command = "uv run pytest --collect-only -q {filters} 2>/dev/null | grep '::'"
run_command = "cd /app && uv run pytest -v --tb=short --no-cov -p no:xdist -o addopts= --junitxml={result_file} {tests}"
test_id_format = "{name}"

In monorepo setups where tests run from a subdirectory, add sandbox_project_root to set the test working directory separately from the repo root:

[offload]
sandbox_repo_root = "/app"
sandbox_project_root = "/app/mypackage"

[framework]
type = "pytest"
command = "uv run pytest"

Note: if discovery or execution requires pre-steps like uv sync --all-packages, use sandbox_init_cmd in the [offload] section rather than inlining them into the discover/run commands.

For the full configuration reference and more examples, see the Offload README.

Configuration reference for fields used above:

[offload]

• max_parallel: Number of concurrent Modal sandboxes (start with 3, optimize later)
• test_timeout_secs: Per-test-batch timeout in seconds (120s is generous for unit tests)
• sandbox_repo_root: Path to the repository root inside the sandbox (e.g. /app). Used for thin-diff patches and as the default test working directory (OFFLOAD_ROOT)
• sandbox_project_root: (optional) Working directory for test execution, if different from sandbox_repo_root. Only needed in monorepo setups where tests run from a subdirectory
• sandbox_init_cmd: Optional command to run during image build, after cwd/copy-dirs are applied (e.g. "uv sync --all-packages")
• post_patch_cmd: Optional command to run after thin-diff patch is applied, before image materialization (e.g. "make generate-client")

[provider] (Modal)

• dockerfile: Path to the Dockerfile for building the sandbox image
• include_cwd: Copy the current working directory into the image (default: false)

[framework] (pytest)

• paths: Directories to search for tests (default: ["tests"])
• command: Full command prefix for pytest invocation (e.g. "uv run pytest"). Replaces the legacy python/extra_args fields
• discovery_args: Extra args appended only during discovery (--collect-only), not execution — e.g. --no-cov to skip coverage tracing when the project uses pytest-cov

[groups.<name>]

• retry_count: Number of retries for failed tests (0 = no retries, 1 = catches transient failures)
• filters: Filter string passed to the framework during discovery (e.g. -m 'not slow')

IMPORTANT: Do not use JUnit reporting plugins or hooks. Offload owns the JUnit XML report — it passes --junitxml to pytest and uses the resulting XML as the single source of truth for matching test results back to discovered tests. Custom conftest hooks, plugins, or pytest options that modify JUnit XML output (e.g. record_xml_attribute, pytest-metadata, or junit_family overrides) will interfere with Offload's test ID resolution and cause incorrect results. If the project has existing JUnit customization in conftest.py or pytest plugins, remove it before proceeding.

Step 5: Verify Vitest Test Discovery (vitest only)

Skip this step if the framework is not vitest.

Offload's vitest integration uses --testNamePattern to select specific tests for parallel execution. This flag matches tests by their space-separated describe/test chain — not by file path. When two tests in different files share the same space-separated name, Offload cannot distinguish them and will error during discovery, blocking all test execution.

Run offload collect to verify that test discovery works:

offload collect

If Offload reports "Duplicate test names found (space-separated)", stop and present the duplicates to the user. Explain:

Offload cannot run vitest tests that share the same space-separated name across different files. This is a requirement of Offload's --testNamePattern-based test selection — without unique names, Offload will error during test discovery and no tests will run.

Then ask: "Would you like me to deduplicate these test names by making them more descriptive? This typically involves wrapping tests in uniquely named describe() blocks or renaming test()/it() calls to include more context."

If the user agrees, rename the tests to make the space-separated names unique. Prefer wrapping in descriptive describe() blocks over renaming individual tests, as this preserves the existing test names while adding disambiguation.

After making changes, run offload collect again. Repeat until offload collect succeeds with no duplicate name errors.

If the user declines deduplication, do not proceed with Offload onboarding — inform them that Offload's vitest integration cannot function with duplicate test names and they will need to resolve this before onboarding can continue.

Additional check: tests with literal > in their names.

After offload collect passes, run npx vitest list --json and check for test names where a >-separated segment itself contains > (e.g. describe('parse') > it('attribute value with >')). Offload uses > as the separator between the file path and the describe/test chain in test IDs, so a literal > in a test name creates ambiguity that causes "Not Run" results.

If any are found, present them to the user and recommend renaming the affected test()/it()/describe() calls to avoid > characters. For example, rename 'attribute value with >' to 'attribute value with greater-than'. This is not a blocking error — offload collect will still pass — but affected tests will be reported as "Not Run" after execution because Offload cannot match their results back to discovered IDs.

Step 6: Create Local Invocation Script

Create scripts/offload-tests.sh:

#!/usr/bin/env bash
#
# Run the project's test suite via Offload (parallel on Modal).
# Requires: Offload (cargo install offload), Modal CLI + credentials
#
set -euo pipefail

if ! command -v offload &> /dev/null; then
    echo "Error: 'offload' not installed. Install with: cargo install offload"
    exit 1
fi

cd "$(git rev-parse --show-toplevel)"
exec offload run --copy-dir ".:/app" "$@"

Make it executable with chmod +x scripts/offload-tests.sh.

The --copy-dir flag tells Offload to bake the local directory into the sandbox image at the given path during the prepare step. The target path must match sandbox_repo_root in offload.toml (e.g. ".:/app" when sandbox_repo_root = "/app"). This is specified at invocation time, not in offload.toml, because it depends on where you're running from.

Use this script (or the equivalent invocation) for all subsequent steps that run Offload.

If the project uses a Makefile, justfile, or Taskfile instead of scripts/, add the invocation there instead to be consistent with existing practice.

Step 7: Update .gitignore

Append Offload artifacts to .gitignore:

# Offload
test-results/

NOTE: Do not confuse .gitignore (which controls what git tracks) with .dockerignore (which controls what gets copied into the sandbox image). The .dockerignore is only created if needed during troubleshooting — see the Troubleshooting section.

Step 8: Run Offload Locally and Verify

Install offload if not already present:

cargo install offload

Run the tests using the invocation script from Step 6:

First, verify test discovery with offload collect:

offload collect

This runs test discovery locally without creating sandboxes or executing tests. Fix any errors until offload collect succeeds and lists the expected tests. Common issues:

1. "No tests discovered": Check that paths in offload.toml points to the correct test directories and the framework command is correct.
2. "Duplicate test names found" (vitest): Duplicate space-separated test names exist. Return to Step 5 and resolve them.
3. Discovery command failed: The framework tool (pytest, cargo nextest, npx vitest) is not installed or not on PATH.

Do not proceed to execution until offload collect lists the expected tests.

Then, establish a baseline by running the test suite directly (without Offload):

# For vitest:
pnpm exec vitest run --project <project>
# For pytest:
<command> --tb=short
# For cargo:
cargo nextest run

Record the number of passed, failed, and skipped tests. Some projects have pre-existing test failures (snapshot mismatches, missing fixtures, flaky tests). These are not your problem to fix — the goal is for Offload's results to match this baseline, not to achieve zero failures.

Then, run the full test suite via Offload:

./scripts/offload-tests.sh

Compare Offload's results against the baseline. Offload is working correctly when:

• The number of passed tests matches the baseline (within 1-2 for flaky tests)
• The number of failed tests matches the baseline
• There are no "Not Run" tests (0 missing)

If Offload's results differ from the baseline, iterate until they match. Common issues and fixes:

1. "Exec format error": .venv or local binaries leaked into the sandbox. See the Troubleshooting section on creating a .dockerignore.
2. "No such file or directory": The sandbox is missing a dependency. Check the Dockerfile has the right runtime and package manager.
3. "Token validation failed": Modal credentials are expired. Run modal token new to refresh.
4. Tests discovered but "Not Run": The test command is failing silently inside the sandbox. Check batch logs in test-results/logs/ for errors. Debug by checking if uv/python/cargo is available in the Dockerfile.
5. More failures than baseline: Offload may be running tests in a different order or with different environment. Check batch logs for the extra failures and compare with the baseline output.
6. Fewer tests discovered than expected: Check that offload collect finds all the tests. If using vitest with --project, ensure the filter matches.

Keep running and fixing until Offload's pass/fail counts match the baseline. Do not proceed to optimization until they match.

Step 9: Optimize Parallelism

Run a simple linear search over max_parallel to minimize total runtime:

1. Test max_parallel values: 1, 2, 3, 4, 6, 8 (keeping other params fixed)
2. For each value, edit offload.toml, run time ./scripts/offload-tests.sh, and record the wall-clock duration from the real line
3. Pick the value with the lowest duration
4. Optionally test retry_count = 0 vs retry_count = 1 at the optimal parallelism

The optimal max_parallel depends on the number of test files and per-test duration. More parallelism has diminishing returns due to sandbox creation overhead.

Report the results as a table to the user and set the optimal values in offload.toml.

Step 11: Enable Checkpoint Mode

Checkpoint mode enables fast incremental image builds by caching a base image at the most recent commit that changed dependency files, then applying a thin diff for subsequent runs. This avoids full image rebuilds on every run and is recommended for all projects.

Skip this step only if the project's build inputs cannot be reliably determined — for example, if dependency files are dynamically generated, scattered across many locations with no fixed pattern, or the project uses an unusual build system where it is unclear which files affect the sandbox image.

To enable checkpoint mode, add a [checkpoint] section to offload.toml listing the files whose changes should trigger a full image rebuild:

[checkpoint]
build_inputs = [
    "Dockerfile",
    "requirements.txt",
    "pyproject.toml",
]

Common build_inputs patterns:

Include the Dockerfile itself in build_inputs — changes to the base image should always trigger a full rebuild. Also include lockfiles (e.g. uv.lock, Cargo.lock, package-lock.json) since they pin exact dependency versions.

Dockerfile requirement: Checkpoint mode uses offload apply-diff inside the sandbox to apply thin diffs on top of the checkpoint image. The Dockerfile must have the offload binary on PATH. If offload is not installed, thin diff will fail and Offload falls back to a full build (with a warning). Projects that install offload for test execution already satisfy this requirement.

After adding the section, run offload validate to check the config and offload checkpoint-status to verify checkpoint detection.

Step 12: Update Agent/Project Instructions (if desired)

First, ask the user: "Would you like to configure Offload as the default test runner for AI agents working in this repository? This requires agents to have access to Modal API credentials."

If the user declines, skip this step entirely and proceed to Step 13.

If the user agrees, ensure that future AI agents working in this repository know to use Offload for running tests:

1. Check for these files (in priority order):

   • CLAUDE.md or .claude/CLAUDE.md — Claude Code project instructions
   • AGENTS.md — Multi-agent coordination instructions
   • CONTRIBUTING.md — Contributor guidelines

2. If at least one exists, read it to understand its structure and find any existing section about running tests (look for headings or instructions mentioning "test", "testing", "verification", "CI", "check").

3. If both CLAUDE.md and AGENTS.md exist, update both. If only one exists, update that one.

4. If none of these files exist, create a CLAUDE.md at the project root. It only needs the testing section — don't fabricate other content.

5. Amend or add a testing section that is directive, not merely suggestive. The instruction must tell agents to use Offload as the way to run tests locally. Do not remove any existing test commands — keep them as a fallback — but make Offload the primary instruction. The section should also reference the /offload skill so agents activate it when running tests, reading logs, or debugging failures. Example:

   ## Running tests
   
   Run the test suite via Offload, which parallelizes execution across Modal cloud sandboxes:
   
   ```bash
   ./scripts/offload-tests.sh
   ```
   
   Prerequisites: Offload (`cargo install offload`) and Modal credentials (`modal token new`).
   Activate the `/offload` skill for test execution, log reading, and failure debugging.
   

   Adapt the exact command to match what was configured in earlier steps (the script path, etc.).

6. Preserve the existing tone and formatting of the file. If it uses a digraph, bullet lists, or a specific heading style, match that style. Do not restructure or reformat existing content.

Step 13: Set Up CI Job (if desired)

Ask the user if they want to set up a CI job to run Offload tests automatically on push/PR. If they decline, skip Steps 13 and 14.

If they want CI, detect the CI system from the repository:

• .github/workflows/ → GitHub Actions
• .gitlab-ci.yml → GitLab CI
• Jenkinsfile → Jenkins
• .circleci/ → CircleCI

If no CI system is detected, inform the user and skip this step.

For GitHub Actions, create .github/workflows/test-offload.yml:

name: Offload Tests

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]
  workflow_dispatch:

jobs:
  offload-tests:
    runs-on: ubuntu-latest
    continue-on-error: true   # Advisory only - never blocks merging
    steps:
      - uses: actions/checkout@v4

      - name: Install system dependencies
        run: sudo apt-get update && sudo apt-get install -y rsync

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "<version>"

      # Include language-specific setup needed for LOCAL test discovery
      # offload discovers tests locally, then executes them remotely
      # Example for uv-based Python project:
      - name: Install uv
        uses: astral-sh/setup-uv@v5

      - name: Install dependencies
        run: uv sync --all-groups

      - name: Install Rust toolchain
        uses: dtolnay/rust-toolchain@stable

      - name: Cache offload binary
        uses: actions/cache@v4
        with:
          path: |
            ~/.cargo/registry
            ~/.cargo/git
            ~/.cargo/bin/offload
          key: cargo-offload-${{ runner.os }}

      - name: Install offload
        run: |
          if ! command -v offload &> /dev/null; then
            cargo install offload
          fi

      - name: Install Modal CLI
        run: pip install modal

      - name: Run tests via offload
        env:
          MODAL_TOKEN_ID: ${{ secrets.MODAL_TOKEN_ID }}
          MODAL_TOKEN_SECRET: ${{ secrets.MODAL_TOKEN_SECRET }}
        run: offload run --copy-dir ".:/app"  # adjust path to match sandbox_repo_root

IMPORTANT: The CI runner needs the project's language toolchain and dependencies installed because Offload discovers tests locally (e.g. uv -m pytest --collect-only), then sends them to Modal for execution. Without local discovery dependencies, Offload will fail with "No such file or directory".

continue-on-error: true makes the job advisory — it always reports success to branch protection, but step-level pass/fail is visible in the Actions UI.

For other CI systems, adapt the same pattern: install Offload + Modal CLI, set Modal secrets as env vars, run offload run.

Step 14: Configure CI Secrets

Tell the user they need to add two repository secrets:

• MODAL_TOKEN_ID: Their Modal API token ID
• MODAL_TOKEN_SECRET: Their Modal API token secret

These can be found in ~/.modal.toml after running modal token new.

For GitHub: Settings → Secrets and variables → Actions → New repository secret

Offer to trigger a CI run (if GitHub Actions) once the user confirms secrets are configured:

gh workflow run test-offload.yml
gh run list --workflow=test-offload.yml --limit=1
gh run watch <run-id> --exit-status

Wait for the run to succeed. If it fails, diagnose and fix the issue, then trigger again.

Troubleshooting Reference

Creating a .dockerignore

If tests fail due to local artifacts leaking into the sandbox (e.g. "Exec format error" from a macOS .venv copied into a Linux sandbox, or stale __pycache__/build directories causing conflicts), create a .dockerignore at the project root to exclude them:

.venv
.git
.github
__pycache__
*.egg-info
test-results
build
dist
target
node_modules

CRITICAL: .venv is the most common culprit. If a local virtual environment (e.g. macOS binaries) gets copied into a Linux sandbox, tests will fail with "Exec format error". This is the most common onboarding failure.

NOTE: Sometimes tests depend on the git repository. If tests fail because .git is missing, remove .git from the .dockerignore.

Summary of Files Created/Modified