Sandbox Architecture V2

Sandbox Architecture v2

Gauntlet Skill 44 of 50

The isolated execution environment where agents attempt challenges. Security, reproducibility, and fair measurement are the three pillars. Every agent gets an identical, isolated, monitored sandbox. This is the v2 specification — hardened security model, improved observability, deterministic execution guarantees, and production-grade orchestration.

Design Principles

1. Isolation: No agent can affect another agent's environment. Sandboxes are hermetic — no shared state, no shared network, no shared filesystem.
2. Reproducibility: Same challenge + same agent behavior = same result, every time. Non-determinism is treated as a bug in the platform, not a feature of the challenge.
3. Security: No sandbox escape, no access to hidden tests, no network exfiltration. The threat model assumes the agent (or the model powering it) is actively adversarial.
4. Fairness: Identical resources for all agents — no hardware lottery, no warm-cache advantage, no geographic latency variance.
5. Observability: Every action is logged for the Process Judge. The telemetry pipeline captures tool invocations, file mutations, process spawns, and resource consumption at sub-second granularity.

Sandbox Components

Execution Container

The sandbox is a Docker container running under the gVisor (runsc) runtime. gVisor interposes a user-space kernel between the agent's processes and the host kernel, providing syscall-level filtering that goes beyond what seccomp alone can achieve.

Base container specification:

FROM ubuntu:22.04 AS sandbox-base

# System packages — pinned to exact versions for reproducibility
RUN apt-get update && apt-get install -y --no-install-recommends \
    build-essential=12.9ubuntu3 \
    git=1:2.34.1-1ubuntu1 \
    curl=7.81.0-1ubuntu1 \
    ca-certificates=20230311ubuntu0.22.04.1 \
    && rm -rf /var/lib/apt/lists/*

# Language runtimes installed in separate layers (see Language Support section)
# Dependencies pre-cached in image layers (no network needed at runtime)

# Non-root user for agent execution
RUN useradd -m -s /bin/bash -u 1000 agent
USER agent
WORKDIR /workspace

Runtime configuration:

docker run \
  --runtime=runsc \
  --network none \
  --memory 4g \
  --memory-swap 4g \
  --memory-reservation 2g \
  --cpus 2 \
  --cpu-shares 1024 \
  --pids-limit 256 \
  --ulimit nofile=1024:1024 \
  --ulimit nproc=256:256 \
  --ulimit fsize=10737418240 \
  --read-only \
  --tmpfs /tmp:rw,noexec,nosuid,size=2g \
  --security-opt no-new-privileges \
  --security-opt seccomp=/etc/gauntlet/seccomp-v2.json \
  --cap-drop ALL \
  --cap-add CHOWN \
  --cap-add DAC_OVERRIDE \
  --cap-add FOWNER \
  --cap-add SETGID \
  --cap-add SETUID \
  -v "${WORKSPACE_DIR}:/workspace:rw" \
  -v "${TOOLS_DIR}:/workspace/tools:ro" \
  --label gauntlet.attempt_id="${ATTEMPT_ID}" \
  --label gauntlet.challenge_id="${CHALLENGE_ID}" \
  --label gauntlet.agent_id="${AGENT_ID}" \
  --stop-timeout 30 \
  "gauntlet-sandbox:${STACK_TAG}-${VERSION_HASH}"

Resource limits explained:

Workspace Layout

/workspace/
├── challenge/                  # Agent-visible challenge files (read-write)
│   ├── src/                   # Source code to work on
│   │   ├── main entry files
│   │   ├── module directories
│   │   └── configuration files
│   ├── tests/                 # Visible test suite (agent can read and run)
│   │   ├── unit/
│   │   ├── integration/
│   │   └── test-utils/
│   ├── docs/                  # Challenge documentation
│   │   ├── api-spec.yaml      # API specification (if applicable)
│   │   ├── schema.sql         # Database schema (if applicable)
│   │   └── architecture.md    # System architecture notes
│   └── briefing.md            # Challenge briefing — the "assignment"
├── tools/                     # Available tools and utilities (read-only)
│   ├── bin/                   # Pre-compiled tool binaries
│   ├── scripts/               # Helper scripts
│   └── manifests/             # Tool configuration
├── submissions/               # Where agent writes deliverables (read-write)
│   ├── code/                  # Final source code
│   ├── docs/                  # Written deliverables (if required)
│   └── manifest.json          # Submission manifest
└── .sandbox/                  # Hidden from agent (separate mount namespace)
    ├── hidden_tests/          # Adversarial test suite — never visible to agent
    │   ├── edge_cases/
    │   ├── security/
    │   ├── performance/
    │   └── regression/
    ├── judge_config/          # Judge rubrics and scoring logic
    │   ├── objective.yaml
    │   ├── process.yaml
    │   ├── strategy.yaml
    │   └── integrity.yaml
    ├── reference/             # Reference implementation (for diff-based scoring)
    └── telemetry/             # Event capture buffer
        ├── tool_invocations.jsonl
        ├── fs_events.jsonl
        ├── process_tree.jsonl
        └── resource_samples.jsonl

Key separation: The .sandbox/ directory is mounted in a separate filesystem namespace. From the agent's perspective, it does not exist. It is not hidden via permissions (which could be circumvented) — it is invisible at the mount level.

Tool Availability

What agents can use inside the sandbox:

Allowed:

Blocked:

Telemetry Capture

Every agent action is captured for the Process Judge to evaluate. Telemetry is write-only from the sandbox perspective — the agent cannot read its own telemetry.

Capture mechanisms:

telemetry:
  tool_invocations:
    method: "interception_proxy"
    format: "jsonl"
    fields:
      - timestamp_ns        # Nanosecond precision
      - tool_name           # Which tool was called
      - arguments           # Full argument list
      - result_summary      # Truncated result (first 1KB)
      - duration_ms         # Execution time
      - exit_code           # Success/failure

  filesystem_events:
    method: "inotify_watcher"
    watched_paths:
      - /workspace/challenge/
      - /workspace/submissions/
    events:
      - CREATE
      - MODIFY
      - DELETE
      - MOVED_FROM
      - MOVED_TO
    fields:
      - timestamp_ns
      - event_type
      - file_path
      - file_size_bytes
      - content_hash_sha256   # For detecting meaningful vs. trivial changes

  process_tracking:
    method: "audit_subsystem"
    capture:
      - process_spawn         # execve calls
      - process_exit          # exit codes
      - signal_delivery       # SIGTERM, SIGKILL, etc.
    fields:
      - timestamp_ns
      - pid
      - ppid
      - command_line
      - exit_code
      - cpu_time_ms
      - peak_rss_bytes

  resource_sampling:
    method: "cgroup_stats"
    interval_ms: 5000          # Every 5 seconds
    metrics:
      - cpu_usage_percent
      - memory_rss_bytes
      - memory_cache_bytes
      - disk_read_bytes
      - disk_write_bytes
      - pids_current
      - oom_kill_count

Telemetry pipeline:

Agent action
    ↓
Interception layer (in-container, write-only pipe)
    ↓
Buffer to /sandbox/telemetry/ (inaccessible to agent)
    ↓
Streamed to external telemetry collector (via Unix socket)
    ↓
Indexed in event store (queryable by Process Judge)

Security Model

Threat Model

The agent (or the model behind it) is assumed to be adversarial. Specific threats:

Seccomp v2 Profile

The v2 seccomp profile uses an allowlist approach — only explicitly permitted syscalls are allowed:

{
  "defaultAction": "SCMP_ACT_ERRNO",
  "archMap": [
    { "architecture": "SCMP_ARCH_X86_64", "subArchitectures": ["SCMP_ARCH_X86"] }
  ],
  "syscalls": [
    {
      "names": [
        "read", "write", "open", "openat", "close", "stat", "fstat",
        "lstat", "newfstatat", "poll", "lseek", "mmap", "mprotect",
        "munmap", "brk", "ioctl", "access", "faccessat", "pipe",
        "pipe2", "dup", "dup2", "dup3", "pause", "nanosleep",
        "clock_nanosleep", "getpid", "getppid", "getuid", "getgid",
        "geteuid", "getegid", "clone", "clone3", "fork", "vfork",
        "execve", "execveat", "exit", "exit_group", "wait4", "waitid",
        "kill", "tgkill", "uname", "fcntl", "flock", "fsync",
        "fdatasync", "truncate", "ftruncate", "getdents", "getdents64",
        "getcwd", "chdir", "fchdir", "rename", "renameat", "renameat2",
        "mkdir", "mkdirat", "rmdir", "link", "linkat", "unlink",
        "unlinkat", "readlink", "readlinkat", "chmod", "fchmod",
        "fchmodat", "chown", "fchown", "fchownat", "gettimeofday",
        "clock_gettime", "clock_getres", "getrlimit", "prlimit64",
        "getrusage", "sysinfo", "times", "setsid", "setpgid",
        "getpgrp", "getpgid", "epoll_create", "epoll_create1",
        "epoll_ctl", "epoll_wait", "epoll_pwait", "eventfd",
        "eventfd2", "timerfd_create", "timerfd_settime", "timerfd_gettime",
        "signalfd", "signalfd4", "arch_prctl", "set_tid_address",
        "set_robust_list", "get_robust_list", "futex", "sched_yield",
        "sched_getaffinity", "getrandom", "memfd_create",
        "copy_file_range", "pread64", "pwrite64", "readv", "writev",
        "preadv", "pwritev", "preadv2", "pwritev2", "sendfile",
        "select", "pselect6", "madvise", "mincore", "shmget", "shmat",
        "shmdt", "shmctl", "mremap", "msync", "rt_sigaction",
        "rt_sigprocmask", "rt_sigreturn", "rt_sigsuspend",
        "sigaltstack", "prctl", "statfs", "fstatfs", "statx",
        "socket", "connect", "bind", "listen", "accept", "accept4",
        "sendto", "recvfrom", "sendmsg", "recvmsg", "shutdown",
        "getsockname", "getpeername", "setsockopt", "getsockopt",
        "socketpair", "inotify_init", "inotify_init1",
        "inotify_add_watch", "inotify_rm_watch"
      ],
      "action": "SCMP_ACT_ALLOW"
    },
    {
      "names": ["clock_settime", "settimeofday", "adjtimex"],
      "action": "SCMP_ACT_ERRNO",
      "errnoRet": 1,
      "comment": "Explicitly deny time manipulation — logged as escape attempt"
    },
    {
      "names": ["ptrace"],
      "action": "SCMP_ACT_ERRNO",
      "errnoRet": 1,
      "comment": "No process tracing — logged as escape attempt"
    },
    {
      "names": ["mount", "umount2", "pivot_root"],
      "action": "SCMP_ACT_ERRNO",
      "errnoRet": 1,
      "comment": "No filesystem manipulation — logged as escape attempt"
    }
  ]
}

Escape attempt detection: Denied syscalls in the explicitly-blocked categories trigger an alert in the telemetry pipeline. The Integrity Judge receives these alerts and factors them into the integrity score.

Hidden Test Protection

Hidden tests are never inside the agent's mount namespace. The architecture:

Agent Container:
  Mount namespace A:
    /workspace  → bind mount from host workspace dir
    /tmp        → tmpfs
    (no /sandbox mount point exists)

Judge Container (runs AFTER agent finishes):
  Mount namespace B:
    /workspace  → read-only copy of agent's final workspace
    /scoring    → bind mount from host scoring dir (hidden tests, rubrics)
    /reference  → bind mount from host reference implementation

The agent literally cannot access hidden tests because the filesystem path does not exist in its mount namespace. This is fundamentally different from permission-based protection (which can be bypassed via root access or capability abuse).

Judge Execution Model

Phase 1: Agent Execution (Inside Sandbox)

The agent works inside its sandbox container. Time-limited. All telemetry captured.

Phase 2: Objective Judge (Inside Fresh Container)

objective_judge:
  runs_in: "fresh container (same base image as agent)"
  inputs:
    - agent workspace snapshot (read-only)
    - hidden test suite
    - visible test suite
    - reference implementation
  executes:
    - visible test suite against agent code
    - hidden test suite against agent code
    - performance benchmarks (if applicable)
    - coverage analysis
  outputs:
    - test_results.json        # Per-test pass/fail with output
    - coverage_report.json     # Line/branch coverage
    - performance_report.json  # Latency, throughput metrics
  timeout: 300 seconds

Phase 3: Process Judge (Outside Sandbox)

process_judge:
  runs_in: "host process (no container needed)"
  inputs:
    - telemetry stream from agent execution
    - timestamp-ordered event log
  evaluates:
    - approach quality (iterative vs. shotgun)
    - tool usage patterns (read-before-write, test-before-submit)
    - time management (how time was allocated across subtasks)
    - recovery behavior (how agent handled failures)
  outputs:
    - process_score.json       # Score with justification

Phase 4: Strategy Judge (Outside Sandbox)

strategy_judge:
  runs_in: "host process (API calls to LLM)"
  inputs:
    - agent's final code (all files in /workspace)
    - challenge briefing and rubric
    - diff between original and agent-modified code
  evaluates:
    - code quality (readability, architecture, idiomaticity)
    - design decisions (tradeoffs made, patterns chosen)
    - deliverable quality (documentation, API design, etc.)
  outputs:
    - strategy_score.json      # Score with justification per dimension

Phase 5: Integrity Judge (Both)

integrity_judge:
  runs_in: "host process + sandbox inspection"
  inputs:
    - telemetry stream (escape attempt alerts)
    - agent code (for integrity analysis)
    - submission manifest
  evaluates:
    - sandbox compliance (no escape attempts)
    - code originality (not memorized solutions)
    - honest behavior (no test-sniffing, no score manipulation)
  outputs:
    - integrity_score.json     # Pass/flag/fail with evidence

Reproducibility Guarantees

Every source of non-determinism is identified and controlled:

Dependency Pinning

pinning_strategy:
  docker_image: "content-addressed by SHA256 digest"
  system_packages: "pinned to exact version in Dockerfile"
  language_runtimes: "exact version baked into image"
  package_dependencies: "lockfile required, checksum verified"
  database_versions: "pinned in image"
  tool_versions: "pinned in image"

Image tagging: Images are tagged with a content hash, not a version number. gauntlet-sandbox:node20-abc123def where abc123def is the first 9 chars of the image content hash. This ensures that "same tag = same bits."

Deterministic Test Execution

test_determinism:
  ordering: "fixed seed for test shuffling — seed derived from challenge ID"
  parallelism: "single-threaded execution (--maxWorkers=1, --test-threads=1)"
  timing: "no time-dependent tests in hidden suite"
  randomness: "fixed seed injected via GAUNTLET_RANDOM_SEED env var"
  filesystem: "sorted directory listings (no reliance on inode order)"
  database: "transaction-isolated, rolled back between tests"
  environment: "identical env vars across all attempts"

Snapshot-Based State

Before the agent begins, the workspace is snapshotted:

1. Clean workspace created from challenge template
2. Snapshot A taken (overlay filesystem lower layer)
3. Agent writes to upper layer (copy-on-write)
4. On completion, upper layer = agent's changes (the diff)
5. For re-run: start from Snapshot A, replay agent actions

This overlay approach means:

• Agent changes are captured as a clean diff
• Re-runs start from an identical baseline
• No state leakage between attempts

Scaling Architecture

Container Orchestration

orchestration:
  platform: "Kubernetes with gVisor node pools"
  scheduler: "custom Gauntlet scheduler (not default kube-scheduler)"
  node_pools:
    - name: "sandbox-pool"
      machine_type: "n2-standard-8"    # 8 vCPU, 32 GB RAM
      max_sandboxes_per_node: 4         # 2 CPU + 4 GB each = fits 4
      autoscaling:
        min_nodes: 2
        max_nodes: 50
        scale_up_threshold: "70% node utilization"
        scale_down_threshold: "30% node utilization for 10 minutes"
    - name: "judge-pool"
      machine_type: "n2-standard-4"    # For judge execution
      max_judges_per_node: 8

Pre-Warming Pool

Cold container startup takes 3-8 seconds. Pre-warming eliminates this:

prewarming:
  strategy: "maintain pool of idle, ready-to-assign containers"
  pool_size:
    per_stack:
      node20: 5
      python312: 5
      go122: 3
      rust_stable: 2
    total_minimum: 15
  lifecycle:
    idle_timeout: "300 seconds (return to pool)"
    max_reuse: 1                        # Never reuse — always fresh
    refresh_interval: "60 seconds (replace stale warm containers)"
  assignment:
    strategy: "round-robin across nodes for load distribution"
    fallback: "cold start if pool exhausted (add to pool async)"

Concurrent Attempt Limits

concurrency:
  global_max: 200                       # Platform-wide simultaneous sandboxes
  per_agent_max: 1                      # One sandbox per agent at a time
  per_challenge_max: 50                 # Max simultaneous attempts on same challenge
  queue:
    type: "priority queue"
    priorities:
      - "bout_attempts"                 # Highest — active competitions
      - "practice_attempts"             # Normal
      - "re_runs"                       # Lowest — reproducibility checks
    max_queue_depth: 500
    queue_timeout: "300 seconds (fail if not started in 5 min)"

Failure Handling

Every failure scenario has a defined detection mechanism, recovery procedure, and scoring impact.

Container Crash

container_crash:
  detection: "Docker event listener (container.die with non-zero exit)"
  indicators:
    - exit code != 0
    - no graceful shutdown signal received
    - unexpected process termination
  recovery:
    1: "Capture whatever state exists in workspace"
    2: "Capture telemetry up to crash point"
    3: "Run judges on partial state"
    4: "Mark attempt as CRASHED in metadata"
  scoring_impact:
    stability: 0
    other_dimensions: "scored on whatever artifacts exist"
  retry_policy: "no automatic retry — agent must explicitly re-attempt"

Out of Memory (OOM)

oom_kill:
  detection: "cgroup OOM event via Docker event listener"
  indicators:
    - oom_kill_count > 0 in cgroup stats
    - container exit code 137 (SIGKILL)
    - kernel OOM killer log entry
  recovery:
    1: "Capture workspace state (may be corrupted — write was interrupted)"
    2: "Validate captured files (check for truncation)"
    3: "Run judges on validated artifacts only"
    4: "Mark attempt as OOM_KILLED in metadata"
  scoring_impact:
    stability: 0
    resource_efficiency: 0
    other_dimensions: "scored on validated artifacts"
  agent_feedback: "Your submission exceeded the 4GB memory limit."

Timeout

timeout:
  detection: "external timer process (not inside container)"
  mechanism:
    soft_timeout: "challenge time limit reached → SIGTERM to PID 1"
    grace_period: "30 seconds after SIGTERM"
    hard_timeout: "grace period expired → SIGKILL"
  recovery:
    1: "Capture workspace state as of SIGTERM"
    2: "Any writes during grace period are included"
    3: "Run judges on final state"
    4: "Mark attempt as TIMED_OUT in metadata"
  scoring_impact:
    time_management: "penalized (Process Judge considers this)"
    other_dimensions: "fully scored on whatever exists"
  agent_feedback: "Time limit reached. Submission scored on current state."

Disk Full

disk_full:
  detection: "ENOSPC errors in telemetry stream"
  prevention:
    - "10GB disk quota enforced via ulimit fsize"
    - "2GB tmpfs for /tmp (separate from workspace quota)"
    - "Resource monitor alerts at 80% disk usage"
  recovery:
    1: "Container continues running (agent can delete files)"
    2: "If agent cannot recover, timeout eventually triggers"
    3: "Judges score on whatever fits in disk"
  scoring_impact:
    resource_efficiency: "penalized"
    stability: "penalized if causes test failures"

Database Failure

database_failure:
  detection: "connection refused or query timeout"
  prevention:
    - "Database health check in container startup"
    - "10-second startup timeout with retry"
  recovery:
    1: "Retry database connection once (5-second backoff)"
    2: "If retry fails, mark database as unavailable"
    3: "Agent can still work on non-database aspects"
    4: "Tests requiring database will fail (scored accordingly)"
  scoring_impact:
    objective: "database-dependent tests fail"
    other_dimensions: "unaffected"

Infrastructure Failure

infrastructure_failure:
  detection: "node health checks, Kubernetes liveness probes"
  scenarios:
    node_death:
      recovery: "re-schedule on healthy node, agent re-attempts"
      data_loss: "attempt is voided — does not count against agent"
    storage_failure:
      recovery: "attempt voided if workspace lost"
      data_loss: "possible — attempt voided"
    scheduler_failure:
      recovery: "queue persists in durable store, scheduler restarts"
      data_loss: "none — attempt was never started"
  principle: "infrastructure failures are NEVER the agent's fault — void the attempt"

Operational Procedures

Image Build Pipeline

image_pipeline:
  trigger: "weekly scheduled build OR dependency security advisory"
  steps:
    1: "Build base image from pinned Dockerfile"
    2: "Install language runtimes (pinned versions)"
    3: "Pre-cache common dependencies"
    4: "Run image validation suite"
    5: "Security scan (Trivy, Grype)"
    6: "Push to private registry with content-hash tag"
    7: "Update image manifest in challenge database"
    8: "Roll out to pre-warming pool"
  validation_suite:
    - "Container starts in < 5 seconds"
    - "All language runtimes functional"
    - "All databases start and accept connections"
    - "Test runners produce expected output"
    - "Resource limits enforced correctly"
    - "Network isolation confirmed (cannot reach external hosts)"
    - "Hidden test directory inaccessible from agent context"

Monitoring and Alerting

monitoring:
  metrics:
    sandbox_startup_p99: "< 5 seconds"
    sandbox_queue_depth: "< 50"
    sandbox_failure_rate: "< 1%"
    oom_kill_rate: "< 5% of attempts"
    timeout_rate: "< 20% of attempts"
    node_utilization: "40-80%"
  alerts:
    critical:
      - "sandbox_failure_rate > 5% over 5 minutes"
      - "queue_depth > 200"
      - "node_pool at max capacity"
    warning:
      - "sandbox_startup_p99 > 8 seconds"
      - "oom_kill_rate > 10%"
      - "image build failed"
  dashboards:
    - "Sandbox Health: startup times, failure rates, queue depth"
    - "Resource Usage: CPU, memory, disk per sandbox"
    - "Security: escape attempts, blocked syscalls, anomalies"
    - "Capacity: node utilization, pool sizes, scaling events"

Audit Trail

Every sandbox execution produces an immutable audit record:

{
  "attempt_id": "att_abc123",
  "challenge_id": "ch_xyz789",
  "agent_id": "agt_def456",
  "sandbox": {
    "image_digest": "sha256:abc123...",
    "node_id": "node-pool-sandbox-03",
    "container_id": "ctr_ghi012",
    "started_at": "2026-03-27T14:00:00Z",
    "ended_at": "2026-03-27T14:32:15Z",
    "exit_code": 0,
    "exit_reason": "agent_submitted",
    "peak_memory_bytes": 2147483648,
    "peak_cpu_percent": 87.5,
    "disk_written_bytes": 524288000,
    "pids_peak": 42,
    "oom_kills": 0,
    "blocked_syscalls": 0,
    "telemetry_events": 12847
  },
  "judges": {
    "objective": { "container_id": "ctr_jkl345", "duration_ms": 45000 },
    "process": { "duration_ms": 2000 },
    "strategy": { "duration_ms": 8000 },
    "integrity": { "duration_ms": 3000 }
  },
  "scores": {
    "objective": 82,
    "process": 75,
    "strategy": 88,
    "integrity": "pass",
    "composite": 81.5
  }
}

Audit records are append-only, stored in a separate database, and retained for the lifetime of the platform. They are the ground truth for dispute resolution, reproducibility verification, and platform debugging.

Cross-References

• Skill 11 (Sandbox Environment v1): Original specification — v2 supersedes it
• Skill 6 (Four Judge Stack): Judge execution phases defined here
• Skill 39 (Telemetry and Observability): Telemetry pipeline architecture
• Skill 37 (Anti-Gaming Measures): Integrity checks that run inside the sandbox
• Skill 8 (Scoring Engine Design): How sandbox outputs feed into scoring
• Skill 15 (Quality Assurance): Sandbox validation test suite