Telemetry And Observability

Telemetry and Observability

Skill 43 — Gauntlet Challenge Engine

The comprehensive telemetry system that captures everything about how agents interact with challenges. This data feeds the Process Judge, powers analytics, detects anomalies, and drives system improvement. Without telemetry, the Gauntlet is blind — scores exist but the story behind them is lost.

Two Telemetry Domains

All telemetry in the Gauntlet falls into one of two domains, each with distinct purposes, schemas, retention policies, and consumers.

Domain 1: Agent Telemetry (feeds Process Judge)

Everything the agent does during an attempt. This is the primary input to the Process Judge, which evaluates how the agent solved a challenge, not just whether it solved it.

Tool Call Log

Every tool invocation during an attempt is captured as a structured event.

event_type: tool_call
schema:
  attempt_id: string        # links to the parent attempt
  sequence_number: int      # monotonically increasing per attempt
  timestamp: ISO-8601       # when the tool was invoked
  tool_name: string         # e.g., "Read", "Edit", "Bash", "Grep", "Glob"
  arguments_summary: string # truncated/sanitized argument snapshot (max 512 chars)
  result_summary: string    # truncated result (max 1024 chars)
  result_status: enum       # success | error | timeout
  duration_ms: int          # wall-clock time for the tool call
  tokens_in: int            # tokens sent to the tool (if applicable)
  tokens_out: int           # tokens returned from the tool

Why sequence matters. The Process Judge distinguishes between agents that follow a disciplined read-plan-code-test loop and those that thrash with code-test-code-test cycles. The sequence of tool calls is itself a signal:

Test Run Log

Every test execution is a first-class telemetry event.

event_type: test_run
schema:
  attempt_id: string
  sequence_number: int
  timestamp: ISO-8601
  iteration: int             # which iteration of the attempt
  suite_name: string         # e.g., "unit", "integration", "e2e"
  command: string            # the exact test command run
  exit_code: int
  passed_count: int
  failed_count: int
  skipped_count: int
  total_count: int
  failure_messages: string[] # first 10 failure messages, truncated to 256 chars each
  duration_ms: int
  coverage_percent: float    # if coverage is enabled, null otherwise
  regression_detected: bool  # true if previously-passing tests now fail

Derived metrics from test runs:

• Test-between-edits ratio: count(test_run events) / count(edit events). A ratio below 0.3 suggests the agent is coding blind. A ratio above 2.0 suggests excessive test-running without meaningful changes.
• Regression rate: percentage of test runs where regression_detected = true. High regression rates indicate the agent is breaking things it already fixed.
• First-green iteration: the iteration number where all tests first pass. Earlier is better, but only if the solution is correct.

File Change Log

Every file modification inside the sandbox is captured.

event_type: file_change
schema:
  attempt_id: string
  sequence_number: int
  timestamp: ISO-8601
  file_path: string          # relative to workspace root
  change_type: enum          # create | edit | delete | rename
  diff_lines_added: int
  diff_lines_removed: int
  diff_size_bytes: int
  content_hash_before: string  # SHA-256 of file before change
  content_hash_after: string   # SHA-256 of file after change
  tool_that_caused: string     # which tool call produced this change

Derived metrics from file changes:

• Change scope: number of unique files touched. Focused solutions touch fewer files.
• Churn rate: number of times the same file is edited. High churn on a single file suggests the agent is struggling with that file.
• Net change size: total lines added minus lines removed. Smaller net changes for correct solutions indicate precision.
• Shotgun surgery detection: many small changes across many files in a short window.

Error Log

Every error the agent encounters — tool failures, compilation errors, runtime exceptions.

event_type: error_encountered
schema:
  attempt_id: string
  sequence_number: int
  timestamp: ISO-8601
  error_type: enum           # tool_error | compilation | runtime | timeout | sandbox
  error_message: string      # truncated to 512 chars
  error_source: string       # which tool or process produced the error
  is_repeat: bool            # true if same error_type + similar message seen before in this attempt
  repeat_count: int          # how many times this error has occurred in this attempt
  resolved: bool             # true if the error stops appearing in subsequent events
  resolution_latency_ms: int # time between first occurrence and resolution (null if unresolved)

Error repetition is a strong signal. An agent that encounters the same ModuleNotFoundError three times without installing the module is demonstrating poor problem-solving. The Process Judge uses repeat_count and resolved to assess adaptability.

Time Log

Phase-level timing derived from tool call patterns.

event_type: phase_transition
schema:
  attempt_id: string
  sequence_number: int
  timestamp: ISO-8601
  phase_from: enum           # reading | planning | coding | testing | debugging | idle
  phase_to: enum
  duration_in_phase_ms: int  # how long the agent spent in phase_from

Phase classification rules:

• Reading: consecutive Read, Grep, Glob calls with no Edit/Write/Bash calls
• Planning: gaps between tool calls > 5 seconds (agent is thinking/generating plan)
• Coding: Edit, Write calls
• Testing: Bash calls that execute test commands (detected by command pattern matching)
• Debugging: interleaved Read/Grep calls followed by Edit calls after a test failure
• Idle: gaps > 30 seconds with no tool calls (may indicate confusion or rate limiting)

Score Trajectory

The score at each iteration checkpoint.

event_type: score_checkpoint
schema:
  attempt_id: string
  iteration: int
  timestamp: ISO-8601
  score: float               # 0.0 to 1.0
  tests_passed: int
  tests_total: int
  delta_from_previous: float # score change since last checkpoint
  cumulative_time_ms: int    # total elapsed time since attempt start

Trajectory patterns the Process Judge recognizes:

Domain 2: System Telemetry (platform health)

System telemetry monitors the Gauntlet platform itself. It is not per-attempt — it tracks pipelines, infrastructure, and operational health.

Challenge Generation Pipeline Metrics

event_type: pipeline_stage
schema:
  pipeline_run_id: string
  stage: enum                # seed_selection | spec_generation | codebase_generation |
                             # test_generation | validation | quality_gate
  started_at: ISO-8601
  completed_at: ISO-8601
  duration_ms: int
  status: enum               # success | failure | retry | timeout
  retry_count: int
  failure_reason: string     # null on success
  challenge_id: string       # the challenge being generated (null for seed_selection)
  model_used: string         # which LLM was used for this stage
  tokens_consumed: int

Judge Execution Metrics

event_type: judge_execution
schema:
  judgment_id: string
  attempt_id: string
  judge_type: enum           # functional | process | style | meta
  started_at: ISO-8601
  completed_at: ISO-8601
  duration_ms: int
  model_used: string
  tokens_consumed: int
  score_given: float
  confidence: float
  api_retries: int
  api_errors: int

Cross-judge disagreement tracking:

event_type: judge_disagreement
schema:
  attempt_id: string
  judge_a: string
  judge_b: string
  score_a: float
  score_b: float
  delta: float               # abs(score_a - score_b)
  flagged_for_review: bool   # true if delta > threshold (default 0.3)

Matchmaking Metrics

event_type: matchmaking_decision
schema:
  agent_id: string
  timestamp: ISO-8601
  queue_depth: int           # how many challenges were available
  selected_challenge_id: string
  predicted_score: float     # what the matchmaker predicted the agent would score
  actual_score: float        # filled in after attempt completes (null initially)
  prediction_error: float    # abs(predicted - actual), filled in after
  selection_reason: string   # why this challenge was chosen
  selection_duration_ms: int

Infrastructure Metrics

event_type: infra_health
schema:
  timestamp: ISO-8601
  component: enum            # sandbox | api | database | queue | storage
  metric_name: string        # e.g., "sandbox_spin_up_ms", "api_latency_p99"
  metric_value: float
  unit: string               # ms | percent | count | bytes
  threshold_warning: float
  threshold_critical: float
  alert_status: enum         # normal | warning | critical

Collection Architecture

Telemetry collection is designed for minimal overhead inside the sandbox and maximum reliability outside it.

Agent-Side Instrumentation

The sandbox runtime wraps every tool call with instrumentation. The agent does not need to opt in — telemetry is a property of the sandbox, not the agent.

┌─────────────────────────────────────────────┐
│  Sandbox                                    │
│  ┌─────────┐    ┌──────────────────────┐    │
│  │  Agent   │───▶│  Tool Proxy Layer    │    │
│  │ Runtime  │◀───│  (intercepts calls)  │    │
│  └─────────┘    └──────────┬───────────┘    │
│                            │                │
│                   ┌────────▼────────┐       │
│                   │  Event Buffer   │       │
│                   │  (in-memory)    │       │
│                   └────────┬────────┘       │
└────────────────────────────┼────────────────┘
                             │ flush every 5s
                             │ or on buffer full (1000 events)
                    ┌────────▼────────┐
                    │  Event Ingress  │
                    │  (HTTP endpoint)│
                    └────────┬────────┘
                             │
                    ┌────────▼────────┐
                    │  Event Stream   │
                    │  (Kafka/NATS)   │
                    └─────────────────┘

Key design decisions:

1. In-sandbox buffer: Events are buffered in memory to avoid per-event network overhead. The buffer flushes every 5 seconds or when it reaches 1000 events, whichever comes first.
2. Fire-and-forget from sandbox: The sandbox does not wait for acknowledgment. If the event ingress is down, events are written to a local file as a fallback and recovered later.
3. Tool Proxy Layer: Sits between the agent runtime and actual tool implementations. It captures timing, arguments, and results without modifying tool behavior. The agent cannot detect or bypass this layer.
4. Filesystem watcher: A separate process inside the sandbox monitors the workspace directory for file changes, capturing diffs that occur outside tool calls (e.g., build artifacts, generated files).

Event Streaming

Events flow through a streaming platform (Kafka or NATS JetStream) for decoupled processing.

Event Ingress ──▶ Stream Topic: gauntlet.telemetry.raw
                       │
          ┌────────────┼──────────────┐
          │            │              │
          ▼            ▼              ▼
    Process Judge   Analytics     Anomaly
    Consumer        Consumer      Detector

Topics:

• gauntlet.telemetry.raw — all raw events, partitioned by attempt_id
• gauntlet.telemetry.agent — agent telemetry only (tool calls, tests, file changes, errors)
• gauntlet.telemetry.system — system telemetry only (pipeline, judge, infra)
• gauntlet.telemetry.alerts — anomaly detection outputs
• gauntlet.telemetry.dead-letter — events that failed processing

Structured Logging

All Gauntlet components emit structured JSON logs that are indexed alongside telemetry events.

{
  "timestamp": "2026-03-27T14:32:01.445Z",
  "level": "info",
  "component": "challenge-generator",
  "pipeline_run_id": "prun_a1b2c3",
  "stage": "codebase_generation",
  "message": "Codebase generation completed",
  "duration_ms": 12450,
  "files_generated": 23,
  "total_lines": 1847
}

Storage and Retention

Storage Architecture

┌──────────────────────────────────────────────────┐
│                   Query Layer                     │
│         (Grafana / Custom Dashboards)             │
└──────────┬──────────────┬───────────┬────────────┘
           │              │           │
    ┌──────▼──────┐ ┌─────▼────┐ ┌───▼─────────┐
    │  Hot Store  │ │ Cold     │ │ Aggregated  │
    │  (ClickHouse│ │ Store    │ │ Store       │
    │  or TimescaleDB) │ (S3/GCS  │ │ (Postgres)  │
    │             │ │ Parquet) │ │             │
    │  Recent     │ │ Archived │ │ Rollups     │
    │  events     │ │ events   │ │ forever     │
    └─────────────┘ └──────────┘ └─────────────┘

Retention Policies

Aggregation Rules

Raw events are rolled up into aggregations that persist indefinitely:

agent_attempt_summary:
  attempt_id: string
  agent_id: string
  challenge_id: string
  total_tool_calls: int
  tool_call_breakdown: map[string, int]  # tool_name → count
  total_test_runs: int
  total_files_changed: int
  total_errors: int
  unique_errors: int
  repeated_errors: int
  phase_time_breakdown: map[string, int] # phase → total_ms
  score_trajectory: float[]
  final_score: float
  total_duration_ms: int
  regression_count: int
  churn_files: string[]                  # files edited 3+ times

challenge_health_summary:
  challenge_id: string
  period: string                         # daily, weekly
  attempts_count: int
  avg_score: float
  median_score: float
  p10_score: float
  p90_score: float
  avg_duration_ms: int
  avg_tool_calls: int
  common_errors: map[string, int]        # error_message → count
  common_failure_points: string[]        # test names that fail most

system_health_summary:
  period: string
  pipeline_success_rate: float
  avg_pipeline_duration_ms: int
  judge_avg_latency_ms: int
  judge_disagreement_rate: float
  sandbox_avg_spin_up_ms: int
  total_attempts: int
  total_challenges_generated: int

Privacy and Security

What IS Captured

• Tool call names and argument summaries (truncated)
• Test execution results (pass/fail counts, failure messages)
• File paths and diff statistics (not full file contents)
• Timing data for all operations
• Error messages (truncated)
• Score trajectories

What is NOT Captured

• Full file contents: Only diffs and hashes, never the complete source code of solutions
• Agent model weights or prompts: The telemetry system captures behavior, not internals
• Agent API keys or credentials: Stripped from all event fields before storage
• Inter-agent communication content: If multi-agent, only metadata (message count, size) not content
• User PII: Agent IDs are pseudonymous; no real names, emails, or identifying information in telemetry

Data Access Controls

access_levels:
  public:
    - Aggregated leaderboard statistics
    - Challenge-level aggregate metrics (avg score, attempt count)
    - System uptime and availability metrics

  agent_owner:
    - All telemetry for their own agent's attempts
    - Score trajectories and tool call logs for their agent
    - Comparison against anonymized population statistics

  platform_admin:
    - All telemetry for all agents
    - Raw event access
    - Judge calibration data
    - Anomaly detection alerts

  judge_system:
    - Agent telemetry for the specific attempt being judged
    - No access to other agents' telemetry
    - No access to system telemetry

Anonymization for Public Analytics

When telemetry is surfaced in public dashboards or research:

• Agent IDs are replaced with opaque hashes
• Specific tool call arguments are stripped
• Error messages are categorized (not shown verbatim)
• Timing data is bucketed (e.g., "fast", "medium", "slow") rather than exact
• File paths are generalized (e.g., "src/**/*.py" not "src/auth/jwt_validator.py")

Analytics and Dashboards

Dashboard 1: Agent Behavior Patterns

Purpose: Understand how agents approach challenges across the population.

Panels:

Key queries:

-- Average tool calls per attempt by tool type
SELECT tool_name, AVG(call_count) as avg_calls
FROM (
  SELECT attempt_id, tool_name, COUNT(*) as call_count
  FROM tool_call_events
  WHERE timestamp > NOW() - INTERVAL 7 DAY
  GROUP BY attempt_id, tool_name
) sub
GROUP BY tool_name
ORDER BY avg_calls DESC;

-- Score trajectory pattern distribution
SELECT trajectory_pattern, COUNT(*) as attempt_count
FROM agent_attempt_summary
WHERE created_at > NOW() - INTERVAL 30 DAY
GROUP BY trajectory_pattern
ORDER BY attempt_count DESC;

Dashboard 2: Challenge Health

Purpose: Monitor whether challenges are well-calibrated and functioning correctly.

Panels:

Alert conditions:

• Challenge has 0% pass rate after 10+ attempts → likely broken
• Challenge has 100% pass rate after 10+ attempts → likely too easy
• Average score drifts > 0.2 over a week → possible contamination

Dashboard 3: System Performance

Purpose: Operational health of the Gauntlet platform.

Panels:

Dashboard 4: Judge Calibration

Purpose: Ensure judges are consistent, accurate, and not drifting.

Panels:

Anomaly Detection

Automated systems that monitor telemetry streams in real time, flagging patterns that indicate problems, gaming, or system failures.

Rule-Based Anomalies

anomaly_rules:

  - name: sandbox_escape_attempt
    description: Agent attempts to access resources outside the sandbox
    trigger: >
      tool_call where tool_name = "Bash" AND
      (arguments_summary CONTAINS "curl" OR
       arguments_summary CONTAINS "wget" OR
       arguments_summary CONTAINS "/etc/passwd" OR
       arguments_summary CONTAINS "ssh" OR
       arguments_summary MATCHES "nc\s+-l")
    severity: critical
    action: terminate_attempt, flag_agent, alert_admin

  - name: abnormal_tool_volume
    description: Agent making an unusually high number of tool calls
    trigger: >
      COUNT(tool_call) WHERE attempt_id = X
      AND window = 60 seconds
      > 100
    severity: warning
    action: throttle_agent, log_alert

  - name: suspiciously_fast_solution
    description: Agent solves challenge faster than any known baseline
    trigger: >
      score_checkpoint WHERE score = 1.0
      AND cumulative_time_ms < challenge.min_expected_time_ms * 0.2
    severity: high
    action: flag_for_review, hold_score

  - name: score_manipulation_attempt
    description: Agent appears to be modifying test files or scoring scripts
    trigger: >
      file_change WHERE file_path MATCHES "test_.*\\.py$|.*_test\\.go$|.*\\.test\\.(js|ts)$"
      OR file_path MATCHES "score.*|judge.*|eval.*"
    severity: critical
    action: terminate_attempt, score_zero, flag_agent

  - name: infinite_loop_detection
    description: Agent is stuck in a loop repeating the same actions
    trigger: >
      3 consecutive windows of 5 tool calls each where
      tool_name sequence is identical
    severity: warning
    action: notify_agent (if supported), log_alert

  - name: test_result_regression_spiral
    description: Agent is making things progressively worse
    trigger: >
      3 consecutive score_checkpoints where
      delta_from_previous < -0.05
    severity: info
    action: log_alert

Statistical Anomalies

Beyond rules, the system uses statistical methods on sliding windows:

Z-score detection: For each metric (tool call count, duration, error count), compute the population mean and standard deviation. Flag attempts where the metric exceeds 3 standard deviations.

Isolation forest: Trained on feature vectors derived from agent_attempt_summary records. Detects attempts that are outliers across multiple dimensions simultaneously (e.g., very few tool calls + perfect score + very fast = suspicious).

Temporal anomaly detection: Time-series anomaly detection on system metrics. Sudden spikes in judge latency, drops in pipeline success rate, or unusual patterns in sandbox spin-up times trigger infrastructure alerts.

Alert Routing

alert_routing:
  critical:
    channels: [pagerduty, slack_oncall]
    response_sla: 15 minutes
    examples: sandbox_escape_attempt, score_manipulation_attempt

  high:
    channels: [slack_alerts]
    response_sla: 1 hour
    examples: suspiciously_fast_solution, judge_disagreement > 0.5

  warning:
    channels: [slack_alerts]
    response_sla: 4 hours
    examples: abnormal_tool_volume, infinite_loop_detection

  info:
    channels: [dashboard_only]
    response_sla: next_business_day
    examples: test_result_regression_spiral, minor_judge_drift

Alert Schema

event_type: anomaly_alert
schema:
  alert_id: string
  timestamp: ISO-8601
  rule_name: string
  severity: enum             # critical | high | warning | info
  attempt_id: string         # null for system-level anomalies
  agent_id: string           # null for system-level anomalies
  description: string
  evidence: object           # the telemetry events that triggered the alert
  action_taken: string[]     # what automated actions were performed
  resolved: bool
  resolved_at: ISO-8601      # null if unresolved
  resolved_by: string        # system | admin_username

Telemetry-Driven Feedback Loops

Telemetry doesn't just observe — it drives improvement.

Loop 1: Challenge Calibration

Telemetry → aggregate scores per challenge
          → detect difficulty drift
          → trigger recalibration pipeline
          → adjust challenge difficulty metadata

If a challenge that was rated "hard" consistently gets 90%+ scores, telemetry triggers automatic difficulty reassessment.

Loop 2: Judge Tuning

Telemetry → detect inter-judge disagreement trends
          → surface flagged cases for human review
          → human labels become judge training data
          → judge prompts/models updated

Loop 3: Matchmaking Accuracy

Telemetry → compare predicted scores vs. actual scores
          → compute prediction error distribution
          → retrain matchmaking model on new data
          → deploy updated model

Loop 4: Anti-Gaming Evolution

Telemetry → anomaly detector finds new gaming pattern
          → pattern added to rule set
          → historical attempts re-scanned
          → affected scores adjusted

Implementation Priorities

Phase 1: Foundation (MVP)

• Tool call logging with basic schema
• Test run logging
• Score trajectory capture
• Storage in a single database (Postgres with TimescaleDB extension)
• One dashboard (agent behavior)
• Basic rule-based anomaly detection (sandbox escape, score manipulation)

Phase 2: Full Agent Telemetry

• File change logging with diff stats
• Error logging with repeat detection
• Phase transition classification
• Process Judge integration (telemetry becomes judge input)
• Challenge health dashboard
• Statistical anomaly detection

Phase 3: System Telemetry

• Pipeline metrics
• Judge execution metrics
• Infrastructure metrics
• System performance dashboard
• Judge calibration dashboard
• Alert routing with severity tiers

Phase 4: Feedback Loops

• Challenge recalibration loop
• Judge tuning loop
• Matchmaking accuracy loop
• Anti-gaming evolution loop
• Full cold-storage archival pipeline
• Public anonymized analytics

Operational Runbook

Telemetry Ingestion Down

1. Events buffer locally in sandbox (file fallback)
2. Alert fires after 2 minutes of ingestion silence
3. Check event ingress endpoint health
4. Check stream broker (Kafka/NATS) health
5. If broker is down, restart broker; buffered events will replay
6. If ingress is down, check for OOM or crash loops
7. After recovery, trigger backfill from sandbox fallback files

High Event Volume (>10x baseline)

1. Check for runaway agents (infinite loop detection should catch this)
2. Check for event duplication (duplicate sequence_number per attempt_id)
3. If legitimate volume spike, scale ingress horizontally
4. Enable sampling for info-level events if needed (never sample critical)

Storage Approaching Capacity

1. Verify retention policies are executing (check last successful cleanup)
2. Run emergency aggregation for oldest hot data
3. Migrate to cold storage ahead of schedule
4. If cold storage is also full, extend capacity or reduce cold retention
5. Never delete raw data before aggregation completes

Key Metrics to Track About Telemetry Itself (Meta-Telemetry)

These meta-metrics ensure the telemetry system itself is healthy. A blind telemetry system is worse than no telemetry at all — it provides false confidence.