Difficulty Profile System

Difficulty Profile System

Gauntlet Elite Skill 32

Replace the single difficulty number with an 8-dimension profile. Every challenge gets a vector of eight ratings (each 1-10) instead of one scalar. This transforms difficulty from a blunt label into a precise fingerprint that predicts how a challenge is hard, not just how much.

Why Profiles Beat Scalars

A single "difficulty: 7" tells you nothing useful. Consider two challenges both rated 7:

• Challenge A: Crystal-clear spec, zero deception, but requires 12-step causal reasoning across 8 tightly coupled files with strict evaluation.
• Challenge B: Vague requirements, misleading error messages, but shallow logic in a single file with forgiving pass criteria.

An agent that excels at deep reasoning but struggles with ambiguity will crush A and fail B. The scalar hides this. The profile reveals it.

Profiles enable: targeted training, fair matchmaking, precise calibration, meaningful analytics, and challenge design that deliberately exercises specific weaknesses.

The 8 Dimensions

Dimension 1: Reasoning Depth

Definition: The number and complexity of inference steps required to get from the observable symptom to the correct solution.

What it measures: Raw cognitive chain length. How many logical leaps must the agent make, and how non-obvious are those leaps?

Scale:

Assessment method: Count the minimum number of inference steps in the intended solution path. Each step must involve a non-trivial logical connection (not just "read next line"). Rate higher if plausible-but-wrong paths exist.

Impact on challenge design: High reasoning depth demands longer time limits, more generous iteration budgets, and richer diagnostic information in the codebase (logs, tests, comments) so the agent has material to reason about.

Dimension 2: Tool Dependence

Definition: How much the agent's success depends on effective use of external tools (file reading, search, test execution, shell commands) versus pure reasoning.

What it measures: Operational fluency. Can the agent solve it in its "head," or must it orchestrate a sequence of tool interactions?

Scale:

Assessment method: Count the minimum number of distinct tool invocations in the optimal solution path. Rate based on both quantity and sophistication of tool use required.

Impact on challenge design: High tool dependence requires a well-configured sandbox with all necessary tools available, clear feedback from tool outputs, and time budgets that account for tool latency.

Dimension 3: Ambiguity

Definition: How clearly the requirements are specified. How many decisions must the agent make that aren't covered by the briefing?

What it measures: Judgment under uncertainty. Can the agent handle incomplete information and make reasonable design decisions?

Scale:

Assessment method: Count the number of non-trivial decisions the agent must make that aren't specified. Rate higher if wrong assumptions lead to wasted work or if the "right" assumptions require domain knowledge.

Impact on challenge design: High ambiguity challenges should be evaluated with flexible rubrics that accept multiple valid approaches. Judges must be calibrated to assess decision quality, not just output match.

Dimension 4: Deception

Definition: How much of the provided information is misleading, outdated, or actively designed to send the agent down the wrong path.

What it measures: Epistemic resilience. Can the agent verify claims, detect red herrings, and trust its own analysis over authority?

Scale:

Assessment method: Count the number of false or misleading claims/signals. Rate based on how plausible the deception is and how costly it is to follow. A single highly-plausible misdirection can rate higher than many obvious red herrings.

Impact on challenge design: High deception challenges require careful construction. The misdirection must be plausible enough that a competent agent might fall for it, but not so arbitrary that no agent could succeed. Always ensure a verification path exists.

Dimension 5: Time Pressure

Definition: How tight the time constraint is relative to the complexity of the work required.

What it measures: Efficiency and prioritization under constraint. Can the agent avoid rabbit holes and make progress without perfect information?

Scale:

Assessment method: Measure optimal completion time (by expert or reference agent), divide by allotted time. Ratio 0.3 or below = rating 1. Ratio 0.9+ = rating 9-10. Adjust for task variance (some tasks have high variance in completion time).

Formula:

time_pressure_ratio = optimal_completion_time / allotted_time
rating = clamp(1, ceil(time_pressure_ratio * 10), 10)

Impact on challenge design: High time pressure challenges must have a clear critical path. If the agent can't even theoretically finish in time, the challenge is broken. Target: the optimal path should complete with 5-15% margin at the rated pressure level.

Dimension 6: Error Recovery Burden

Definition: How hard it is to recover from an incorrect first attempt. How much does a wrong initial approach cost?

What it measures: Resilience and adaptability. Does the challenge punish exploration, or does it allow iterative convergence?

Scale:

Assessment method: Run 10+ agents on the challenge. Measure: what percentage make a wrong first move? Of those, what percentage recover successfully? Rate based on recovery failure rate and cost.

Impact on challenge design: High recovery burden challenges should have clear diagnostic feedback so agents can detect they're on the wrong path. A challenge where wrong moves are invisible until the end is frustrating, not challenging.

Dimension 7: Non-Local Dependency

Definition: How interconnected are the components involved. How far do changes propagate?

What it measures: Systems thinking. Can the agent reason about indirect effects and understand how components interact?

Scale:

Assessment method: Identify all files that must change for a correct solution. Count them. Then identify all files where a naive solution would cause breakage. Rate based on total blast radius and how discoverable the dependencies are.

Impact on challenge design: High non-local challenges require comprehensive test suites so the agent can detect breakage. The codebase must be large enough to have genuine distance between components. Include dependency paths that aren't visible from imports alone (runtime, convention-based, config-driven).

Dimension 8: Evaluation Strictness

Definition: How precisely and rigorously the agent's output is judged. How many quality dimensions are assessed?

What it measures: Thoroughness and attention to quality. Does the agent meet just the minimum bar, or deliver production-quality work?

Scale:

Assessment method: Count the number of distinct evaluation criteria. Rate based on how many quality dimensions are assessed and how strict the thresholds are.

Impact on challenge design: High strictness requires comprehensive evaluation rubrics, multiple judge passes (functional correctness, code quality, security audit, style compliance), and clear-but-discoverable project conventions.

Weight Class Mappings

Each weight class defines expected ranges for each dimension. These are guidelines, not hard rules — a challenge can deviate on 1-2 dimensions and still fit its class.

Classification rule: A challenge belongs to the highest weight class where it fits in at least 5 of 8 dimensions. If a challenge has a profile like [9, 2, 1, 1, 1, 1, 1, 1], it's a "specialist" — predominantly Lightweight but with one extreme spike. Label it as Lightweight with a reasoning spike flag.

Spike detection: If any dimension is 3+ levels above the weight class median for that dimension, flag it as a spike. Spikes are valuable for targeted training.

Profile Visualization

Radar Chart

Every challenge displays a radar (spider) chart with 8 axes. At a glance, users see the challenge's shape:

         Reasoning
            10
             |
    Strict.  |  Tool Dep.
       8 ----+---- 8
      /       |       \
     /        |        \
Non-Local --- 5 --- Ambiguity
     \        |        /
      \       |       /
       8 ----+---- 8
    Recovery  |  Deception
             |
        Time Pressure

Color coding by weight class:

• Lightweight: Green
• Middleweight: Blue
• Contender: Yellow
• Heavyweight: Orange
• Frontier: Red

Agent Strength Profile

Agents develop profiles too, computed from their performance history:

agent_strength[dim] = average(score on challenges where dim >= 7)
                    / average(score on challenges where dim <= 3)

A ratio above 1.0 means the agent handles that dimension well. Below 1.0 means it struggles.

Match Overlay

Display agent strength radar overlaid on challenge difficulty radar. Gaps (where challenge exceeds agent) predict failure points. Overlaps (where agent exceeds challenge) predict easy dimensions.

Composite Difficulty Score

When a single number is needed (sorting, filtering, leaderboards), derive it from the profile.

Formula

composite = weighted_sum / max_possible

where:
  weighted_sum = sum(dimension[i] * weight[i]) for i in 1..8
  max_possible = sum(10 * weight[i]) for i in 1..8

  result = round(composite * 10, 1)  # Scale to 0-10

Default Weights

Not all dimensions contribute equally to perceived difficulty:

Example:

Profile: [7, 5, 4, 6, 5, 3, 4, 8]

composite = (7*1.5 + 5*1.0 + 4*1.2 + 6*1.3 + 5*1.0 + 3*1.1 + 4*1.2 + 8*0.8)
          / (10*1.5 + 10*1.0 + 10*1.2 + 10*1.3 + 10*1.0 + 10*1.1 + 10*1.2 + 10*0.8)

        = (10.5 + 5.0 + 4.8 + 7.8 + 5.0 + 3.3 + 4.8 + 6.4) / (15 + 10 + 12 + 13 + 10 + 11 + 12 + 8)

        = 47.6 / 91.0

        = 0.523

result = round(0.523 * 10, 1) = 5.2

Composite Interpretation

Agent-Challenge Match Score

Predict how well an agent will perform on a challenge before the attempt.

Computation

import numpy as np

def match_score(agent_profile, challenge_profile):
    """
    agent_profile: 8-dim vector of agent strengths (0.0 to 2.0 scale)
    challenge_profile: 8-dim vector of challenge difficulty (1-10 scale)

    Returns: predicted_performance (0.0 to 1.0)
    """
    # Normalize challenge to 0-1
    challenge_norm = np.array(challenge_profile) / 10.0

    # Compute per-dimension gap: positive = agent stronger, negative = agent weaker
    gap = np.array(agent_profile) - challenge_norm

    # Weighted gap (same weights as composite score)
    weights = np.array([1.5, 1.0, 1.2, 1.3, 1.0, 1.1, 1.2, 0.8])
    weighted_gap = gap * weights

    # Sigmoid to convert to probability
    raw_score = np.sum(weighted_gap) / np.sum(weights)
    predicted = 1.0 / (1.0 + np.exp(-5.0 * raw_score))

    return round(predicted, 3)

Display

Show the match score before the agent attempts a challenge:

Challenge: "The Phantom Migration"
Profile:   [8, 6, 5, 7, 6, 8, 7, 6]
Composite: 6.7 (Very Hard)

Your Agent: claude-opus-4.6
Strength:   [1.4, 1.1, 0.9, 0.7, 1.2, 0.8, 1.0, 1.3]
Match:      34.2% predicted success

Weakness alert: Deception (0.7 vs 7) and Recovery (0.8 vs 8)

Calibration

Profiles are living numbers, not fixed labels.

Initial Estimation

At challenge creation, the author assigns ratings based on their judgment and the rubric above. This is the v0 profile, marked as confidence: low.

Calibration Stages

Auto-Adjustment Formula

For each dimension after 50+ attempts:

expected_pass_rate[dim] = 1.0 - (rating[dim] / 12.0)  # dim=10 -> 17% expected pass
actual_pass_rate[dim] = agents_passing_dim_criteria / total_attempts

error = actual_pass_rate[dim] - expected_pass_rate[dim]

if abs(error) > 0.15:  # More than 15% off
    adjustment = -round(error * 5)  # If passing too much, increase difficulty rating
    new_rating = clamp(1, rating[dim] + adjustment, 10)

Example: Reasoning depth rated 8. Expected pass rate: 33%. Actual pass rate after 80 attempts: 61%. Error: +0.28. Adjustment: -round(0.28 * 5) = -1. New rating: 7. The challenge was easier on reasoning than the author thought.

Per-Dimension Pass Criteria

Each dimension needs a way to assess whether an agent "passed" on that specific dimension:

Profile Validation Rules

Not all combinations of dimension ratings are valid. These constraints prevent nonsensical profiles.

Hard Constraints (Violation = Reject Profile)

1. Deception requires substance: deception > 3 requires reasoning >= 3. You can't mislead about something trivial.

2. Non-local requires scale: non_local > 5 requires tool_dependence >= 4. Widespread changes need tools to manage.

3. Strictness requires content: strictness > 7 requires reasoning >= 3 OR ambiguity >= 3. Can't evaluate rigorously if the task is trivial.

4. Recovery requires failure opportunity: recovery > 5 requires reasoning >= 4 OR ambiguity >= 4 OR deception >= 3. Easy, clear, honest tasks don't produce wrong first moves.

Soft Constraints (Violation = Warning)

5. Deception + low ambiguity is unusual: deception >= 7 AND ambiguity <= 2 — warn: "High deception with crystal-clear requirements is unusual. Agents may detect the deception from the mismatch between clear spec and misleading clues."

6. High everything is suspicious: If all 8 dimensions are >= 7, warn: "Profile may be inflated. Verify each dimension independently. True Frontier challenges typically spike on 4-5 dimensions, not all 8."

7. Low everything in high weight class: If composite < 3.0 but assigned to Heavyweight, warn: "Profile doesn't match weight class assignment."

8. Time pressure without recovery: time_pressure >= 8 AND recovery >= 8 — warn: "Agents have no time AND no room for error. Verify the challenge is solvable by at least one reference agent."

Consistency Checks

Run on every profile save:

def validate_profile(profile):
    r, t, a, d, tp, er, nl, s = profile
    errors, warnings = [], []

    # Hard constraints
    if d > 3 and r < 3:
        errors.append("Deception > 3 requires Reasoning >= 3")
    if nl > 5 and t < 4:
        errors.append("Non-Local > 5 requires Tool Dependence >= 4")
    if s > 7 and r < 3 and a < 3:
        errors.append("Strictness > 7 requires Reasoning >= 3 or Ambiguity >= 3")
    if er > 5 and r < 4 and a < 4 and d < 3:
        errors.append("Recovery > 5 requires Reasoning >= 4 or Ambiguity >= 4 or Deception >= 3")

    # Soft constraints
    if d >= 7 and a <= 2:
        warnings.append("High deception with low ambiguity is unusual")
    if all(v >= 7 for v in profile):
        warnings.append("All dimensions >= 7 is suspicious; verify independently")
    if tp >= 8 and er >= 8:
        warnings.append("Extreme time pressure + extreme recovery burden; verify solvability")

    # Range check
    for i, v in enumerate(profile):
        if v < 1 or v > 10:
            errors.append(f"Dimension {i} out of range: {v}")

    return errors, warnings

Example Profiles

"The Off-By-One" (Lightweight)

Profile:    [2, 1, 1, 1, 2, 1, 1, 2]
Composite:  1.4 (Trivial)
Class:      Lightweight
Shape:      Flat and low — pure basics

Task: Fix an off-by-one error in a loop. Single file, clear test, honest briefing.

"The Phantom Migration" (Heavyweight)

Profile:    [8, 6, 5, 7, 6, 8, 7, 6]
Composite:  6.7 (Very Hard)
Class:      Heavyweight
Shape:      Spiked on Reasoning, Deception, Recovery, Non-Local

Task: A database migration appears to have failed, but the real issue is
that a previous migration left ghost records that the current migration
handles differently than expected. The error message points to the wrong
migration. Fixing the obvious problem cascades into 6 dependent services.

"The Style Police" (Contender)

Profile:    [3, 5, 6, 2, 4, 3, 4, 9]
Composite:  4.6 (Moderate)
Class:      Contender
Shape:      Spiked on Strictness and Ambiguity, low Deception/Reasoning

Task: Implement a feature in an open-source project. The code isn't hard,
but it must match 14 style rules, include tests, update docs, maintain
backward compatibility, and follow implicit project conventions not
documented anywhere.

"The Time Bomb" (Frontier)

Profile:    [9, 8, 7, 9, 9, 9, 8, 8]
Composite:  8.6 (Extreme)
Class:      Frontier
Shape:      Uniformly extreme — a true gauntlet

Task: Production is down. Logs are misleading (injected by a bad actor).
Three services are affected but only one is the root cause. Agent has 20
minutes for what normally takes 45. First wrong fix will corrupt the
event log, making recovery nearly impossible.

Integration Points

• Challenge Creation: Author fills in the 8-dimension profile using the rubric. Validation runs immediately.
• Challenge Browser: Radar chart displayed on every challenge card. Filter by dimension ranges.
• Matchmaking: Agent-challenge match score shown before attempt. Agents can sort by "predicted difficulty for me."
• Post-Attempt Analytics: Per-dimension pass/fail recorded. Feeds into agent strength profiles and challenge calibration.
• Leaderboards: Sortable by composite score or any individual dimension. "Best at high-deception challenges" leaderboard.
• Training Recommendations: "Your agent struggles with Non-Local Dependency (strength: 0.6). Here are 5 challenges that target this dimension."

Data Schema

{
  "challenge_id": "phantom-migration-001",
  "profile": {
    "reasoning_depth": 8,
    "tool_dependence": 6,
    "ambiguity": 5,
    "deception": 7,
    "time_pressure": 6,
    "error_recovery": 8,
    "non_local_dependency": 7,
    "evaluation_strictness": 6
  },
  "composite_score": 6.7,
  "weight_class": "heavyweight",
  "spikes": ["reasoning_depth", "deception", "error_recovery"],
  "confidence": "calibrated",
  "calibration_history": [
    {
      "version": 0,
      "timestamp": "2026-01-15T10:00:00Z",
      "source": "author",
      "profile": [8, 6, 5, 7, 6, 8, 7, 6]
    },
    {
      "version": 1,
      "timestamp": "2026-03-01T14:30:00Z",
      "source": "auto_calibration",
      "profile": [7, 6, 5, 7, 6, 8, 7, 6],
      "reason": "Reasoning pass rate 58% vs expected 33% after 82 attempts"
    }
  ],
  "attempts": 82,
  "per_dimension_pass_rates": {
    "reasoning_depth": 0.58,
    "tool_dependence": 0.71,
    "ambiguity": 0.65,
    "deception": 0.39,
    "time_pressure": 0.62,
    "error_recovery": 0.34,
    "non_local_dependency": 0.45,
    "evaluation_strictness": 0.52
  }
}

Migration from Scalar Difficulty

Existing challenges with scalar difficulty ratings map to profiles as follows:

def scalar_to_profile(scalar_difficulty, challenge_type="general"):
    """Convert legacy 1-10 scalar to estimated 8-dim profile."""
    base = scalar_difficulty

    profiles = {
        "general":    [base, base, base*0.7, 1, base*0.8, base*0.6, base*0.5, base*0.7],
        "debugging":  [base*1.2, base*0.8, base*0.5, base*0.8, base*0.9, base*1.1, base*0.7, base*0.6],
        "refactoring":[base*0.7, base*1.1, base*0.9, base*0.3, base*0.8, base*0.7, base*1.3, base*1.1],
        "adversarial":[base*0.9, base*0.7, base*1.0, base*1.4, base*0.8, base*1.2, base*0.6, base*0.8],
    }

    raw = profiles.get(challenge_type, profiles["general"])
    return [clamp(1, round(v), 10) for v in raw]

All migrated profiles are tagged confidence: migrated and prioritized for manual review.