Tricky2 Benchmark Evaluating Human Error

Tricky^2: Taxonomy-Guided Mixed-Origin Bug Analysis

This skill enables Claude to systematically analyze code for mixed-origin bugs -- defects that arise from both human developers and LLM code generators coexisting in the same program. Based on the Tricky^2 framework, it applies a five-category error taxonomy (Input/Output, Variable/Data, Logic/Condition, Loop/Iteration, Function/Procedure) to classify, localize, and repair bugs while tracking whether each defect is human-originated, LLM-originated, or a compound interaction of both. This is critical in modern development where AI-assisted code (Copilot, ChatGPT, Claude) is interleaved with human-written logic, creating subtle error interactions that single-origin debugging misses.

When to Use

• When reviewing a pull request that mixes human-written code with AI-generated suggestions and you need to assess defect risk
• When a user asks "find bugs in this code" and the code was partially generated by an LLM (Copilot, ChatGPT, etc.)
• When debugging a program that works partially but has subtle logic errors that may stem from human-AI handoff points
• When the user wants to classify whether a bug was likely introduced by a human or an LLM based on its characteristics
• When auditing AI-assisted codebases for security vulnerabilities, since LLM bugs tend toward high-risk security patterns
• When a multi-bug program resists repair -- interacting errors from different origins often defeat single-fix strategies

Key Technique

The Tricky^2 approach rests on a taxonomy-guided error classification that organizes all bugs into five categories:

1. Input/Output -- incorrect reading, parsing, or formatting of I/O operations
2. Variable/Data -- wrong variable use, type mismatches, off-by-one in data structures
3. Logic/Condition -- flawed boolean expressions, missing edge cases, wrong comparison operators
4. Loop/Iteration -- incorrect bounds, infinite loops, off-by-one in iteration
5. Function/Procedure -- wrong function calls, incorrect parameter passing, missing return values

The key insight is that human bugs and LLM bugs have different signatures. Human bugs tend toward greater structural complexity -- deeply nested logic errors, incorrect algorithm choices, subtle edge-case misses. LLM bugs lean toward simpler but more dangerous patterns: unused variables, hallucinated API calls, security-vulnerable constructs, and data-misuse errors. When both types coexist in the same program, they interact -- an LLM's incorrect variable initialization can mask a human's off-by-one error, or a human's flawed condition can compound with an LLM's wrong loop bound to produce failures neither would cause alone.

The practical implication: mixed-origin code requires multi-pass analysis. Single-pass debugging finds individual bugs but misses interaction effects. The Tricky^2 workflow separates origin classification from localization from repair, because fixing an LLM bug in isolation can expose or worsen a latent human bug. Programs with both human and LLM errors have measurably lower repair success rates than programs with bugs from a single origin.

Step-by-Step Workflow

1. Identify code provenance boundaries. Determine which sections were human-written vs. AI-generated. Look for git blame annotations, inline comments like // generated by copilot, or ask the user. If provenance is unknown, proceed with origin-agnostic analysis but flag likely LLM patterns (see step 4).

2. Run the five-category taxonomy scan. Walk through the code and tag every suspicious construct against the taxonomy:

   • Input/Output: Check all input(), scanf, cin, file reads, API responses for missing validation or format mismatches
   • Variable/Data: Check variable initialization, scope, type conversions, and whether every declared variable is actually used
   • Logic/Condition: Check all if/else/ternary conditions for boundary correctness, negation errors, and short-circuit evaluation bugs
   • Loop/Iteration: Check all for/while/do-while for off-by-one, termination conditions, and iterator invalidation
   • Function/Procedure: Check all function calls for correct argument count/types, return value handling, and side effects

3. Classify each found bug by likely origin. Apply these heuristics:

   • Likely human: Complex algorithmic errors, subtle edge cases, domain-specific logic mistakes, inconsistent naming suggesting evolving understanding
   • Likely LLM: Unused variables/imports, hallucinated function names or API methods that don't exist, plausible-looking but semantically wrong library calls, security-vulnerable patterns (unsanitized input, hardcoded credentials), overly generic error handling

4. Map interaction points between bugs. For each pair of bugs, determine if they interact:

   • Independent: Fixing one doesn't affect the other (separate code paths)
   • Masking: One bug hides the symptoms of another (e.g., early return prevents reaching a second bug)
   • Compounding: Both bugs contribute to the same failure (e.g., wrong initialization + wrong loop bound)
   • Cascading: One bug's output feeds into another bug's input

5. Prioritize by interaction type. Repair compounding and cascading bugs together as a unit. Fix masking bugs from the outside in (fix the masker first to expose the masked bug). Independent bugs can be fixed in any order.

6. Generate minimal patches for each bug. For each defect, produce the smallest possible code change that fixes it. Do not refactor surrounding code. Preserve the original program structure.

7. Verify patches against interaction effects. After generating all patches, mentally (or actually) apply them together and check: Does fixing bug A change the manifestation of bug B? If so, revise the patch for B in the context of A's fix.

8. Report findings in a structured format. For each bug, output: location (file:line), taxonomy category, likely origin (human/LLM/unknown), interaction type with other bugs, and the proposed fix.

Concrete Examples

Example 1: Python function with interacting human + LLM bugs

User: "Review this function for bugs -- I wrote the algorithm but Copilot filled in some of the implementation."

def find_kth_largest(nums, k):
    import heapq
    heap = []
    for num in nums:
        heapq.heappush(heap, num)
        if len(heap) > k:
            heapq.heappop(heap)
    return heap[0] if heap else None

Approach:

1. Taxonomy scan identifies no obvious bugs in this version -- but the user reports wrong output for find_kth_largest([3,1,4,1,5], 0).
2. Variable/Data bug (likely human): k=0 causes len(heap) > 0 to always be true, so the heap is always emptied to size 0. The function returns None instead of raising an error for invalid k. Origin: human, because the algorithm design didn't account for k=0.
3. Check for LLM patterns: The import heapq inside the function is a minor style issue (LLM pattern -- Copilot often places imports locally). The if heap else None is a defensive pattern typical of LLMs that silently swallows errors.
4. Interaction: The LLM's silent None return (Function/Procedure category) masks the human's missing input validation (Input/Output category). Without the None fallback, the code would raise an IndexError, making the k=0 bug immediately visible.

Output:

Bug 1: [Line 7] Input/Output - No validation that k >= 1
  Origin: Human (algorithm design gap)
  Fix: Add `if k < 1 or k > len(nums): raise ValueError("k out of range")`

Bug 2: [Line 7] Function/Procedure - Silent None return masks errors
  Origin: LLM (defensive pattern hiding failures)
  Interaction: MASKING Bug 1 -- suppresses IndexError that would reveal invalid k
  Fix: Remove `if heap else None`, let IndexError propagate or raise explicitly

Repair order: Fix Bug 2 first (unmask), then fix Bug 1 (validate input).

Example 2: C++ with compounding loop and logic bugs

User: "This sorting function has issues. The swap logic was AI-generated."

void bubbleSort(int arr[], int n) {
    for (int i = 0; i < n; i++) {           // human
        for (int j = 0; j < n - 1; j++) {   // human
            if (arr[j] >= arr[j+1]) {        // AI-generated
                int temp = arr[j];           // AI-generated
                arr[j+1] = arr[j];           // AI-generated
                arr[j] = temp;               // AI-generated
            }
        }
    }
}

Approach:

1. Loop/Iteration (human): Outer loop runs n times but should run n-1 times. Minor inefficiency, not a correctness bug since the extra pass is redundant.
2. Logic/Condition (LLM): >= should be > for a stable sort. Using >= swaps equal elements unnecessarily, breaking stability.
3. Variable/Data (LLM): Line arr[j+1] = arr[j] overwrites arr[j+1] before saving it. Should be arr[j] = arr[j+1] (swap is backwards). The temp variable holds arr[j], then arr[j+1] is overwritten with arr[j], then arr[j] gets the original arr[j] from temp. The swap doesn't actually swap.
4. Interaction: Bugs 2 and 3 are compounding -- even if the comparison were correct, the broken swap means no elements ever move. The array is returned unchanged.

Output:

Bug 1: [Line 3] Loop/Iteration - Outer loop iterates n times, should be n-1
  Origin: Human | Severity: Low (redundant pass, not incorrect)
  Fix: Change `i < n` to `i < n - 1`

Bug 2: [Line 4] Logic/Condition - >= causes unstable sort
  Origin: LLM | Severity: Medium
  Fix: Change `>=` to `>`

Bug 3: [Line 6] Variable/Data - Swap is broken, arr[j+1] overwritten before use
  Origin: LLM | Severity: Critical
  Interaction: COMPOUNDING with Bug 2 -- both affect swap correctness
  Fix: Change `arr[j+1] = arr[j]` to `arr[j] = arr[j+1]`

Repair order: Fix Bugs 2 and 3 together (compounding pair), then Bug 1.

Example 3: Java with cascading errors across human-AI boundary

User: "Debug this -- the API call was written by ChatGPT, the rest is mine."

public List<User> getActiveUsers(Database db) {
    String query = "SELECT * FROM users WHERE status = 'active'";
    ResultSet rs = db.execute(query);           // human
    List<User> users = new ArrayList<>();
    while (rs.next()) {                         // human
        User u = new User();
        u.setName(rs.getString("username"));    // ChatGPT
        u.setAge(rs.getInt("age"));             // ChatGPT
        u.setEmail(rs.getString("mail"));       // ChatGPT
        users.add(u);
    }
    return users;
}

Approach:

1. Function/Procedure (LLM): rs.getString("username") -- likely column is name or user_name, not username. Hallucinated column name. Same for "mail" which is likely "email".
2. Input/Output (human): ResultSet is never closed -- resource leak. No try-with-resources or finally block.
3. Interaction: Bug 1 is cascading into Bug 2 -- when the hallucinated column name throws a SQLException, the unclosed ResultSet from Bug 2 means the database connection leaks on every failed call.

Output:

Bug 1: [Lines 7,9] Function/Procedure - Hallucinated column names
  Origin: LLM (classic hallucination of plausible-but-wrong identifiers)
  Fix: Verify actual column names against schema; likely "name"/"email"

Bug 2: [Line 3] Input/Output - ResultSet never closed (resource leak)
  Origin: Human (missing resource management)
  Interaction: CASCADING from Bug 1 -- SQLException from wrong columns
    causes early exit without closing ResultSet
  Fix: Wrap in try-with-resources or add finally block to close rs

Repair order: Fix Bug 2 first (resource safety), then Bug 1 (correct columns).

Best Practices

• Do: Always check for interaction effects between bugs before proposing fixes. A fix that works in isolation can break when applied alongside other fixes to the same program.
• Do: Classify bug origin even when uncertain -- mark as "likely human" or "likely LLM" with reasoning. This helps the user understand their codebase's risk profile.
• Do: Report bugs in the structured format (location, category, origin, interaction, fix) so findings are actionable and trackable.
• Do: Prioritize compounding and cascading bug pairs over independent bugs, since these cause the most confusing failures.
• Avoid: Fixing only the most obvious bug and stopping. Mixed-origin code statistically contains more interacting bugs than single-origin code.
• Avoid: Assuming LLM-generated code is correct because it "looks clean." LLM bugs tend to be syntactically perfect but semantically wrong -- hallucinated APIs, plausible-but-incorrect variable names, and silently wrong data transformations.

Error Handling

• Unknown provenance: If you cannot determine which code is human vs. AI-generated, analyze all bugs origin-agnostically but flag patterns characteristic of each origin. Report "origin: unknown" with reasoning for your best guess.
• No bugs found: If the taxonomy scan finds no issues, explicitly state which categories were checked and that no defects were identified. Do not invent bugs.
• Ambiguous interaction: If two bugs might or might not interact depending on input, classify the interaction as "conditional" and describe the triggering condition.
• Contradictory fixes: If fixing bug A requires a change that worsens bug B, flag this explicitly and propose an alternative that addresses both simultaneously.

Limitations

• This approach is most effective for C++, Python, and Java (the languages in the Tricky^2 corpus). The taxonomy applies to other languages but the origin-classification heuristics are less validated.
• Origin classification is probabilistic, not definitive. Without git blame or explicit provenance markers, the human/LLM distinction is a best guess based on error patterns.
• The five-category taxonomy covers common bug types but does not capture concurrency bugs, distributed systems errors, or build/configuration defects.
• Interaction analysis scales quadratically with bug count. For programs with more than ~10 bugs, prioritize the most severe defects rather than exhaustively mapping all interactions.
• The technique assumes bugs are localized to specific lines or small regions. Architectural-level defects (wrong design pattern, incorrect system decomposition) are outside scope.

Reference

Paper: Tricky^2: Towards a Benchmark for Evaluating Human and LLM Error Interactions (Granger et al., 2026). Look for: the five-category error taxonomy definition, the human vs. LLM bug characteristic comparison (Table showing structural complexity vs. security risk tradeoffs), and the key finding that mixed-origin programs have lower repair success rates than single-origin programs.