# Tricky2 Benchmark Evaluating Human Error > Taxonomy-guided analysis of mixed human+LLM bugs in code. Classifies bug origins, localizes interacting defects, and repairs hybrid-origin errors. Use when: 'review this AI-generated code for bugs', 'find bugs in this human+AI codebase', 'classify whether this bug is human or LLM', 'audit code written by both humans and copilot', 'debug interacting errors in mixed-origin code', 'analyze bug patterns in AI-assisted development'. - Skill: `ndpvt-web/tricky2-benchmark-evaluating-human-error` (Agent Skill) - Install (CLI): `npx skillmds@latest add ndpvt-web/tricky2-benchmark-evaluating-human-error` - Raw SKILL.md: https://api.skillmd.com/api/skills/ndpvt-web/tricky2-benchmark-evaluating-human-error/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: AI & ML - Author: ndpvt-web (https://skillmd.com/u/ndpvt-web) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/ndpvt-web/tricky2-benchmark-evaluating-human-error --- # Tricky^2: Taxonomy-Guided Mixed-Origin Bug Analysis This skill enables Claude to systematically analyze code for **mixed-origin bugs** -- defects that arise from both human developers and LLM code generators coexisting in the same program. Based on the Tricky^2 framework, it applies a five-category error taxonomy (Input/Output, Variable/Data, Logic/Condition, Loop/Iteration, Function/Procedure) to classify, localize, and repair bugs while tracking whether each defect is human-originated, LLM-originated, or a compound interaction of both. This is critical in modern development where AI-assisted code (Copilot, ChatGPT, Claude) is interleaved with human-written logic, creating subtle error interactions that single-origin debugging misses. ## When to Use - When reviewing a pull request that mixes human-written code with AI-generated suggestions and you need to assess defect risk - When a user asks "find bugs in this code" and the code was partially generated by an LLM (Copilot, ChatGPT, etc.) - When debugging a program that works partially but has subtle logic errors that may stem from human-AI handoff points - When the user wants to classify whether a bug was likely introduced by a human or an LLM based on its characteristics - When auditing AI-assisted codebases for security vulnerabilities, since LLM bugs tend toward high-risk security patterns - When a multi-bug program resists repair -- interacting errors from different origins often defeat single-fix strategies ## Key Technique The Tricky^2 approach rests on a **taxonomy-guided error classification** that organizes all bugs into five categories: 1. **Input/Output** -- incorrect reading, parsing, or formatting of I/O operations 2. **Variable/Data** -- wrong variable use, type mismatches, off-by-one in data structures 3. **Logic/Condition** -- flawed boolean expressions, missing edge cases, wrong comparison operators 4. **Loop/Iteration** -- incorrect bounds, infinite loops, off-by-one in iteration 5. **Function/Procedure** -- wrong function calls, incorrect parameter passing, missing return values The key insight is that **human bugs and LLM bugs have different signatures**. Human bugs tend toward greater structural complexity -- deeply nested logic errors, incorrect algorithm choices, subtle edge-case misses. LLM bugs lean toward simpler but more dangerous patterns: unused variables, hallucinated API calls, security-vulnerable constructs, and data-misuse errors. When both types coexist in the same program, they **interact** -- an LLM's incorrect variable initialization can mask a human's off-by-one error, or a human's flawed condition can compound with an LLM's wrong loop bound to produce failures neither would cause alone. The practical implication: **mixed-origin code requires multi-pass analysis**. Single-pass debugging finds individual bugs but misses interaction effects. The Tricky^2 workflow separates origin classification from localization from repair, because fixing an LLM bug in isolation can expose or worsen a latent human bug. Programs with both human and LLM errors have measurably lower repair success rates than programs with bugs from a single origin. ## Step-by-Step Workflow 1. **Identify code provenance boundaries.** Determine which sections were human-written vs. AI-generated. Look for git blame annotations, inline comments like `// generated by copilot`, or ask the user. If provenance is unknown, proceed with origin-agnostic analysis but flag likely LLM patterns (see step 4). 2. **Run the five-category taxonomy scan.** Walk through the code and tag every suspicious construct against the taxonomy: - **Input/Output**: Check all `input()`, `scanf`, `cin`, file reads, API responses for missing validation or format mismatches - **Variable/Data**: Check variable initialization, scope, type conversions, and whether every declared variable is actually used - **Logic/Condition**: Check all `if`/`else`/ternary conditions for boundary correctness, negation errors, and short-circuit evaluation bugs - **Loop/Iteration**: Check all `for`/`while`/`do-while` for off-by-one, termination conditions, and iterator invalidation - **Function/Procedure**: Check all function calls for correct argument count/types, return value handling, and side effects 3. **Classify each found bug by likely origin.** Apply these heuristics: - **Likely human**: Complex algorithmic errors, subtle edge cases, domain-specific logic mistakes, inconsistent naming suggesting evolving understanding - **Likely LLM**: Unused variables/imports, hallucinated function names or API methods that don't exist, plausible-looking but semantically wrong library calls, security-vulnerable patterns (unsanitized input, hardcoded credentials), overly generic error handling 4. **Map interaction points between bugs.** For each pair of bugs, determine if they interact: - **Independent**: Fixing one doesn't affect the other (separate code paths) - **Masking**: One bug hides the symptoms of another (e.g., early return prevents reaching a second bug) - **Compounding**: Both bugs contribute to the same failure (e.g., wrong initialization + wrong loop bound) - **Cascading**: One bug's output feeds into another bug's input 5. **Prioritize by interaction type.** Repair compounding and cascading bugs together as a unit. Fix masking bugs from the outside in (fix the masker first to expose the masked bug). Independent bugs can be fixed in any order. 6. **Generate minimal patches for each bug.** For each defect, produce the smallest possible code change that fixes it. Do not refactor surrounding code. Preserve the original program structure. 7. **Verify patches against interaction effects.** After generating all patches, mentally (or actually) apply them together and check: Does fixing bug A change the manifestation of bug B? If so, revise the patch for B in the context of A's fix. 8. **Report findings in a structured format.** For each bug, output: location (file:line), taxonomy category, likely origin (human/LLM/unknown), interaction type with other bugs, and the proposed fix. ## Concrete Examples **Example 1: Python function with interacting human + LLM bugs** User: "Review this function for bugs -- I wrote the algorithm but Copilot filled in some of the implementation." ```python def find_kth_largest(nums, k): import heapq heap = [] for num in nums: heapq.heappush(heap, num) if len(heap) > k: heapq.heappop(heap) return heap[0] if heap else None ``` Approach: 1. Taxonomy scan identifies no obvious bugs in this version -- but the user reports wrong output for `find_kth_largest([3,1,4,1,5], 0)`. 2. **Variable/Data bug** (likely human): `k=0` causes `len(heap) > 0` to always be true, so the heap is always emptied to size 0. The function returns `None` instead of raising an error for invalid k. Origin: human, because the algorithm design didn't account for k=0. 3. Check for LLM patterns: The `import heapq` inside the function is a minor style issue (LLM pattern -- Copilot often places imports locally). The `if heap else None` is a defensive pattern typical of LLMs that silently swallows errors. 4. **Interaction**: The LLM's silent `None` return (Function/Procedure category) **masks** the human's missing input validation (Input/Output category). Without the `None` fallback, the code would raise an `IndexError`, making the k=0 bug immediately visible. Output: ``` Bug 1: [Line 7] Input/Output - No validation that k >= 1 Origin: Human (algorithm design gap) Fix: Add `if k < 1 or k > len(nums): raise ValueError("k out of range")` Bug 2: [Line 7] Function/Procedure - Silent None return masks errors Origin: LLM (defensive pattern hiding failures) Interaction: MASKING Bug 1 -- suppresses IndexError that would reveal invalid k Fix: Remove `if heap else None`, let IndexError propagate or raise explicitly Repair order: Fix Bug 2 first (unmask), then fix Bug 1 (validate input). ``` **Example 2: C++ with compounding loop and logic bugs** User: "This sorting function has issues. The swap logic was AI-generated." ```cpp void bubbleSort(int arr[], int n) { for (int i = 0; i < n; i++) { // human for (int j = 0; j < n - 1; j++) { // human if (arr[j] >= arr[j+1]) { // AI-generated int temp = arr[j]; // AI-generated arr[j+1] = arr[j]; // AI-generated arr[j] = temp; // AI-generated } } } } ``` Approach: 1. **Loop/Iteration** (human): Outer loop runs `n` times but should run `n-1` times. Minor inefficiency, not a correctness bug since the extra pass is redundant. 2. **Logic/Condition** (LLM): `>=` should be `>` for a stable sort. Using `>=` swaps equal elements unnecessarily, breaking stability. 3. **Variable/Data** (LLM): Line `arr[j+1] = arr[j]` overwrites `arr[j+1]` before saving it. Should be `arr[j] = arr[j+1]` (swap is backwards). The temp variable holds `arr[j]`, then `arr[j+1]` is overwritten with `arr[j]`, then `arr[j]` gets the original `arr[j]` from temp. **The swap doesn't actually swap.** 4. **Interaction**: Bugs 2 and 3 are **compounding** -- even if the comparison were correct, the broken swap means no elements ever move. The array is returned unchanged. Output: ``` Bug 1: [Line 3] Loop/Iteration - Outer loop iterates n times, should be n-1 Origin: Human | Severity: Low (redundant pass, not incorrect) Fix: Change `i < n` to `i < n - 1` Bug 2: [Line 4] Logic/Condition - >= causes unstable sort Origin: LLM | Severity: Medium Fix: Change `>=` to `>` Bug 3: [Line 6] Variable/Data - Swap is broken, arr[j+1] overwritten before use Origin: LLM | Severity: Critical Interaction: COMPOUNDING with Bug 2 -- both affect swap correctness Fix: Change `arr[j+1] = arr[j]` to `arr[j] = arr[j+1]` Repair order: Fix Bugs 2 and 3 together (compounding pair), then Bug 1. ``` **Example 3: Java with cascading errors across human-AI boundary** User: "Debug this -- the API call was written by ChatGPT, the rest is mine." ```java public List getActiveUsers(Database db) { String query = "SELECT * FROM users WHERE status = 'active'"; ResultSet rs = db.execute(query); // human List users = new ArrayList<>(); while (rs.next()) { // human User u = new User(); u.setName(rs.getString("username")); // ChatGPT u.setAge(rs.getInt("age")); // ChatGPT u.setEmail(rs.getString("mail")); // ChatGPT users.add(u); } return users; } ``` Approach: 1. **Function/Procedure** (LLM): `rs.getString("username")` -- likely column is `name` or `user_name`, not `username`. Hallucinated column name. Same for `"mail"` which is likely `"email"`. 2. **Input/Output** (human): `ResultSet` is never closed -- resource leak. No try-with-resources or finally block. 3. **Interaction**: Bug 1 is **cascading** into Bug 2 -- when the hallucinated column name throws a `SQLException`, the unclosed `ResultSet` from Bug 2 means the database connection leaks on every failed call. Output: ``` Bug 1: [Lines 7,9] Function/Procedure - Hallucinated column names Origin: LLM (classic hallucination of plausible-but-wrong identifiers) Fix: Verify actual column names against schema; likely "name"/"email" Bug 2: [Line 3] Input/Output - ResultSet never closed (resource leak) Origin: Human (missing resource management) Interaction: CASCADING from Bug 1 -- SQLException from wrong columns causes early exit without closing ResultSet Fix: Wrap in try-with-resources or add finally block to close rs Repair order: Fix Bug 2 first (resource safety), then Bug 1 (correct columns). ``` ## Best Practices - **Do:** Always check for interaction effects between bugs before proposing fixes. A fix that works in isolation can break when applied alongside other fixes to the same program. - **Do:** Classify bug origin even when uncertain -- mark as "likely human" or "likely LLM" with reasoning. This helps the user understand their codebase's risk profile. - **Do:** Report bugs in the structured format (location, category, origin, interaction, fix) so findings are actionable and trackable. - **Do:** Prioritize compounding and cascading bug pairs over independent bugs, since these cause the most confusing failures. - **Avoid:** Fixing only the most obvious bug and stopping. Mixed-origin code statistically contains more interacting bugs than single-origin code. - **Avoid:** Assuming LLM-generated code is correct because it "looks clean." LLM bugs tend to be syntactically perfect but semantically wrong -- hallucinated APIs, plausible-but-incorrect variable names, and silently wrong data transformations. ## Error Handling - **Unknown provenance**: If you cannot determine which code is human vs. AI-generated, analyze all bugs origin-agnostically but flag patterns characteristic of each origin. Report "origin: unknown" with reasoning for your best guess. - **No bugs found**: If the taxonomy scan finds no issues, explicitly state which categories were checked and that no defects were identified. Do not invent bugs. - **Ambiguous interaction**: If two bugs might or might not interact depending on input, classify the interaction as "conditional" and describe the triggering condition. - **Contradictory fixes**: If fixing bug A requires a change that worsens bug B, flag this explicitly and propose an alternative that addresses both simultaneously. ## Limitations - This approach is most effective for **C++, Python, and Java** (the languages in the Tricky^2 corpus). The taxonomy applies to other languages but the origin-classification heuristics are less validated. - Origin classification is **probabilistic, not definitive**. Without git blame or explicit provenance markers, the human/LLM distinction is a best guess based on error patterns. - The five-category taxonomy covers common bug types but does not capture **concurrency bugs**, **distributed systems errors**, or **build/configuration defects**. - Interaction analysis scales quadratically with bug count. For programs with more than ~10 bugs, prioritize the most severe defects rather than exhaustively mapping all interactions. - The technique assumes bugs are **localized to specific lines or small regions**. Architectural-level defects (wrong design pattern, incorrect system decomposition) are outside scope. ## Reference **Paper**: [Tricky^2: Towards a Benchmark for Evaluating Human and LLM Error Interactions](https://arxiv.org/abs/2601.18949v1) (Granger et al., 2026). Look for: the five-category error taxonomy definition, the human vs. LLM bug characteristic comparison (Table showing structural complexity vs. security risk tradeoffs), and the key finding that mixed-origin programs have lower repair success rates than single-origin programs.