tricky2-benchmark-evaluating-human-error

name

tricky2-benchmark-evaluating-human-error

description

Taxonomy-guided analysis of mixed human+LLM bugs in code. Classifies bug origins, localizes interacting defects, and repairs hybrid-origin errors. Use when: 'review this AI-generated code for bugs', 'find bugs in this human+AI codebase', 'classify whether this bug is human or LLM', 'audit code written by both humans and copilot', 'debug interacting errors in mixed-origin code', 'analyze bug patterns in AI-assisted development'.

Tricky^2: Taxonomy-Guided Mixed-Origin Bug Analysis

This skill enables Claude to systematically analyze code for mixed-origin bugs -- defects that arise from both human developers and LLM code generators coexisting in the same program. Based on the Tricky^2 framework, it applies a five-category error taxonomy (Input/Output, Variable/Data, Logic/Condition, Loop/Iteration, Function/Procedure) to classify, localize, and repair bugs while tracking whether each defect is human-originated, LLM-originated, or a compound interaction of both. This is critical in modern development where AI-assisted code (Copilot, ChatGPT, Claude) is interleaved with human-written logic, creating subtle error interactions that single-origin debugging misses.

When to Use

• When reviewing a pull request that mixes human-written code with AI-generated suggestions and you need to assess defect risk
• When a user asks "find bugs in this code" and the code was partially generated by an LLM (Copilot, ChatGPT, etc.)
• When debugging a program that works partially but has subtle logic errors that may stem from human-AI handoff points
• When the user wants to classify whether a bug was likely introduced by a human or an LLM based on its characteristics
• When auditing AI-assisted codebases for security vulnerabilities, since LLM bugs tend toward high-risk security patterns
• When a multi-bug program resists repair -- interacting errors from different origins often defeat single-fix strategies

Key Technique

The Tricky^2 approach rests on a taxonomy-guided error classification that organizes all bugs into five categories:

1. Input/Output -- incorrect reading, parsing, or formatting of I/O operations
2. Variable/Data -- wrong variable use, type mismatches, off-by-one in data structures
3. Logic/Condition -- flawed boolean expressions, missing edge cases, wrong comparison operators
4. Loop/Iteration -- incorrect bounds, infinite loops, off-by-one in iteration
5. Function/Procedure -- wrong function calls, incorrect parameter passing, missing return values

The key insight is that human bugs and LLM bugs have different signatures. Human bugs tend toward greater structural complexity -- deeply nested logic errors, incorrect algorithm choices, subtle edge-case misses. LLM bugs lean toward simpler but more dangerous patterns: unused variables, hallucinated API calls, security-vulnerable constructs, and data-misuse errors. When both types coexist in the same program, they interact -- an LLM's incorrect variable initialization can mask a human's off-by-one error, or a human's flawed condition can compound with an LLM's wrong loop bound to produce failures neither would cause alone.

The practical implication: mixed-origin code requires multi-pass analysis. Single-pass debugging finds individual bugs but misses interaction effects. The Tricky^2 workflow separates origin classification from localization from repair, because fixing an LLM bug in isolation can expose or worsen a latent human bug. Programs with both human and LLM errors have measurably lower repair success rates than programs with bugs from a single origin.

Step-by-Step Workflow

1. Identify code provenance boundaries. Determine which sections were human-written vs. AI-generated. Look for git blame annotations, inline comments like // generated by copilot, or ask the user. If provenance is unknown, proceed with origin-agnostic analysis but flag likely LLM patterns (see step 4).

2. Run the five-category taxonomy scan. Walk through the code and tag every suspicious construct against the taxonomy:

   • Input/Output: Check all input(), scanf, cin, file reads, API responses for missing validation or format mismatches
   • Variable/Data: Check variable initialization, scope, type conversions, and whether every declared variable is actually used
   • Logic/Condition: Check all if/else/ternary conditions for boundary correctness, negation errors, and short-circuit evaluation bugs
   • Loop/Iteration: Check all for/while/do-while for off-by-one, termination conditions, and iterator invalidation
   • Function/Procedure: Check all function calls for correct argument count/types, return value handling, and side effects

3. Classify each found bug by likely origin. Apply these heuristics:

   • Likely human: Complex algorithmic errors, subtle edge cases, domain-specific logic mistakes, inconsistent naming suggesting evolving understanding
   • Likely LLM: Unused variables/imports, hallucinated function names or API methods that don't exist, plausible-looking but semantically wrong library calls, security-vulnerable patterns (unsanitized input, hardcoded credentials), overly generic error handling

4. Map interaction points between bugs. For each pair of bugs, determine if they interact:

   • Independent: Fixing one doesn't affect the other (separate code paths)
   • Masking: One bug hides the symptoms of another (e.g., early return prevents reaching a second bug)
   • Compounding: Both bugs contribute to the same failure (e.g., wrong initialization + wrong loop bound)
   • Cascading: One bug's output feeds into another bug's input

5. Prioritize by interaction type. Repair compounding and cascading bugs together as a unit. Fix masking bugs from the outside in (fix the masker first to expose the masked bug). Independent bugs can be fixed in any order.

Concrete Examples

Example 1: Python function with interacting human + LLM bugs

User: "Review this function for bugs -- I wrote the algorithm but Copilot filled in some of the implementation."

def find_kth_largest(nums, k):
    import heapq
    heap = []
    for num in nums:
        heapq.heappush(heap, num)
        if len(heap) > k:
            heapq.heappop(heap)
    return heap[0] if heap else None

Approach:

1. Taxonomy scan identifies no obvious bugs in this version -- but the user reports wrong output for find_kth_largest([3,1,4,1,5], 0).
2. Variable/Data bug (likely human): k=0 causes len(heap) > 0 to always be true, so the heap is always emptied to size 0. The function returns None instead of raising an error for invalid k. Origin: human, because the algorithm design didn't account for k=0.
3. Check for LLM patterns: The import heapq inside the function is a minor style issue (LLM pattern -- Copilot often places imports locally). The if heap else None is a defensive pattern typical of LLMs that silently swallows errors.
4. Interaction: The LLM's silent None return (Function/Procedure category) masks the human's missing input validation (Input/Output category). Without the None fallback, the code would raise an IndexError, making the k=0 bug immediately visible.

Output:

Bug 1: [Line 7] Input/Output - No validation that k >= 1
  Origin: Human (algorithm design gap)
  Fix: Add `if k < 1 or k > len(nums): raise ValueError("k out of range")`

Bug 2: [Line 7] Function/Procedure - Silent None return masks errors
  Origin: LLM (defensive pattern hiding failures)
  Interaction: MASKING Bug 1 -- suppresses IndexError that would reveal invalid k
  Fix: Remove `if heap else None`, let IndexError propagate or raise explicitly

Repair order: Fix Bug 2 first (unmask), then fix Bug 1 (validate input).

Example 2: C++ with compounding loop and logic bugs

User: "This sorting function has issues. The swap logic was AI-generated."

void bubbleSort(int arr[], int n) {
    for (int i = 0; i < n; i++) {           // human
        for (int j = 0; j < n - 1; j++) {   // human
            if (arr[j] >= arr[j+1]) {        // AI-generated
                int temp = arr[j];           // AI-generated
                arr[j+1] = arr[j];           // AI-generated
                arr[j] = temp;               // AI-generated
            }
        }
    }
}

Approach:

1. Loop/Iteration (human): Outer loop runs n times but should run n-1 times. Minor inefficiency, not a correctness bug since the extra pass is redundant.
2. Logic/Condition (LLM): >= should be > for a stable sort. Using >= swaps equal elements unnecessarily, breaking stability.
3. Variable/Data (LLM): Line arr[j+1] = arr[j] overwrites arr[j+1] before saving it. Should be arr[j] = arr[j+1] (swap is backwards). The temp variable holds arr[j], then arr[j+1] is overwritten with arr[j], then arr[j] gets the original arr[j] from temp. The swap doesn't actually swap.
4. Interaction: Bugs 2 and 3 are compounding -- even if the comparison were correct, the broken swap means no elements ever move. The array is returned unchanged.

Output:

Bug 1: [Line 3] Loop/Iteration - Outer loop iterates n times, should be n-1
  Origin: Human | Severity: Low (redundant pass, not incorrect)
  Fix: Change `i < n` to `i < n - 1`

Bug 2: [Line 4] Logic/Condition - >= causes unstable sort
  Origin: LLM | Severity: Medium
  Fix: Change `>=` to `>`

Bug 3: [Line 6] Variable/Data - Swap is broken, arr[j+1] overwritten before use
  Origin: LLM | Severity: Critical
  Interaction: COMPOUNDING with Bug 2 -- both affect swap correctness
  Fix: Change `arr[j+1] = arr[j]` to `arr[j] = arr[j+1]`

Repair order: Fix Bugs 2 and 3 together (compounding pair), then Bug 1.

Example 3: Java with cascading errors across human-AI boundary

User: "Debug this -- the API call was written by ChatGPT, the rest is mine."

public List<User> getActiveUsers(Database db) {
    String query = "SELECT * FROM users WHERE status = 'active'";
    ResultSet rs = db.execute(query);           // human
    List<User> users = new ArrayList<>();
    while (rs.next()) {                         // human
        User u = new User();
        u.setName(rs.getString("username"));    // ChatGPT
        u.setAge(rs.getInt("age"));             // ChatGPT
        u.setEmail(rs.getString("mail"));       // ChatGPT
        users.add(u);
    }
    return users;
}

Approach:

1. Function/Procedure (LLM): rs.getString("username") -- likely column is name or user_name, not username. Hallucinated column name. Same for "mail" which is likely "email".
2. Input/Output (human): ResultSet is never closed -- resource leak. No try-with-resources or finally block.
3. Interaction: Bug 1 is cascading into Bug 2 -- when the hallucinated column name throws a SQLException, the unclosed ResultSet from Bug 2 means the database connection leaks on every failed call.

Output:

Bug 1: [Lines 7,9] Function/Procedure - Hallucinated column names
  Origin: LLM (classic hallucination of plausible-but-wrong identifiers)
  Fix: Verify actual column names against schema; likely "name"/"email"

Bug 2: [Line 3] Input/Output - ResultSet never closed (resource leak)
  Origin: Human (missing resource management)
  Interaction: CASCADING from Bug 1 -- SQLException from wrong columns
    causes early exit without closing ResultSet
  Fix: Wrap in try-with-resources or add finally block to close rs

Repair order: Fix Bug 2 first (resource safety), then Bug 1 (correct columns).

Best Practices

• Do: Always check for interaction effects between bugs before proposing fixes. A fix that works in isolation can break when applied alongside other fixes to the same program.
• Do: Classify bug origin even when uncertain -- mark as "likely human" or "likely LLM" with reasoning. This helps the user understand their codebase's risk profile.
• Do: Report bugs in the structured format (location, category, origin, interaction, fix) so findings are actionable and trackable.
• Do: Prioritize compounding and cascading bug pairs over independent bugs, since these cause the most confusing failures.
• Avoid: Fixing only the most obvious bug and stopping. Mixed-origin code statistically contains more interacting bugs than single-origin code.
• Avoid: Assuming LLM-generated code is correct because it "looks clean." LLM bugs tend to be syntactically perfect but semantically wrong -- hallucinated APIs, plausible-but-incorrect variable names, and silently wrong data transformations.

Error Handling

• Unknown provenance: If you cannot determine which code is human vs. AI-generated, analyze all bugs origin-agnostically but flag patterns characteristic of each origin. Report "origin: unknown" with reasoning for your best guess.
• No bugs found: If the taxonomy scan finds no issues, explicitly state which categories were checked and that no defects were identified. Do not invent bugs.
• Ambiguous interaction: If two bugs might or might not interact depending on input, classify the interaction as "conditional" and describe the triggering condition.
• Contradictory fixes: If fixing bug A requires a change that worsens bug B, flag this explicitly and propose an alternative that addresses both simultaneously.

Limitations

• This approach is most effective for C++, Python, and Java (the languages in the Tricky^2 corpus). The taxonomy applies to other languages but the origin-classification heuristics are less validated.
• Origin classification is probabilistic, not definitive. Without git blame or explicit provenance markers, the human/LLM distinction is a best guess based on error patterns.
• The five-category taxonomy covers common bug types but does not capture concurrency bugs, distributed systems errors, or build/configuration defects.
• Interaction analysis scales quadratically with bug count. For programs with more than ~10 bugs, prioritize the most severe defects rather than exhaustively mapping all interactions.
• The technique assumes bugs are localized to specific lines or small regions. Architectural-level defects (wrong design pattern, incorrect system decomposition) are outside scope.

Reference

Paper: Tricky^2: Towards a Benchmark for Evaluating Human and LLM Error Interactions (Granger et al., 2026). Look for: the five-category error taxonomy definition, the human vs. LLM bug characteristic comparison (Table showing structural complexity vs. security risk tradeoffs), and the key finding that mixed-origin programs have lower repair success rates than single-origin programs.