bad-case-analyzer

name	bad-case-analyzer
description	Analyzes model predictions to identify and categorize bad cases (FP/FN) with detailed metrics. Generates interactive HTML reports for human review with filtering by error type, predicted/actual class, and identified error patterns. Use when: (1) After model evaluation, (2) Analyzing classification errors, (3) Conducting error analysis for model improvement, (4) Reviewing prediction quality for NLP classification tasks.
license	Apache 2.0
allowed-tools	["Read","Bash","Write","Glob"]

Bad Case Analyzer

Overview

Analyzes model prediction errors in NLP classification tasks. Computes comprehensive metrics (accuracy, precision, recall, F1, confusion matrix), identifies error patterns, and generates interactive HTML reports for efficient human review with advanced filtering capabilities.

Workflow

Step 1: Load Predictions

Input Format (JSONL/JSON/CSV):

{"text": "sample", "true_label": "positive", "predicted_label": "negative", "confidence": 0.75}

Required fields: text/input, true_label/label, predicted_label/prediction

Step 2: Calculate Metrics

Compute for each class:

• Precision = TP / (TP + FP)
• Recall = TP / (TP + FN)
• F1 = 2 * (Precision * Recall) / (Precision + Recall)
• Support = number of true samples

Overall: Accuracy, Confusion Matrix

Step 3: Identify Bad Cases

Categorize errors:

• False Positives (FP): Predicted X but true label ≠ X
• False Negatives (FN): True label X but predicted ≠ X

Step 4: Analyze Patterns

Auto-detect:

• Top confusion pairs (most frequent misclassifications)
• Confidence distribution on errors
• High-confidence vs low-confidence errors
• Text length correlations

Step 5: Generate HTML Report

Create interactive report with:

• Metrics dashboard (sortable table)
• Confusion matrix (clickable cells)
• Error pattern summary
• Bad case browser with filters:
  • Error type (FP/FN)
  • True/Predicted label
  • Confidence range
  • Text search
  • Pattern category

HTML Report Template

See Report Template for the complete interactive HTML template.

Key features:

• Real-time filtering
• Sortable columns
• Export capabilities
• Pattern-based grouping

Example Usage

User: "Analyze bad cases from predictions.jsonl"

Claude:

1. Loads predictions
2. Calculates metrics
3. Identifies 142 errors (85 FP, 57 FN)
4. Finds top confusion: positive→negative (34 cases)
5. Generates bad_case_report.html

Related Resources

• Metrics Guide - Classification metrics explained
• Error Patterns - Common patterns and solutions