| name | bad-case-analyzer |
| description | Analyzes model predictions to identify and categorize bad cases (FP/FN) with detailed
metrics. Generates interactive HTML reports for human review with filtering by error
type, predicted/actual class, and identified error patterns. Use when: (1) After model
evaluation, (2) Analyzing classification errors, (3) Conducting error analysis for
model improvement, (4) Reviewing prediction quality for NLP classification tasks.
|
| license | Apache 2.0 |
| allowed-tools | ["Read","Bash","Write","Glob"] |
Bad Case Analyzer
Overview
Analyzes model prediction errors in NLP classification tasks. Computes comprehensive metrics (accuracy, precision, recall, F1, confusion matrix), identifies error patterns, and generates interactive HTML reports for efficient human review with advanced filtering capabilities.
Workflow
Step 1: Load Predictions
Input Format (JSONL/JSON/CSV):
{"text": "sample", "true_label": "positive", "predicted_label": "negative", "confidence": 0.75}
Required fields: text/input, true_label/label, predicted_label/prediction
Step 2: Calculate Metrics
Compute for each class:
- Precision = TP / (TP + FP)
- Recall = TP / (TP + FN)
- F1 = 2 * (Precision * Recall) / (Precision + Recall)
- Support = number of true samples
Overall: Accuracy, Confusion Matrix
Step 3: Identify Bad Cases
Categorize errors:
- False Positives (FP): Predicted X but true label ≠ X
- False Negatives (FN): True label X but predicted ≠ X
Step 4: Analyze Patterns
Auto-detect:
- Top confusion pairs (most frequent misclassifications)
- Confidence distribution on errors
- High-confidence vs low-confidence errors
- Text length correlations
Step 5: Generate HTML Report
Create interactive report with:
- Metrics dashboard (sortable table)
- Confusion matrix (clickable cells)
- Error pattern summary
- Bad case browser with filters:
- Error type (FP/FN)
- True/Predicted label
- Confidence range
- Text search
- Pattern category
HTML Report Template
See Report Template for the complete interactive HTML template.
Key features:
- Real-time filtering
- Sortable columns
- Export capabilities
- Pattern-based grouping
Example Usage
User: "Analyze bad cases from predictions.jsonl"
Claude:
- Loads predictions
- Calculates metrics
- Identifies 142 errors (85 FP, 57 FN)
- Finds top confusion: positive→negative (34 cases)
- Generates
bad_case_report.html
Related Resources
