label-quality-audit

name	label-quality-audit
description	Audit label quality using confident learning (Northcutt et al.), cross-validation noise detection, and per-class error analysis. Identifies mislabeled examples for review.
tags	["label-quality","confident-learning","noise-detection","data-cleaning","dataset-curation","ml-ops"]

Label Quality Audit

Overview

Label noise is the most insidious data quality problem — it's invisible until the model learns the wrong thing. Confident learning (Northcutt et al., 2021) identifies likely mislabeled examples using out-of-sample predicted probabilities.

When to Use

Use when: training data labels come from crowd workers, automated systems, or weak supervision. Do not use on expert-validated reference data unless auditing for drift.

Confident Learning Pipeline

import numpy as np
from sklearn.model_selection import cross_val_predict
from sklearn.ensemble import RandomForestClassifier

def confident_learning_audit(X, y, n_folds=5):
    """
    Returns indices of likely mislabeled examples.
    Based on Northcutt et al. "Confident Learning: Estimating
    Uncertainty in Dataset Labels" (JMLR 2021).
    """
    n_classes = len(np.unique(y))
    
    # 1. Out-of-sample predicted probabilities
    proba = cross_val_predict(
        RandomForestClassifier(n_estimators=, random_state=),
        X, y, cv=n_folds, method=
    )
    
    
    
    confident_joint = np.zeros((n_classes, n_classes))
     i  ((y)):
        true_class = y[i]
        pred_class = np.argmax(proba[i])
        confidence = proba[i][pred_class]
        
        
        class_threshold = np.percentile(proba[:, pred_class], )
         confidence > class_threshold:
            confident_joint[true_class][pred_class] += 
    
    
    issues = []
    per_class_thresholds = {
        k: np.percentile(proba[:, k], )  k  (n_classes)
    }
    
     i  ((y)):
        pred_class = np.argmax(proba[i])
         (pred_class != y[i]  
            proba[i][pred_class] > per_class_thresholds[pred_class]):
            issues.append(i)
    
    
    noise_rates = {}
     k  (n_classes):
        n_in_class = np.(y == k)
        n_noisy = np.((np.array(issues) != y[np.array(issues)]) & 
                         (y[np.array(issues) == k]))
        noise_rates[k] = n_noisy / n_in_class  n_in_class >   
    
     {
        : issues,
        : (issues),
        : (issues) / (y),
        : noise_rates,
        : confident_joint,
    }