config-knowledge-distillation

name	config-knowledge-distillation
title	ConfiG: Confidence-Guided Data Augmentation for Knowledge Distillation Under Covariate Shift
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2506.02294
keywords	["knowledge-distillation","data-augmentation","covariate-shift","robustness"]
description	Improve student model robustness under covariate shift by using diffusion-based augmentation that targets spurious features via teacher-student disagreement.

ConfiG: Confidence-Guided Data Augmentation for Knowledge Distillation

Core Concept

When training data contains spurious features absent at test time, knowledge distillation can preserve these biases in student models. ConfiG addresses this by generating augmented images that maximize disagreement between teacher and student, targeting exactly the spurious correlations the student learned. This diffusion-based approach enables students to overcome dataset biases while maintaining knowledge transfer.

Architecture Overview

• Problem: Knowledge distillation transfers spurious correlations from biased datasets, degrading generalization to unseen groups
• Solution: Confidence-guided diffusion generates adversarial examples leveraging teacher-student disagreement
• Mechanism: Optimize latent variables to maximize teacher confidence while minimizing student confidence, targeting spurious features
• Theoretical Grounding: Proposition 1 proves confidence-guided augmentation reduces distributional generalization gap
• Synergy: Works with model-centric bias mitigation (TAB, etc.), showing data and model approaches are complementary

Implementation

Step 1: Understand Covariate Shift and Generalization Decomposition

import torch
import numpy as np
from typing import Tuple, List

class CovariateShiftAnalyzer:
    """Analyze how spurious features affect generalization"""

    def decompose_error(self, teacher_model, student_model,
                       train_data, test_data,
                       group_labels_test) -> Dict:
        """
        Decompose generalization error into two components:
        1. Teacher quality: how well teacher generalizes
        2. Distributional gap: how much student differs from teacher distribution

        Key insight: ConfiG targets reducing the gap, not improving teacher quality.
        """

        # Evaluate on training distribution (in-distribution)
        train_acc = self.evaluate_accuracy(student_model, train_data)

        # Evaluate on test distribution
        test_acc = self.evaluate_accuracy(student_model, test_data)

        # Evaluate per test group
        group_accs = {}
        for group_id in np.unique(group_labels_test):
            group_mask = group_labels_test == group_id
            group_test = test_data[group_mask]
            group_accs[group_id] = self.evaluate_accuracy(student_model, group_test)

        # Decompose error
        overall_gap = train_acc - test_acc
        group_gaps = {g: train_acc - acc for g, acc in group_accs.items()}

        print("=== Generalization Analysis ===")
        print(f"Overall train accuracy: {train_acc:.1%}")
        print(f"Overall test accuracy: {test_acc:%}")
        ()
        ()

         group_id, acc  group_accs.items():
            ()

         {
            : overall_gap,
            : group_gaps,
            : group_accs,
        }

     () -> []:
        

        spurious_correlations = []

        
        unique_groups = np.unique(group_labels)

         group_id  unique_groups:
            group_mask = group_labels == group_id

            
            group_data = train_data[group_mask]
            other_data = train_data[~group_mask]

            
            
            feature_divergence = ._compute_feature_divergence(
                group_data, other_data
            )

            
            top_divergent = (
                (feature_divergence),
                key= x: x[],
                reverse=
            )[:]

             feature_idx, divergence  top_divergent:
                spurious_correlations.append({
                    : group_id,
                    : feature_idx,
                    : divergence,
                })

         spurious_correlations

     () -> np.ndarray:
        

        
        divergences = []

         feature_idx  (group_data.shape[]):
            
            hist_group = np.histogram(group_data[:, feature_idx], bins=)[]
            hist_other = np.histogram(other_data[:, feature_idx], bins=)[]

            
            hist_group = hist_group / (np.(hist_group) + )
            hist_other = hist_other / (np.(hist_other) + )

            
            kl = np.(hist_group * np.log((hist_group + ) / (hist_other + )))
            divergences.append(kl)

         np.array(divergences)

Step 2: Implement Confidence-Guided Augmentation

import torch.nn.functional as F

class ConfidenceGuidedDiffusion:
    """Generate augmented images targeting spurious features"""

    def __init__(self, diffusion_model, teacher_model, student_model):
        self.diffusion = diffusion_model  # Pretrained diffusion (e.g., Stable Diffusion)
        self.teacher = teacher_model
        self.student = student_model

    def generate_augmented_sample(self, image: torch.Tensor,
                                 label: int,
                                 gamma: float = 2.0,
                                 num_iterations: int = 100) -> torch.Tensor:
        """
        Generate augmented image by optimizing latent vector z.

        Objective: maximize loss(z) = t(z)^γ + (1-f(z))^γ
        where:
          t(z) = teacher confidence on augmented sample
          f(z) = student confidence on augmented sample
          γ = 2.0 (empirically optimal)

        This targets spurious features the student learned.
        """

        # Initialize random latent in diffusion space
        z = torch.randn(1, 4, image.shape[1]//8, image.shape[2]//8)
        z.requires_grad = True

        # Optimizer for latent variables
        optimizer = torch.optim.Adam([z], lr=0.01)

        for iteration in range(num_iterations):
            # Decode latent to image
            augmented_image = self.diffusion.decode(z)

            
             torch.no_grad():
                teacher_logits = .teacher(augmented_image)
                student_logits = .student(augmented_image)

            teacher_probs = F.softmax(teacher_logits, dim=-)
            student_probs = F.softmax(student_logits, dim=-)

            
            teacher_conf = teacher_probs[, label]  
            student_conf = student_probs[, label]  

            
            loss = (teacher_conf ** gamma) + (( - student_conf) ** gamma)

            
            optimizer.zero_grad()
            (-loss).backward()  
            optimizer.step()

             (iteration + ) %  == :
                (
                      
                      
                      )

        
         torch.no_grad():
            augmented = .diffusion.decode(z)

         augmented

     () -> [torch.Tensor, torch.Tensor]:
        

        
         torch.no_grad():
            student_logits = .student(train_images)
            student_probs = F.softmax(student_logits, dim=-)
            student_confidence = torch.(student_probs, dim=-)[]

        
        num_to_augment = ((train_images) * augmentation_ratio)
        uncertain_indices = torch.argsort(student_confidence)[:num_to_augment]

        
        augmented_images = []
        augmented_labels = []

         idx  uncertain_indices:
            image = train_images[idx].unsqueeze()
            label = train_labels[idx].item()

            ()

            augmented = .generate_augmented_sample(image, label)

            augmented_images.append(augmented)
            augmented_labels.append(label)

        
        all_images = torch.cat([train_images] + augmented_images, dim=)
        all_labels = torch.cat([
            train_labels,
            torch.tensor(augmented_labels, device=train_labels.device)
        ], dim=)

         all_images, all_labels

Step 3: Implement Knowledge Distillation with ConfiG

class KDWithConfiG:
    """Knowledge distillation enhanced with confidence-guided augmentation"""

    def __init__(self, teacher_model, student_model,
                 diffusion_model, temperature: float = 4.0):
        self.teacher = teacher_model
        self.student = student_model
        self.diffusion = diffusion_model
        self.temperature = temperature

        self.confidence_aug = ConfidenceGuidedDiffusion(
            diffusion_model, teacher_model, student_model
        )

        self.optimizer = torch.optim.Adam(student_model.parameters(), lr=1e-4)

    def distillation_loss(self, student_logits: torch.Tensor,
                         teacher_logits: torch.Tensor) -> torch.Tensor:
        """KL divergence between student and teacher distributions"""

        student_probs = F.softmax(student_logits / self.temperature, dim=-1)
        teacher_probs = F.softmax(teacher_logits / self.temperature, dim=-1)

        kl_loss = F.kl_div(
            torch.log(student_probs + 1e-8),
            teacher_probs.detach(),
            reduction='batchmean'
        )

        return kl_loss * (self.temperature ** 2)

    def training_step(self, batch_images: torch.Tensor,
                     batch_labels: torch.Tensor) -> float:
        """Single training step on original + augmented data"""

        # Forward pass
        student_logits = self.student(batch_images)

         torch.no_grad():
            teacher_logits = .teacher(batch_images)

        
        loss = .distillation_loss(student_logits, teacher_logits)

        
        .optimizer.zero_grad()
        loss.backward()
        .optimizer.step()

         loss.item()

     () -> :
        

        history = {: []}

         epoch  (num_epochs):
            ()

            
            loss = .training_step(train_images, train_labels)
            history[].append(loss)

            ()

            
             (epoch + ) %  == :
                ()

                aug_images, aug_labels = .confidence_aug.generate_augmented_dataset(
                    train_images, train_labels, augmentation_ratio=
                )

                ()

                
                 aug_epoch  ():
                    aug_loss = .training_step(aug_images, aug_labels)
                    ()

         history

Step 4: Integration with Model-Centric Bias Mitigation

class HybridBiasMitigation:
    """Combine data-centric (ConfiG) and model-centric (TAB) approaches"""

    def __init__(self, teacher_model, student_model, diffusion_model):
        self.kd_config = KDWithConfiG(
            teacher_model, student_model, diffusion_model
        )

        # Model-centric: Train-aware batch normalization (TAB)
        self.model_centric = TrainAwareBatchNorm(student_model)

    def train_hybrid(self, train_images: torch.Tensor,
                    train_labels: torch.Tensor,
                    group_labels: torch.Tensor) -> Dict:
        """
        Data-centric: ConfiG augmentation targeting spurious features
        Model-centric: TAB encouraging invariant features
        """

        print("Starting hybrid bias mitigation training...")
        print("Data-centric: Confidence-guided augmentation")
        print("Model-centric: Train-aware batch normalization")

        # Phase 1: Data-centric augmentation
        kd_history = self.kd_config.train_with_augmentation(
            train_images, train_labels, num_epochs=10
        )

        # Phase 2: Model-centric refinement with TAB
        tab_history = self.model_centric.train_with_tab(
            train_images, train_labels, group_labels, num_epochs=5
        )

        return {
            'kd_history': kd_history,
            'tab_history': tab_history,
        }

Practical Guidance

1. Identify Spurious Correlations: Start by analyzing your training data for features that correlate with labels but are likely absent in test data (e.g., background, lighting, specific objects).

2. Teacher Quality Matters: ConfiG assumes teacher model is robust. If teacher is biased, augmentation won't help. Use a well-trained teacher or synthetic data.

3. Gamma Parameter: gamma=2.0 is empirically optimal. Higher γ concentrates learning on high-disagreement samples; lower γ spreads it more broadly.

4. Augmentation Ratio: Augment 30-50% of training data, focusing on uncertain samples. Over-augmentation can degrade in-distribution performance.

5. Hybrid Approach Works Best: Combine data-centric (ConfiG) augmentation with model-centric approaches (TAB, group normalization). They're complementary and achieve better results together.

6. Computational Cost: Diffusion-based augmentation is expensive (100+ iterations per image). Pre-generate augmented dataset in batch, don't do online.

Reference

• Paper: ConfiG (2506.02294)
• Key Innovation: Confidence-guided diffusion targeting spurious features
• Architecture: Teacher-student disagreement → augmentation objective
• Datasets: CelebA, SpuCo Birds, Spurious ImageNet
• Result: Superior performance under covariate shift compared to prior augmentation methods