stars-snn-data-free-knowledge-distillation

name	stars-snn-data-free-knowledge-distillation
description	STARS (Spike Tail-Aware Relational Synthesis) - plug-and-play method for ANN-to-SNN Data-Free Knowledge Distillation (DFKD). Augments BN-guided synthesis with Relational Consistency Alignment and Tail-Aware Regularization. Achieves up to 4.6% improvement on CIFAR-10 and 6.7% on CIFAR-100. Activation: SNN knowledge distillation, data-free distillation, ANN-to-SNN conversion, tail-aware regularization, relational consistency, spike threshold dynamics, 无数据蒸馏, 跨模态蒸馏.
license	Complete terms in LICENSE.txt
metadata	{"arxiv_id":"2605.27409","published":"2026-05-28","authors":"Shuhan Ye, Yi Yu, Qixin Zhang, Hui Lu, Jiaming He, Qinggang Zhang, Li Shen, Xudong Jiang","tags":["snn","knowledge-distillation","data-free","ann-to-snn","threshold-crossing","tail-probability","bn-matching","relational-consistency"]}

STARS: Spike Tail-Aware Relational Synthesis for ANN-to-SNN DFKD

Plug-and-play method for ANN-to-SNN data-free knowledge distillation that addresses the fundamental mismatch between ANN-oriented BN matching and SNN threshold-crossing dynamics.

Problem Statement

ANN-to-SNN knowledge distillation helps narrow the performance gap between ANNs and SNNs. However, in practical deployment settings, the original training data are often unavailable. Existing Data-Free Knowledge Distillation (DFKD) methods face a critical limitation:

ANN-oriented constraints (BN statistics matching) primarily regularize mean and variance, which remain under-constrained for SNN students whose responses depend on threshold-crossing dynamics, not just statistical moments.

Core Innovation: Two Complementary Objectives

STARS augments standard BN-guided synthesis with two key objectives:

1. Relational Consistency Alignment (RCA)

Preserves cross-sample relational consistency between teacher (ANN) and student (SNN):

• Ensures relative ordering of samples is maintained
• Captures manifold structure that BN matching ignores
• Prevents collapse to generic synthetic distributions

2. Tail-Aware Regularization (TAR)

Regularizes threshold-relevant tail probabilities through soft exceedance over teacher-derived thresholds:

• Directly targets the dynamics that govern SNN spike generation
• Focuses on the distribution tails where threshold crossing occurs
• Bridges the gap between continuous ANN outputs and discrete SNN spikes

Key Insight

SNN students require threshold-relevant information that BN statistics (mean/variance) do not capture. The tail probabilities of the distribution determine:

• Whether a neuron fires (crosses threshold)
• Spike timing (when threshold is crossed)
• Firing rate (frequency of threshold crossing)

Standard DFKD focuses on the bulk of the distribution; SNNs care about the tails.

Performance Results

Dataset	Baseline DFKD	STARS	Improvement
CIFAR-10	~85%	~89.6%	+4.6%
CIFAR-100	~60%	~66.7%	+6.7%
Tiny-ImageNet	~45%	~48-50%	+3-5%

STARS consistently improves conventional DFKD baselines and even surpasses several KD methods that use real data.

Implementation Components

Synthetic Data Generation Process

# STARS synthetic batch generation
def stars_synthesis(teacher_ann, snn_student, num_samples):
    # Step 1: BN-guided synthesis (standard DFKD)
    synthetic_data = bn_guided_generation(teacher_ann, num_samples)
    
    # Step 2: Relational Consistency Alignment
    teacher_features = teacher_ann.extract_features(synthetic_data)
    student_features = snn_student.extract_features(synthetic_data)
    
    # Preserve cross-sample relational structure
    rca_loss = compute_relational_consistency(
        teacher_features, student_features
    )
    
    # Step 3: Tail-Aware Regularization
    thresholds = estimate_spike_thresholds(teacher_ann)
    tail_probs = compute_tail_probabilities(student_features, thresholds)
    
    tar_loss = tail_aware_regularization(tail_probs, thresholds)
    
    # Combined optimization
    total_loss = bn_loss + alpha * rca_loss + beta * tar_loss
    
    return optimize_synthetic_batch(synthetic_data, total_loss)

Relational Consistency Alignment

def compute_relational_consistency(teacher_feat, student_feat):
    """
    Preserve cross-sample relational structure.
    
    Key: Maintain relative ordering and manifold geometry.
    """
    # Teacher relational matrix
    T_rel = compute_pairwise_relations(teacher_feat)
    
    # Student relational matrix
    S_rel = compute_pairwise_relations(student_feat)
    
    # Align relational structure
    loss = F.mse_loss(T_rel, S_rel)
    
    return loss

def compute_pairwise_relations(features):
    """
    Compute relational structure between samples.
    
    Options:
    1. Distance matrix (Euclidean/Cosine)
    2. Similarity matrix
    3. Ranking-based relations
    """
    # Normalize features
    features = F.normalize(features, dim=1)
    
    # Cosine similarity matrix
    sim_matrix = features @ features.T
    
    return sim_matrix

Tail-Aware Regularization

def tail_aware_regularization(student_output, teacher_threshold):
    """
    Regularize threshold-relevant tail probabilities.
    
    Focus on distribution tails where threshold crossing occurs.
    """
    # Estimate spike threshold from teacher
    threshold = estimate_threshold_from_teacher(teacher_output)
    
    # Compute exceedance probability (tail probability)
    # P(output > threshold) = area under tail
    exceedance = torch.sigmoid(student_output - threshold)
    
    # Tail loss: encourage distribution tail to match threshold region
    tail_loss = -torch.log(exceedance + 1e-8)  # Soft exceedance
    
    # Weight by distance from threshold (focus on critical region)
    weights = compute_tail_weights(student_output, threshold)
    
    return (tail_loss * weights).mean()

def estimate_threshold_from_teacher(teacher_output):
    """
    Derive spike threshold from ANN teacher outputs.
    
    Strategy: Use percentile or energy-based threshold estimation.
    """
    # Option 1: Percentile-based
    threshold = torch.quantile(teacher_output, 0.95)  # Top 5%
    
    # Option 2: Energy-based (maintain firing rate)
    # threshold = optimize_for_firing_rate(teacher_output, target_rate=0.1)
    
    return threshold

Threshold Estimation Strategies

Strategy 1: Percentile-Based

• Use top percentile of ANN outputs as threshold estimate
• Captures the high-activation region where SNN firing occurs

Strategy 2: Distribution Matching

• Match ANN output distribution to SNN membrane potential distribution
• Ensure similar firing rate statistics

Strategy 3: Layer-Wise Adaptation

• Different thresholds for different SNN layers
• Account for hierarchical threshold dynamics

Comparison with Existing Methods

Method	Data Requirement	SNN-Specific	Threshold-Aware	Performance
Standard KD	Requires original data	No	No	High
DFKD-BN	None	No	No	Medium
DFKD-Feature	None	Partial	No	Medium
STARS	None	Yes	Yes	High

STARS is the first DFKD method explicitly designed for SNN threshold-crossing dynamics.

Integration with SNN Training Frameworks

SpikingJelly Implementation

from spikingjelly.clock_driven import neuron, functional
import torch.nn.functional as F

class STARSDFKD:
    def __init__(self, teacher_ann, student_snn, alpha=0.5, beta=0.3):
        self.teacher = teacher_ann
        self.student = student_snn
        self.alpha = alpha  # RCA weight
        self.beta = beta    # TAR weight
    
    def synthesize_batch(self, batch_size):
        """Generate synthetic data with STARS objectives."""
        # Initialize random noise
        synthetic = torch.randn(batch_size, 3, 32, 32)
        synthetic.requires_grad = True
        
        optimizer = torch.optim.Adam([synthetic], lr=0.1)
        
        for step in range(100):
            # BN matching loss (standard DFKD)
            bn_loss = self.bn_matching_loss(synthetic)
            
            # Relational Consistency Alignment
            rca_loss = self.relational_consistency_loss(synthetic)
            
            # Tail-Aware Regularization
            tar_loss = self.tail_aware_regularization_loss(synthetic)
            
            # Combined optimization
            total_loss = bn_loss + self.alpha * rca_loss + self.beta * tar_loss
            
            optimizer.zero_grad()
            total_loss.backward()
            optimizer.step()
        
        return synthetic.detach()
    
    def tail_aware_regularization_loss(self, synthetic_data):
        """Implement TAR for SNN threshold dynamics."""
        # Get ANN teacher outputs
        teacher_out = self.teacher(synthetic_data)
        
        # Get SNN student outputs (membrane potentials)
        student_out = self.student(synthetic_data)
        
        # Estimate threshold from teacher
        threshold = torch.quantile(teacher_out.flatten(), 0.95)
        
        # Compute tail exceedance
        exceedance = torch.sigmoid(student_out - threshold)
        
        # Tail regularization loss
        tar_loss = -torch.log(exceedance.mean() + 1e-8)
        
        return tar_loss

Hyperparameter Guidelines

Alpha (RCA Weight)

• Range: [0.3, 0.7]
• Higher alpha: Stronger relational structure preservation
• Recommendation: 0.5 for general use

Beta (TAR Weight)

• Range: [0.2, 0.5]
• Higher beta: More threshold-aware optimization
• Recommendation: 0.3 for CIFAR-scale datasets

Threshold Percentile

• Range: [0.90, 0.98]
• Higher percentile: More selective firing
• Recommendation: 0.95 for standard SNNs

Application Scenarios

1. Edge Deployment

• Scenario: Deploy SNN on neuromorphic hardware without access to original training data
• Benefit: Generate synthetic data locally, distill ANN knowledge to SNN
• Use case: Medical devices, autonomous drones, IoT sensors

2. Privacy-Preserving Distillation

• Scenario: Distill proprietary ANN models to public SNN architectures
• Benefit: Transfer knowledge without sharing original datasets
• Use case: Model licensing, cross-organization deployment

3. Architecture Search Support

• Scenario: Evaluate candidate SNN architectures without training on real data
• Benefit: Rapid architecture screening via synthetic data distillation
• Use case: SNN design optimization, hardware-aware search

Pitfalls and Solutions

Pitfall 1: Threshold Misalignment

Problem: Teacher-derived thresholds may not match SNN firing thresholds. Solution: Adaptive threshold estimation using layer-wise membrane potential statistics.

Pitfall 2: Synthetic Data Collapse

Problem: Synthetic batches converge to generic distributions, losing diversity. Solution: RCA prevents collapse by enforcing relational structure preservation.

Pitfall 3: Scale Mismatch

Problem: ANN output scale differs from SNN membrane potential scale. Solution: Normalize outputs before threshold estimation, use relative thresholds.

Research Connections

STARS bridges two key domains:

1. Data-Free Knowledge Distillation: Transfer knowledge without access to original data
2. SNN Threshold Dynamics: Discrete spike generation governed by threshold crossing

The intersection reveals that SNN-specific distillation requires tail-aware optimization, not just mean/variance matching.

Related Skills

• circulate-firing-snn-training - Direct SNN training with enhanced neurons
• ann-to-snn-conversion - ANN-to-SNN conversion methods
• snn-performance-analysis - SNN evaluation and benchmarking
• knowledge-distillation-patterns - General KD frameworks

Experimental Validation

Datasets Tested

• CIFAR-10: 10-class image classification
• CIFAR-100: 100-class fine-grained classification
• Tiny-ImageNet: 200-class small-scale ImageNet

ANN-SNN Pairs

• Teacher: ResNet-18, VGG-16, MobileNet
• Student: Spiking ResNet, Spiking VGG, Spiking MobileNet

Comparison Baselines

• Standard DFKD (BN matching only)
• Feature-based DFKD
• Real data KD (upper bound reference)

References

• Paper: arXiv:2605.27409 - "STARS: Spike Tail-Aware Relational Synthesis for ANN-to-SNN Data-Free Knowledge Distillation"
• Related: DFKD literature, SNN conversion methods, threshold-based spiking dynamics

Validation

After creating or updating this skill, run:

python3 ~/.hermes/skills/skill-creator/scripts/quick_validate.py ~/.hermes/skills/ai_collection/stars-snn-data-free-knowledge-distillation