stars-snn-data-free-knowledge-distillation

# STARS: Spike Tail-Aware Relational Synthesis for ANN-to-SNN DFKD Plug-and-play method for ANN-to-SNN data-free knowledge distillation that addresses the fundamental mismatch between ANN-oriented BN matching and SNN threshold-crossing dynamics. ## Problem Statement ANN-to-SNN knowledge distillation helps narrow the performance gap between ANNs and SNNs. However, in practical deployment settings, the original training data are often unavailable. Existing **Data-Free Knowledge Distillation (DFKD)** methods face a critical limitation: **ANN-oriented constraints** (BN statistics matching) primarily regularize **mean and variance**, which remain **under-constrained** for SNN students whose responses depend on **threshold-crossing dynamics**, not just statistical moments. ## Core Innovation: Two Complementary Objectives STARS augments standard BN-guided synthesis with two key objectives: ### 1. Relational Consistency Alignment (RCA) Preserves **cross-sample relational consistency** between teacher (ANN) and student (SNN): - Ensures relative ordering of samples is maintained - Captures manifold structure that BN matching ignores - Prevents collapse to generic synthetic distributions ### 2. Tail-Aware Regularization (TAR) Regularizes **threshold-relevant tail probabilities** through soft exceedance over teacher-derived thresholds: - Directly targets the dynamics that govern SNN spike generation - Focuses on the distribution tails where threshold crossing occurs - Bridges the gap between continuous ANN outputs and discrete SNN spikes ## Key Insight SNN students require **threshold-relevant information** that BN statistics (mean/variance) do not capture. The tail probabilities of the distribution determine: - Whether a neuron fires (crosses threshold) - Spike timing (when threshold is crossed) - Firing rate (frequency of threshold crossing) Standard DFKD focuses on the bulk of the distribution; SNNs care about the **tails**. ## Performance Results | Dataset | Baseline DFKD | STARS | Improvement | |---------|--------------|-------|-------------| | CIFAR-10 | ~85% | ~89.6% | **+4.6%** | | CIFAR-100 | ~60% | ~66.7% | **+6.7%** | | Tiny-ImageNet | ~45% | ~48-50% | **+3-5%** | STARS consistently improves conventional DFKD baselines and even **surpasses several KD methods that use real data**. ## Implementation Components ### Synthetic Data Generation Process ```python # STARS synthetic batch generation def stars_synthesis(teacher_ann, snn_student, num_samples): # Step 1: BN-guided synthesis (standard DFKD) synthetic_data = bn_guided_generation(teacher_ann, num_samples) # Step 2: Relational Consistency Alignment teacher_features = teacher_ann.extract_features(synthetic_data) student_features = snn_student.extract_features(synthetic_data) # Preserve cross-sample relational structure rca_loss = compute_relational_consistency( teacher_features, student_features ) # Step 3: Tail-Aware Regularization thresholds = estimate_spike_thresholds(teacher_ann) tail_probs = compute_tail_probabilities(student_features, thresholds) tar_loss = tail_aware_regularization(tail_probs, thresholds) # Combined optimization total_loss = bn_loss + alpha * rca_loss + beta * tar_loss return optimize_synthetic_batch(synthetic_data, total_loss) ``` ### Relational Consistency Alignment ```python def compute_relational_consistency(teacher_feat, student_feat): """ Preserve cross-sample relational structure. Key: Maintain relative ordering and manifold geometry. """ # Teacher relational matrix T_rel = compute_pairwise_relations(teacher_feat) # Student relational matrix S_rel = compute_pairwise_relations(student_feat) # Align relational structure loss = F.mse_loss(T_rel, S_rel) return loss def compute_pairwise_relations(features): """ Compute relational structure between samples. Options: 1. Distance matrix (Euclidean/Cosine) 2. Similarity matrix 3. Ranking-based relations """ # Normalize features features = F.normalize(features, dim=1) # Cosine similarity matrix sim_matrix = features @ features.T return sim_matrix ``` ### Tail-Aware Regularization ```python def tail_aware_regularization(student_output, teacher_threshold): """ Regularize threshold-relevant tail probabilities. Focus on distribution tails where threshold crossing occurs. """ # Estimate spike threshold from teacher threshold = estimate_threshold_from_teacher(teacher_output) # Compute exceedance probability (tail probability) # P(output > threshold) = area under tail exceedance = torch.sigmoid(student_output - threshold) # Tail loss: encourage distribution tail to match threshold region tail_loss = -torch.log(exceedance + 1e-8) # Soft exceedance # Weight by distance from threshold (focus on critical region) weights = compute_tail_weights(student_output, threshold) return (tail_loss * weights).mean() def estimate_threshold_from_teacher(teacher_output): """ Derive spike threshold from ANN teacher outputs. Strategy: Use percentile or energy-based threshold estimation. """ # Option 1: Percentile-based threshold = torch.quantile(teacher_output, 0.95) # Top 5% # Option 2: Energy-based (maintain firing rate) # threshold = optimize_for_firing_rate(teacher_output, target_rate=0.1) return threshold ``` ## Threshold Estimation Strategies ### Strategy 1: Percentile-Based - Use top percentile of ANN outputs as threshold estimate - Captures the high-activation region where SNN firing occurs ### Strategy 2: Distribution Matching - Match ANN output distribution to SNN membrane potential distribution - Ensure similar firing rate statistics ### Strategy 3: Layer-Wise Adaptation - Different thresholds for different SNN layers - Account for hierarchical threshold dynamics ## Comparison with Existing Methods | Method | Data Requirement | SNN-Specific | Threshold-Aware | Performance | |--------|-----------------|--------------|-----------------|-------------| | Standard KD | Requires original data | No | No | High | | DFKD-BN | None | No | No | Medium | | DFKD-Feature | None | Partial | No | Medium | | **STARS** | None | Yes | Yes | **High** | STARS is the first DFKD method explicitly designed for SNN threshold-crossing dynamics. ## Integration with SNN Training Frameworks ### SpikingJelly Implementation ```python from spikingjelly.clock_driven import neuron, functional import torch.nn.functional as F class STARSDFKD: def __init__(self, teacher_ann, student_snn, alpha=0.5, beta=0.3): self.teacher = teacher_ann self.student = student_snn self.alpha = alpha # RCA weight self.beta = beta # TAR weight def synthesize_batch(self, batch_size): """Generate synthetic data with STARS objectives.""" # Initialize random noise synthetic = torch.randn(batch_size, 3, 32, 32) synthetic.requires_grad = True optimizer = torch.optim.Adam([synthetic], lr=0.1) for step in range(100): # BN matching loss (standard DFKD) bn_loss = self.bn_matching_loss(synthetic) # Relational Consistency Alignment rca_loss = self.relational_consistency_loss(synthetic) # Tail-Aware Regularization tar_loss = self.tail_aware_regularization_loss(synthetic) # Combined optimization total_loss = bn_loss + self.alpha * rca_loss + self.beta * tar_loss optimizer.zero_grad() total_loss.backward() optimizer.step() return synthetic.detach() def tail_aware_regularization_loss(self, synthetic_data): """Implement TAR for SNN threshold dynamics.""" # Get ANN teacher outputs teacher_out = self.teacher(synthetic_data) # Get SNN student outputs (membrane potentials) student_out = self.student(synthetic_data) # Estimate threshold from teacher threshold = torch.quantile(teacher_out.flatten(), 0.95) # Compute tail exceedance exceedance = torch.sigmoid(student_out - threshold) # Tail regularization loss tar_loss = -torch.log(exceedance.mean() + 1e-8) return tar_loss ``` ## Hyperparameter Guidelines ### Alpha (RCA Weight) - Range: [0.3, 0.7] - Higher alpha: Stronger relational structure preservation - Recommendation: 0.5 for general use ### Beta (TAR Weight) - Range: [0.2, 0.5] - Higher beta: More threshold-aware optimization - Recommendation: 0.3 for CIFAR-scale datasets ### Threshold Percentile - Range: [0.90, 0.98] - Higher percentile: More selective firing - Recommendation: 0.95 for standard SNNs ## Application Scenarios ### 1. Edge Deployment - Scenario: Deploy SNN on neuromorphic hardware without access to original training data - Benefit: Generate synthetic data locally, distill ANN knowledge to SNN - Use case: Medical devices, autonomous drones, IoT sensors ### 2. Privacy-Preserving Distillation - Scenario: Distill proprietary ANN models to public SNN architectures - Benefit: Transfer knowledge without sharing original datasets - Use case: Model licensing, cross-organization deployment ### 3. Architecture Search Support - Scenario: Evaluate candidate SNN architectures without training on real data - Benefit: Rapid architecture screening via synthetic data distillation - Use case: SNN design optimization, hardware-aware search ## Pitfalls and Solutions ### Pitfall 1: Threshold Misalignment **Problem**: Teacher-derived thresholds may not match SNN firing thresholds. **Solution**: Adaptive threshold estimation using layer-wise membrane potential statistics. ### Pitfall 2: Synthetic Data Collapse **Problem**: Synthetic batches converge to generic distributions, losing diversity. **Solution**: RCA prevents collapse by enforcing relational structure preservation. ### Pitfall 3: Scale Mismatch **Problem**: ANN output scale differs from SNN membrane potential scale. **Solution**: Normalize outputs before threshold estimation, use relative thresholds. ## Research Connections STARS bridges two key domains: 1. **Data-Free Knowledge Distillation**: Transfer knowledge without access to original data 2. **SNN Threshold Dynamics**: Discrete spike generation governed by threshold crossing The intersection reveals that SNN-specific distillation requires **tail-aware optimization**, not just mean/variance matching. ## Related Skills - `circulate-firing-snn-training` - Direct SNN training with enhanced neurons - `ann-to-snn-conversion` - ANN-to-SNN conversion methods - `snn-performance-analysis` - SNN evaluation and benchmarking - `knowledge-distillation-patterns` - General KD frameworks ## Experimental Validation ### Datasets Tested - CIFAR-10: 10-class image classification - CIFAR-100: 100-class fine-grained classification - Tiny-ImageNet: 200-class small-scale ImageNet ### ANN-SNN Pairs - Teacher: ResNet-18, VGG-16, MobileNet - Student: Spiking ResNet, Spiking VGG, Spiking MobileNet ### Comparison Baselines - Standard DFKD (BN matching only) - Feature-based DFKD - Real data KD (upper bound reference) ## References - Paper: arXiv:2605.27409 - "STARS: Spike Tail-Aware Relational Synthesis for ANN-to-SNN Data-Free Knowledge Distillation" - Related: DFKD literature, SNN conversion methods, threshold-based spiking dynamics ## Validation After creating or updating this skill, run: ```bash python3 ~/.hermes/skills/skill-creator/scripts/quick_validate.py ~/.hermes/skills/ai_collection/stars-snn-data-free-knowledge-distillation ```