poisson-gradient-estimation

# Poisson Gradient Estimation for Latent Variable Models > First systematic comparison of Exponential Arrival Time (EAT) and Gumbel-SoftMax (GSM) methods for differentiating through Poisson-distributed latent variables in computational neuroscience. ## Metadata - **Source**: arXiv:2602.03896 - **Authors**: Michael Ibrahim, Hanqi Zhao, Eli Sennesh, Zhi Li, Anqi Wu - **Published**: 2026-02-03 - **Category**: stat.ML ## Core Methodology ### Problem Statement Poisson-distributed latent variable models are widely used in computational neuroscience, but differentiating through discrete stochastic samples remains challenging. This work addresses gradient estimation for: - Variational autoencoders with Poisson latents - Partially observable generalized linear models (inferring latent neural connectivity from spike trains) ### Key Methods Compared #### 1. Exponential Arrival Time (EAT) Simulation - Simulates Poisson process via exponential inter-arrival times - **Modified EAT contribution**: Unbiased first moment (exactly matching firing rate), reduced second-moment bias - **Advantages**: Better distributional fidelity, higher robustness to hyperparameters #### 2. Gumbel-SoftMax (GSM) Relaxation - Continuous relaxation of discrete sampling - Differentiable approximation to Poisson sampling - **Trade-offs**: Simpler implementation but may introduce bias ## Implementation Guide ### Prerequisites ```python import torch import numpy as np ``` ### Modified EAT Method ```python def modified_eat_sample(rate, shape): """ Modified Exponential Arrival Time sampling for Poisson variables. Provides unbiased first moment (exact firing rate matching). Args: rate: Poisson rate parameter (lambda) shape: Output shape Returns: Poisson samples with unbiased first moment """ # Standard Poisson sampling with moment correction return torch.poisson(rate) # Simplified - see paper for full def compute_poisson_gradient_eat(model, data, rate_param): """Compute gradients using EAT method.""" # Enable gradient computation through stochastic nodes # Implementation depends on specific model architecture pass ``` ### GSM Alternative ```python def gumbel_softmax_poisson(logits, temperature=1.0, hard=False): """ Gumbel-SoftMax relaxation for Poisson-like discrete sampling. Args: logits: Log probabilities temperature: Softmax temperature (lower = more discrete) hard: If True, returns hard samples with soft gradients """ gumbel_noise = -torch.log(-torch.log(torch.rand_like(logits) + 1e-10) + 1e-10) y_soft = torch.softmax((logits + gumbel_noise) / temperature, dim=-1) if hard: # Straight-through estimator index = y_soft.max(dim=-1, keepdim=True)[1] y_hard = torch.zeros_like(logits).scatter_(-1, index, 1.0) return (y_hard - y_soft).detach() + y_soft return y_soft ``` ### Evaluation Framework ```python def evaluate_gradient_method(model, data_loader, method='eat'): """ Evaluate gradient estimation method on downstream tasks. Metrics: 1. Distributional fidelity: KL divergence from true Poisson 2. Gradient quality: Gradient variance, bias 3. Task performance: ELBO, prediction accuracy """ results = { 'kl_divergence': [], 'gradient_variance': [], 'task_performance': [] } for batch in data_loader: if method == 'eat': gradient = compute_poisson_gradient_eat(model, batch) elif method == 'gsm': gradient = compute_poisson_gradient_gsm(model, batch) # Evaluate metrics # ... return results ``` ## Applications ### 1. Variational Autoencoders with Poisson Latents ```python class PoissonVAE(nn.Module): def __init__(self, input_dim, latent_dim): super().__init__() self.encoder = nn.Sequential( nn.Linear(input_dim, 256), nn.ReLU(), nn.Linear(256, latent_dim * 2) ) self.decoder = nn.Sequential( nn.Linear(latent_dim, 256), nn.ReLU(), nn.Linear(256, input_dim), nn.Sigmoid() ) def forward(self, x): params = self.encoder(x) rate = torch.exp(params[:, :latent_dim]) z = modified_eat_sample(rate, rate.shape) x_recon = self.decoder(z) return x_recon, rate, z ``` ### 2. Latent Neural Connectivity Inference Infer latent connectivity from observed spike trains in partially observable GLMs. ## Recommendations for Practitioners | Method | Best For | Avoid When | |--------|----------|------------| | Modified EAT | High fidelity, robust gradients | Simple/quick implementation | | GSM | Quick prototyping, simple models | High precision required | **Key Findings**: 1. Modified EAT exhibits better overall performance (often comparable to exact gradients) 2. Substantially higher robustness to hyperparameter choices 3. Preferred for production systems requiring stable training ## Pitfalls - **Hyperparameter sensitivity**: GSM can be sensitive to temperature scheduling - **Numerical stability**: EAT requires careful handling of near-zero rates - **Computational cost**: EAT may be slower than GSM in some implementations ## Related Skills - spiking-neural-network-analysis - computational-neuroscience-in-llm-era - neural-population-dynamics ## References - arXiv:2602.03896 - A Hitchhiker's Guide to Poisson Gradient Estimation