maximum-entropy-network-structure-function

# Maximum Entropy Networks for Structure-Function Relationships ## Overview This methodology proposes a **normative, training-algorithm-independent** approach to understanding how neural network function constrains connectivity. Instead of training networks with gradient descent and examining the resulting structure, it derives the unique maximum-entropy connectivity distribution subject to task constraints. **Core insight**: Task constraints + entropy maximization = population structure emergence, matching gradient-trained networks quantitatively across learning regimes. ## When to Use - Analyzing why neural populations have specific connectivity patterns - Understanding context-dependent computation in neural circuits - Designing normative (principled) models of neural connectivity - Validating that trained neural network structure reflects task requirements rather than training artifacts - Studying transitions between specialized and unspecialized neural populations ## Methodology ### 1. Framework Setup ```python # Represent connectivity as probability distribution over single-neuron weights # W ~ p(W) where W ∈ R^{N×N} # Express task requirements as constraints on the distribution # E[f_k(W)] = c_k for k = 1, ..., K (task constraints) # Maximize Shannon entropy H[p] = -∫ p(W) log p(W) dW # subject to constraints → unique maximum entropy distribution ``` ### 2. Key Mathematical Result The maximum entropy distribution over connectivity takes the Boltzmann form: ``` p(W) ∝ exp(-∑_k λ_k f_k(W)) ``` where λ_k are Lagrange multipliers determined by constraints. **Analytical tractability**: Map nonlinear networks onto gain-modulated linear models: ``` output = G(c) · W · input # G(c) = context-dependent gain matrix # Nonlinear network ≅ gain-modulated linear model analytically ``` ### 3. Context-Dependent Input Selection Task For a network selecting relevant inputs based on context c: ```python # Task constraint: correct input selected for each context # E[correct_output | context=c] = target_c for all contexts c # Result: emergence of distinct neuron populations # Each population defined by contextual gain pattern g_i(c) # Population i responds to context c with gain g_i(c) ``` ### 4. Phase Transitions Two key parameters drive population structure transitions: ``` Weight scale β: β → 0: random, unstructured connectivity β → ∞: structured, task-optimized connectivity Number of contexts K: K small: context-specialized populations emerge K large → ∞: unspecialized, random populations ``` ### 5. Matching Gradient-Trained Networks The maximum entropy framework **quantitatively matches** networks trained by: - Gradient descent (SGD, Adam) - Hebbian learning - Various learning rates and regularization schemes This suggests maximum entropy is a fundamental principle, not a property of any particular algorithm. ## Implementation Steps ### Step 1: Define Task Constraints ```python def compute_task_constraints(task, network_params): """ Extract task requirements as statistical constraints. Returns: constraints: list of (function, target_value) pairs """ constraints = [] for context in task.contexts: # Constraint: network produces correct output for this context f_k = lambda W, c=context: task.evaluate(W, c) c_k = task.target_output(context) constraints.append((f_k, c_k)) return constraints # Apply to context-dependent input selection contexts = [0, 1, 2, ..., K-1] # K contexts targets = [input_0, input_1, ..., input_{K-1}] # correct input per context ``` ### Step 2: Compute Maximum Entropy Distribution ```python from scipy.optimize import minimize def maximum_entropy_connectivity(constraints, beta=1.0): """ Compute maximum entropy distribution parameters. Returns: lambdas: Lagrange multipliers p_star: maximum entropy distribution """ # Solve dual problem: minimize free energy F(λ) = log Z(λ) + λ·c def free_energy(lambdas): Z = compute_partition_function(lambdas, beta) return np.log(Z) + lambdas @ constraint_values result = minimize(free_energy, x0=np.zeros(len(constraints))) lambdas = result.x return lambdas ``` ### Step 3: Analyze Population Structure ```python def analyze_populations(lambdas, network_params): """ Identify emergent neural populations from maximum entropy solution. Each population = cluster of neurons with similar gain patterns g(c). """ # Compute expected gains per neuron per context gains = compute_expected_gains(lambdas, network_params) # Cluster neurons by gain pattern from sklearn.cluster import KMeans populations = KMeans(n_clusters=K).fit_predict(gains) return populations, gains ``` ## Key Results ### Population Emergence - With K contexts, maximum entropy solution naturally creates **K distinct populations** - Each population i has a characteristic gain pattern: g_i(c) = high for context i, low otherwise - This matches experimental observations of mixed-selectivity neurons in PFC ### Transition Diagrams ``` β (weight scale) vs K (contexts): β large ↑ | STRUCTURED STRUCTURED | SPECIALIZED → UNSPECIALIZED | (K populations) (random) | | RANDOM RANDOM | SPECIALIZED UNSPECIALIZED +----------------------→ K large ``` ### Quantitative Match with Gradient Descent For 2-layer networks on context-dependent tasks: - **Population count**: MaxEnt predicts K populations; gradient descent produces K populations - **Gain patterns**: Correlation > 0.95 between MaxEnt and trained networks - **Selectivity**: Both show parallel transition from structured to random at same β threshold ## Applications ### 1. Understanding Prefrontal Cortex ```python # PFC shows mixed selectivity to contexts, tasks, stimuli # Maximum entropy + task constraints predicts this without assuming specific circuit contexts = ['attention_left', 'attention_right', 'task_A', 'task_B'] # → predicts ~4 mixed-selectivity populations matching electrophysiology ``` ### 2. Normative Model Validation ```python # Test: does trained network match maximum entropy prediction? def validate_against_maxent(trained_W, task): maxent_stats = compute_maxent_statistics(task) trained_stats = compute_statistics(trained_W) correlation = np.corrcoef(maxent_stats, trained_stats)[0,1] print(f"MaxEnt-Training correlation: {correlation:.3f}") # Values > 0.9 indicate task constraints (not training) drive structure ``` ### 3. Network Design ```python # Design networks that maximally express task structure # vs. networks that are maximally random (robust) optimal_beta = find_task_performance_threshold(task) network = sample_from_maxent_distribution(lambdas, beta=optimal_beta) ``` ## Pitfalls - **Constraint definition matters**: Poorly defined task constraints → wrong population structure - **Analytical tractability**: Only works with gain-modulated linear approximation; nonlinear regimes require MCMC sampling - **Assumes task is fully specified**: Real neural circuits have many implicit constraints not captured by explicit task definition - **Phase boundaries are task-specific**: K and β thresholds depend heavily on the specific computation ## Key References - **Primary**: Hruza & Ostojic (2026). "Balancing structure and randomness: maximum entropy networks for context-dependent computations." arXiv:2605.25607 - Jaynes (1957). Maximum entropy principle - Rigotti et al. (2013). Mixed selectivity in PFC, Nature - Sompolinsky & Zippelius (1982). Gain modulation in neural networks ## Activation Keywords maximum entropy, neural connectivity, structure-function, context-dependent computation, normative model, population structure, gain modulation, weight scale, mixed selectivity, prefrontal cortex, task constraints, Shannon entropy, Boltzmann distribution