maximum-entropy-neural-connectivity

name	maximum-entropy-neural-connectivity
description	Maximum entropy principle for neural network connectivity — normative framework for understanding how task constraints shape neural connectivity structure without gradient descent.
tags	["neuroscience","neural-networks","maximum-entropy","connectivity","computational-neuroscience","brain-network"]
category	ai_collection
version	1.0
source	arXiv:2605.25607
authors	["Ludwig Hruza","Srdjan Ostojic"]

Maximum Entropy Neural Connectivity

Overview

Maximum entropy framework for deriving minimally-biased neural network connectivity from task constraints, independent of learning algorithms. Bridges normative theory with gradient-descent trained networks.

Trigger conditions: Use when studying relationship between neural network structure and function, connectivity analysis, normative models of brain networks, context-dependent computation, neuro-AI alignment.

Core Methodology

Principle

Express connectivity as a probability distribution over single-neuron weights, impose task requirements as constraints, and find the distribution maximizing Shannon entropy — the unique "least biased" solution consistent with functional requirements.

Mathematical Framework

1. Prior: Start with a homogeneous (uninformative) distribution over weights
2. Constraints: Express task requirements (e.g., input selection, context-dependent gating) as expected-value constraints on the distribution
3. Maximization: Solve max-entropy subject to constraints → yields exponential family distribution
4. Weight scale parameter λ: Controls balance between random/structured connectivity

Key Analytical Tractability

• Map nonlinear networks → gain-modulated linear models
• Enables closed-form maximum entropy inference in 2-layer feedforward networks
• Context-dependent input-selection tasks become analytically tractable

Key Findings

1. Emergent neural populations: Maximizing entropy under task constraints spontaneously produces neuron populations each defined by their pattern of contextual gain modulation

2. Context-number transition: Increasing number of contexts drives a phase transition from context-specialized → unspecialized/random populations

3. Weight-scale transition: Increasing weight scale drives transition from structured → random stimulus selectivity

4. Matches gradient descent: Maximum entropy connectivity matches both qualitatively AND quantitatively the structure of gradient-descent-trained networks across different learning regimes

Implications for Neuroscience

• Neural connectivity structure can be understood as entropy maximization under task constraints, not just gradient-based learning
• Provides a task-independent normative principle for brain connectivity
• Explains why random connectivity can achieve task performance (high-entropy solution)
• Suggests that the brain may solve tasks through the minimum-structure solution consistent with functional requirements

Implementation Steps

1. Define task constraints: Formalize cognitive task requirements as expected-value constraints on weight distributions
2. Set up Lagrangian: L(p) = H(p) - Σλᵢ⟨fᵢ(W)⟩ for each constraint fᵢ
3. Solve max-entropy: Optimize using exponential family parameterization
4. Gain modulation mapping: Transform nonlinear activations to linear gain-modulated model for tractability
5. Vary weight scale λ: Trace phase transitions from structured to random regime
6. Compare to gradient descent: Validate by training equivalent network and comparing connectivity statistics

Code Pattern (Python)

import numpy as np
from scipy.optimize import minimize
from scipy.special import softmax

def max_entropy_connectivity(task_constraints, n_neurons, n_contexts, weight_scale=1.0):
    """
    Compute maximum entropy connectivity distribution subject to task constraints.
    
    Args:
        task_constraints: list of (constraint_fn, target_value) tuples
        n_neurons: number of neurons in layer
        n_contexts: number of context signals
        weight_scale: λ parameter controlling structure/randomness balance
    
    Returns:
        weight_distribution: parameters of the exponential family distribution
        populations: identified neural population clusters
    """
    # Gain-modulated linear approximation
    # W_eff[i,j] = g_i(c) * W[i,j] where g_i(c) is context-dependent gain
    
    # Parameterize as exponential family
    # p(W) ∝ exp(Σ λᵢ fᵢ(W) - weight_scale * ||W||²)
    
    def neg_entropy(lambdas):
        # Compute -H(p_lambda) using partition function
        pass
    
    def constraint_violations(lambdas):
        # Check if E_p[fᵢ(W)] = target_i for all i
        pass
    
    # Solve dual problem
    result = minimize(neg_entropy, x0=np.zeros(len(task_constraints)),
                     constraints={'type': 'eq', 'fun': constraint_violations})
    
    return result

def identify_populations(weight_distribution, n_contexts):
    """Cluster neurons by contextual gain modulation pattern."""
    # Each neuron i has a gain profile g_i = [g_i(c1), g_i(c2), ...]
    
     sklearn.cluster  KMeans
    gains = compute_contextual_gains(weight_distribution, n_contexts)
    km = KMeans(n_clusters=n_contexts + )  
    populations = km.fit_predict(gains)
     populations

Key Parameters

Parameter	Description	Effect
Weight scale λ	Controls entropy vs structure	High λ → random; Low λ → structured
n_contexts	Number of context signals	More contexts → transition to unspecialized
Constraint tightness	How strongly task imposes structure	Tighter → more structured connectivity

Pitfalls

• Analytical tractability only in 2-layer feedforward: Deep networks require numerical approximations
• Gain-modulation mapping is approximate: Nonlinear networks mapped to linear gains as approximation
• Degenerate tasks: Tasks with redundant constraints may not uniquely determine connectivity
• Finite-sample effects: Maximum entropy solutions assume infinite population; finite networks show fluctuations

Connections to Existing Skills

• Related to: maximum-entropy-network-structure-function, neural-population-dynamics, brain-connectivity-analysis
• Compare to: gradient-descent connectivity analysis in low-rank-rnn-learning-dynamics
• Extends: normative neuroscience frameworks in efficient-coding-criticality

References

• Hruza & Ostojic (2026). "Balancing structure and randomness: maximum entropy networks for context-dependent computations." arXiv:2605.25607
• Jaynes (1957). Information theory and statistical mechanics.
• Sompolinsky et al. (1988). Chaos in random neural networks.