polca-stochastic-generative-optimization

name	polca-stochastic-generative-optimization
title	POLCA: Stochastic Generative Optimization with LLM
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2603.14769
keywords	["Generative Optimization","LLM as Optimizer","Stochastic Feedback","Priority Queues","Meta-Learning"]
description	Use language models as optimizers to iteratively improve complex systems (prompts, code, agent policies) through noisy feedback and sampling. POLCA maintains a priority queue of candidates, explores with epsilon-nets, and learns meta-insights from trial history.

POLCA: Stochastic Generative Optimization with Language Models

Most optimization of LLM-based systems happens offline through manual iteration. POLCA inverts this by making the language model itself the optimizer, receiving both numerical rewards and textual feedback to propose improved solutions iteratively. This approach handles real-world stochasticity from noisy evaluations, sampling variance, and system non-determinism—enabling continuous improvement of prompts, agents, and code under realistic conditions.

The key technical contribution is combining three mechanisms: priority queue management for exploration-exploitation tradeoff, epsilon-nets for diversity preservation, and an LLM-based summarizer that learns from historical trials. The framework includes theoretical convergence guarantees under stochasticity.

Core Concept

POLCA treats a generative language model as an optimization algorithm where:

1. Input: Current candidate solution + numerical reward + textual feedback
2. Process: LLM proposes improved candidates by reasoning about past attempts
3. Output: Diverse set of candidate solutions ranked by potential improvement
4. Feedback Loop: Evaluate candidates, aggregate insights, feed back to LLM

This setup enables the model to learn meta-patterns across attempts without requiring explicit gradient computation or hand-engineered search strategies.

Architecture Overview

• Priority Queue — Maintains sorted candidate solutions; candidate ordering balances high-score exploitation with under-explored diversity
• Epsilon-Net Mechanism — Preserves exploration by tracking which parameter regions have been sufficiently sampled
• LLM Summarizer — Generates textual meta-insights from trial history to condition future proposals; learns general improvement patterns
• Numerical Reward Signal — Primary feedback for ranking candidates; can be noisy or adversarial
• Textual Feedback — Secondary signal that explains why solutions succeeded/failed, enabling meta-learning
• Stochastic Dynamics — Explicitly models multiple sources of noise (feedback, sampling, system behavior)

Implementation Steps

Start by setting up the priority queue and epsilon-net tracking to manage the exploration-exploitation balance.

import heapq
from collections import defaultdict
import numpy  np

 :
    

     ():
        .epsilon = epsilon  
        .candidates = []  
        .explored_regions = ()
        .max_size = max_size

     ():
        
        
        region_key = ._discretize(candidate)

         region_key  .explored_regions:
            
            best_in_region = ._best_in_region(region_key)
             score <= best_in_region + :  
                 

        
        heapq.heappush(.candidates,
                      (-score, (.candidates), candidate, feedback))
        .explored_regions.add(region_key)

        
         (.candidates) > .max_size:
            heapq.heappop(.candidates)

         

     ():
        
         [candidate  _, _, candidate, _  .candidates[:k]]

     ():
        
         ((((c)) % bins)  c  candidate[:])

     ():
        
         (((-score, score)  neg_score, _, _, _
                    .candidates
                    ._discretize(_) == region_key),
                  default=(, ))[]