longcat-flash-prover

name	longcat-flash-prover
title	LongCat-Flash-Prover: Hierarchical Importance Sampling for Formal Reasoning
version	0.0.3
engine	skillxiv-v0.0.3-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2603.21065
keywords	["Formal Reasoning","Reinforcement Learning","Policy Optimization","Theorem Proving"]
description	Integrate agentic tool interaction (Lean4 compiler, syntax checkers) with curriculum-based RL for formal reasoning. Replace standard importance sampling with Hierarchical Importance Sampling Policy Optimization (HisPO): sequence-level masking removes train-inference discrepancies, token-level masking filters inconsistent tokens, staleness control manages policy drift. Achieves 97.1% auto-formalization (vs 83% baseline), 95.5% MiniF2F-Test (72 attempts vs 1,024+), 70.8% ProverBench.

Component Identification

Old Design (Standard Supervised Finetuning)

• No tool interaction (no Lean verification)
• Single-turn generation without feedback
• No curriculum on interaction complexity
• Standard supervised loss without importance weighting

New Design (Tool-Integrated GRPO with HisPO)

• Direct integration with Lean4 compiler and syntax checkers
• Curriculum progression: single-turn → multi-turn tool sequences
• Hierarchical importance sampling addressing train-inference discrepancies
• Legality detection preventing reward hacking

Motivation & Problem Statement

Formal reasoning requires multi-turn interaction with verification tools (compilers, type checkers) to generate correct proofs. Standard supervised learning fails because:

1. Distribution mismatch: Single-turn generation doesn't match multi-turn interaction patterns
2. Long-horizon instability: Standard importance sampling destabilizes over long proof sequences
3. Train-inference gap: Greedy decoding diverges from training distribution in low-confidence regions

HisPO directly addresses these through geometric averaging of token-level importance ratios.

The Modification

Hierarchical Importance Sampling Policy Optimization (HisPO)

HisPO decomposes importance sampling into two controlled components:

# Standard importance sampling (problematic for long sequences):
# IS_standard = p_new(τ) / p_old(τ) = Π(p_new(a_t) / p_old(a_t))
# Problem: multiplication of ratios across long sequences causes exponential variance

# HisPO solution: Sequence-level and token-level masking
# Step 1: Token-level ratio computation
token_ratios = [p_new(a_t) / p_old(a_t) for t in range(seq_len)]

# Step 2: Detect train-inference discrepancies
# Geometric average (more stable than arithmetic)
geometric_ratio = exp(mean(log(token_ratios)))

# Step 3: Sequence-level masking
# Remove sequences where geometric_ratio indicates large discrepancy
if geometric_ratio < threshold  geometric_ratio > inv_threshold:
    mask_sequence =   
:
    mask_sequence = 


token_mask = [
      |log(token_ratio)| > token_threshold  
     token_ratio  token_ratios
]



staleness_mask = compute_staleness(token_timestamps, policy_update_time)


loss = (mask_sequence * token_mask * staleness_mask) * policy_gradient(τ)