perk-long-context-test-time-learning

name	perk-long-context-test-time-learning
title	PERK: Long-Context Reasoning as Parameter-Efficient Test-Time Learning
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2507.06415
keywords	["Long-Context","Test-Time Learning","LoRA","Gradient-Based Adaptation","Parameter Efficiency"]
description	Enable language models to reason over extremely long contexts (128K tokens) by encoding context into lightweight LoRA adapters during test time, achieving 20% performance improvements without full model retraining.

PERK: Long-Context Reasoning via Test-Time Parameter Adaptation

Standard approaches to long-context reasoning require expensive full model fine-tuning or suffer from position bias when information appears at different locations in lengthy documents. PERK reframes the problem: instead of treating a long context as a sequential stream, compress it into parameter updates that capture its key information. This enables the model to reason over massive contexts using only adapter parameters, not full weights.

The core insight is that context encoding and reasoning can be decoupled through bilevel optimization. An inner loop encodes context segments into a LoRA adapter, while an outer loop learns how to reason over the encoded information. Truncated gradient unrolling makes this computationally tractable on a single GPU.

Core Concept

PERK treats long-context reasoning as a meta-learning problem solved at test time. Rather than storing contexts in memory, it learns to represent contexts as parameter updates to a lightweight adapter. This approach offers three advantages: it scales to very long sequences, reduces memory overhead dramatically, and avoids position bias by processing context as permutation-invariant batches.

The method splits a long context into chunks, encodes each chunk via gradient descent into a LoRA adapter (inner loop), then optimizes how well the adapted model answers queries over the encoded context (outer loop). Dynamic learning rates per layer and per step further refine the adaptation process.

Architecture Overview

• LoRA Adapter: Lightweight parameter updates with rank 256 (or tunable lower), applied to transformer layer projections
• Inner Loop: Encodes context segments using causal language modeling; gradient descent compresses context into adapter parameters
• Outer Loop: Learns meta-parameters and per-layer learning rates to optimize reasoning performance
• Truncated Gradient Unrolling: Stores computational graphs only for final T steps (typically 2-3), dramatically reducing memory from O(T_inner) to O(T_truncated)
• Batch Processing: Context chunks processed in permutation-invariant batches with explicit indexing to preserve order information

Implementation

Step 1: Prepare Context and Query

Split the long document into overlapping chunks of approximately 256 tokens. Initialize a LoRA adapter with rank 256 (or lower for efficiency). This example shows preparing context for a retrieval task:

import torch
import torch.nn.functional as F
from peft  get_peft_model, LoraConfig


lora_config = LoraConfig(
    r=,
    lora_alpha=,
    target_modules=[, ],
    lora_dropout=,
    bias=,
    task_type=
)
peft_model = get_peft_model(model, lora_config)


context = 
chunk_size = 
chunks = [context[i:i+chunk_size]
           i  (, (context), chunk_size)]