| name | perk-long-context-test-time-learning |
| title | PERK: Long-Context Reasoning as Parameter-Efficient Test-Time Learning |
| version | 0.0.2 |
| engine | skillxiv-v0.0.2-claude-opus-4.6 |
| license | MIT |
| url | https://arxiv.org/abs/2507.06415 |
| keywords | ["Long-Context","Test-Time Learning","LoRA","Gradient-Based Adaptation","Parameter Efficiency"] |
| description | Enable language models to reason over extremely long contexts (128K tokens) by encoding context into lightweight LoRA adapters during test time, achieving 20% performance improvements without full model retraining. |
PERK: Long-Context Reasoning via Test-Time Parameter Adaptation
Standard approaches to long-context reasoning require expensive full model fine-tuning or suffer from position bias when information appears at different locations in lengthy documents. PERK reframes the problem: instead of treating a long context as a sequential stream, compress it into parameter updates that capture its key information. This enables the model to reason over massive contexts using only adapter parameters, not full weights.
The core insight is that context encoding and reasoning can be decoupled through bilevel optimization. An inner loop encodes context segments into a LoRA adapter, while an outer loop learns how to reason over the encoded information. Truncated gradient unrolling makes this computationally tractable on a single GPU.
Core Concept
PERK treats long-context reasoning as a meta-learning problem solved at test time. Rather than storing contexts in memory, it learns to represent contexts as parameter updates to a lightweight adapter. This approach offers three advantages: it scales to very long sequences, reduces memory overhead dramatically, and avoids position bias by processing context as permutation-invariant batches.
The method splits a long context into chunks, encodes each chunk via gradient descent into a LoRA adapter (inner loop), then optimizes how well the adapted model answers queries over the encoded context (outer loop). Dynamic learning rates per layer and per step further refine the adaptation process.
Architecture Overview
- LoRA Adapter: Lightweight parameter updates with rank 256 (or tunable lower), applied to transformer layer projections
- Inner Loop: Encodes context segments using causal language modeling; gradient descent compresses context into adapter parameters
- Outer Loop: Learns meta-parameters and per-layer learning rates to optimize reasoning performance
- Truncated Gradient Unrolling: Stores computational graphs only for final T steps (typically 2-3), dramatically reducing memory from O(T_inner) to O(T_truncated)
- Batch Processing: Context chunks processed in permutation-invariant batches with explicit indexing to preserve order information
Implementation
Step 1: Prepare Context and Query
Split the long document into overlapping chunks of approximately 256 tokens. Initialize a LoRA adapter with rank 256 (or lower for efficiency). This example shows preparing context for a retrieval task:
import torch
import torch.nn.functional as F
from peft get_peft_model, LoraConfig
lora_config = LoraConfig(
r=,
lora_alpha=,
target_modules=[, ],
lora_dropout=,
bias=,
task_type=
)
peft_model = get_peft_model(model, lora_config)
context =
chunk_size =
chunks = [context[i:i+chunk_size]
i (, (context), chunk_size)]
