メニュー
Improve RL training efficiency by 2.5× using angle concentration between token hidden states as a cost-effective data scheduling signal, selecting high-gradient samples dynamically.
デフォルトでは、最初にソースを確認する Prompt が選択されています。直接コマンドに切り替えるか、ローカルコピーをダウンロードすることもできます。
インストールを決める前に、SKILL.md と SkillsMP に表示されている付属ファイルをお読みください。
Codex または Claude でインストール この Prompt をコピーして Codex、Claude、または他のアシスタントに貼り付けると、Skill ページを確認してインストールできます。
直接コマンドでは確認用 Prompt が省略されます。実行前にソースを確認してください。
npx skills add https://github.com/ADu2021/skillXiv --skill gain-rl-angle-concentrationコマンドは1行のまま表示されます。コピー前に横へスクロールして全体を確認してください。
ローカルで確認しますか？SkillsMP が現在取得できるファイルをダウンロードできます。
Convert arXiv papers into ready-to-use agent skills using category-aware extraction. First classifies the paper into one or more of 11 research categories, then applies a specialized extraction pipeline for each category — because different types of papers produce different types of usable knowledge. A single paper can yield multiple skills if it spans categories. Use this skill whenever the user wants to turn a paper into a skill, extract practical techniques from research, build a skill library from papers, convert arXiv papers into reusable agent instructions, or batch-process multiple papers into skills. Also trigger when someone asks about extracting actionable knowledge from papers, making research practical for LLM agents, or systematically converting academic contributions into structured agent capabilities.
Establish regret bounds for behavior cloning with discretized actions combining statistical error and quantization error terms. Prove smoothness requirements for safe quantizer design, show that learning-based quantizers fail these requirements, and propose model-based augmentation to reduce error dependence from H² to H.
Dynamically allocate LoRA ranks per-layer during fine-tuning instead of using fixed uniform ranks. Learn optimal rank for each layer and subject via variational framework with discretized exponential distribution, reducing memory footprint while maintaining fidelity and text-alignment.
SOC 職業分類に基づく
SKILL.md を表示中
| name | gain-rl-angle-concentration |
| title | Angles Don't Lie: Unlocking Training-Efficient RL Through the Model's Own Signals |
| version | 0.0.2 |
| engine | skillxiv-v0.0.2-claude-opus-4.6 |
| license | MIT |
| url | https://arxiv.org/abs/2506.02281 |
| keywords | ["reinforcement-learning","data-selection","angle-concentration","efficiency","gradient-signals"] |
| description | Improve RL training efficiency by 2.5× using angle concentration between token hidden states as a cost-effective data scheduling signal, selecting high-gradient samples dynamically. |
GAIN-RL identifies a powerful cost-free signal hidden in every LLM: the geometric relationship between token hidden states (cosine similarity / "angle concentration"). The key discovery: models with concentrated angles between tokens exhibit larger gradient norms, indicating higher learning capacity for that example. Rather than expensive external metrics or heuristics, the framework uses intrinsic model signals to schedule training data dynamically. This enables 2.5× training speedup using only 50% of data while surpassing full-dataset performance, through three components: angle-based data ranking, Gaussian probability sampling, and dynamic curriculum adjustment.
The approach is remarkably simple yet effective: higher angle concentration → higher learning potential → prioritize early in training.
def compute_angle_concentration(model, example, device='cuda'):
"""
Calculate intra-segment and inter-segment angle concentration.
Angle concentration = cosine similarity between consecutive token hidden states.
Higher concentration indicates higher learning potential.
"""
# Forward pass to get hidden states
with torch.no_grad():
outputs = model(
example['input_ids'].to(device),
output_hidden_states=True
)
hidden_states = outputs.hidden_states # [num_layers, seq_len, hidden_dim]
angle_concentration = {}
# Compute intra-segment angles (within reasoning segment)
intra_angles = []
layer_idx, layer_states (hidden_states):
i ((layer_states) - ):
h_i = layer_states[i]
h_next = layer_states[i + ]
h_i_norm = h_i / (torch.norm(h_i) + )
h_next_norm = h_next / (torch.norm(h_next) + )
cosine_sim = torch.dot(h_i_norm, h_next_norm).item()
intra_angles.append(cosine_sim)
angle_concentration[] = np.mean(intra_angles)
answer_boundary = example.get(, (hidden_states[-]) // )
inter_angles = []
layer_idx, layer_states (hidden_states):
answer_boundary < (layer_states):
reasoning_token = layer_states[answer_boundary - ]
answer_token = layer_states[answer_boundary]
reasoning_norm = reasoning_token / (torch.norm(reasoning_token) + )
answer_norm = answer_token / (torch.norm(answer_token) + )
cosine_sim = torch.dot(reasoning_norm, answer_norm).item()
inter_angles.append(cosine_sim)
angle_concentration[] = np.mean(inter_angles) inter_angles
combined_signal = * angle_concentration[] + * angle_concentration[]
{
: angle_concentration[],
: angle_concentration[],
: combined_signal
}
def rank_dataset_by_angles(model, dataset, batch_size=32, device='cuda'):
"""
Rank entire dataset by angle concentration in single offline pass.
Expensive computation only done once, reused across epochs.
"""
ranked_data = []
print("Computing angle concentration for all examples...")
for batch_idx in tqdm(range(0, len(dataset), batch_size)):
batch = dataset[batch_idx : batch_idx + batch_size]
for example in batch:
try:
angles = compute_angle_concentration(model, example, device)
combined_score = angles['combined']
ranked_data.append({
'example_idx': len(ranked_data),
'example': example,
'angle_score': combined_score,
'angle_intra': angles['intra'],
'angle_inter': angles['inter']
})
except Exception as e:
# Skip examples that cause errors
print(f"Skipping example due to error: {e}")
continue
# Sort by combined angle score (descending)
ranked_data.sort(key=lambda x: x['angle_score'], reverse=True)
print(f"Ranked {len(ranked_data)} examples")
print(f"Top 10 angle scores: {[d['angle_score'] for d in ranked_data[:10]]}")
print(f"Bottom 10 angle scores: {[d['angle_score'] for d in ranked_data[-10:]]}")
return ranked_data
def gaussian_curriculum_sampling(ranked_data, epoch, total_epochs,
batch_size=32, num_samples=1000):
"""
Sample from dataset using Gaussian distribution over rankings.
Start with high-concentration examples, gradually introduce harder examples.
"""
n = len(ranked_data)
# Mean position starts at 0 (easiest, highest concentration)
# Progressively shifts right (harder, lower concentration) over epochs
mu_t = (epoch / total_epochs) * (n / 2) # Shift from 0 to n/2
# Standard deviation controls spread (how many examples to consider)
sigma_t = n / (2 * total_epochs) * (1 + epoch / total_epochs)
# Gaussian probability for each ranked position
positions = np.arange(n)
probabilities = (1 / (sigma_t * np.sqrt(2 * np.pi))) * np.exp(
-0.5 * ((positions - mu_t) / sigma_t) ** 2
)
# Normalize to valid probability distribution
probabilities = probabilities / np.sum(probabilities)
# Sample batch indices according to Gaussian
sampled_indices = np.random.choice(
n, size=min(num_samples, batch_size), p=probabilities, replace=False
)
sampled_batch = [ranked_data[i]['example'] for i in sampled_indices]
return sampled_batch
def update_curriculum_mean(current_accuracy, baseline_accuracy,
previous_mu, n_samples, learning_rate=0.5):
"""
Dynamically adjust curriculum difficulty based on model performance.
If accuracy improves, move to harder examples faster.
"""
# Accuracy improvement metric
acc_improvement = current_accuracy - baseline_accuracy
# Tanh-based smooth update rule
mu_update_1 = (n_samples / 2) * np.tanh(0.1 * acc_improvement) + (n_samples / 2) * np.tanh(0.5)
# Also incorporate angle signal directly (easier examples have higher scores)
angle_signal_update = (n_samples / 2) * 0.3 # Mild contribution
# Combined update with momentum
new_mu = previous_mu + learning_rate * (mu_update_1 + angle_signal_update)
# Clamp to valid range [0, n_samples]
new_mu = np.clip(new_mu, 0, n_samples)
return new_mu
def train_with_angle_scheduling(model, ranked_dataset, num_epochs=10,
batch_size=32, learning_rate=1e-5):
"""
Complete training loop using angle concentration for data scheduling.
Achieves 2.5× speedup using 50% of data.
"""
optimizer = torch.optim.AdamW(model.parameters(), lr=learning_rate)
baseline_accuracy = evaluate_model(model, ranked_dataset[:100])
mu_t = 0.0 # Start with easiest examples
for epoch in range(num_epochs):
# Sample batch using Gaussian curriculum
batch = gaussian_curriculum_sampling(
ranked_dataset, epoch, num_epochs,
batch_size=batch_size,
num_samples=len(ranked_dataset) // 2 # Use only 50% of data
)
# Training step
loss = 0.0
for example in batch:
output = model(example['input_ids'])
batch_loss = compute_loss(output, example)
loss += batch_loss / len(batch)
loss.backward()
optimizer.step()
optimizer.zero_grad()
# Periodically evaluate and update curriculum
if epoch % 2 == 0:
current_accuracy = evaluate_model(model, ranked_dataset)
mu_t = update_curriculum_mean(
current_accuracy, baseline_accuracy, mu_t, len(ranked_dataset)
)
print(f"Epoch {epoch}: Accuracy={current_accuracy:.1%}, Curriculum_mean={mu_t:.1f}")
return model
When to Apply:
Implementation Steps:
Performance Expectations:
Configuration Tuning:
Key Hyperparameters:
ANGLE_CONFIG = {
'initial_mu': 0.0, # Start with easiest examples
'final_mu_fraction': 0.5, # End at middle difficulty
'sigma_factor': 2.0, # Gaussian width control
'curriculum_update_freq': 2, # Update every N epochs
'learning_rate_mu': 0.5, # Mean shift learning rate
'data_fraction': 0.5 # Use 50% of training data
}
Monitoring Metrics:
Common Issues:
Implemented on Qwen-2.5-7B and Qwen-3-4B for RL fine-tuning on mathematical (MATH-500, AIME24) and coding (HumanEval) tasks. Demonstrates 2.5× training speedup using only 50% of training data while surpassing full-dataset performance. Works across model architectures—angle concentration is a model-agnostic learning signal. Training uses 4-16 A100 GPUs with standard RL algorithms (GRPO, DPO).