메뉴
Improves LLM reasoning by rewarding correct solutions that exhibit rare high-level strategies, preventing exploration collapse and discovering more diverse solution approaches across mathematics, physics, and medical reasoning.
기본적으로 소스를 먼저 확인하는 Prompt가 선택됩니다. 직접 명령으로 전환하거나 로컬 사본을 다운로드할 수도 있습니다.
설치 여부를 결정하기 전에 SKILL.md와 SkillsMP에 표시된 보조 파일을 읽어 보세요.
Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.
직접 명령은 검토 Prompt를 거치지 않습니다. 실행하기 전에 소스를 확인하세요.
npx skills add https://github.com/ADu2021/skillXiv --skill uniqueness-aware-rl명령은 한 줄로 유지됩니다. 복사하기 전에 가로로 스크롤해 전체 내용을 확인하세요.
로컬 사본을 원하시나요? SkillsMP에서 현재 제공할 수 있는 파일을 다운로드하세요.
Convert arXiv papers into ready-to-use agent skills using category-aware extraction. First classifies the paper into one or more of 11 research categories, then applies a specialized extraction pipeline for each category — because different types of papers produce different types of usable knowledge. A single paper can yield multiple skills if it spans categories. Use this skill whenever the user wants to turn a paper into a skill, extract practical techniques from research, build a skill library from papers, convert arXiv papers into reusable agent instructions, or batch-process multiple papers into skills. Also trigger when someone asks about extracting actionable knowledge from papers, making research practical for LLM agents, or systematically converting academic contributions into structured agent capabilities.
Establish regret bounds for behavior cloning with discretized actions combining statistical error and quantization error terms. Prove smoothness requirements for safe quantizer design, show that learning-based quantizers fail these requirements, and propose model-based augmentation to reduce error dependence from H² to H.
Dynamically allocate LoRA ranks per-layer during fine-tuning instead of using fixed uniform ranks. Learn optimal rank for each layer and subject via variational framework with discretized exponential distribution, reducing memory footprint while maintaining fidelity and text-alignment.
SOC 직업 분류 기준
SKILL.md 표시 중
| name | uniqueness-aware-rl |
| title | Rewarding the Rare: Uniqueness-Aware RL for Creative Problem Solving in LLMs |
| version | 0.0.2 |
| engine | skillxiv-v0.0.2-claude-opus-4.6 |
| license | MIT |
| url | https://arxiv.org/abs/2601.08763 |
| keywords | ["exploration","RL","creative-problem-solving","diversity-reward","reasoning"] |
| description | Improves LLM reasoning by rewarding correct solutions that exhibit rare high-level strategies, preventing exploration collapse and discovering more diverse solution approaches across mathematics, physics, and medical reasoning. |
Enhance exploration in LLM reasoning by rewarding correct solutions based on their rarity of approach. Rather than treating all correct solutions equally, provide higher rewards for solutions using novel high-level strategies, preventing models from converging to single solution patterns.
Group solutions by high-level reasoning approach, ignoring surface-level variation.
# Strategy clustering for exploration diversity
class SolutionStrategyClusterer:
def __init__(self, strategy_classifier_model):
self.classifier = strategy_classifier_model
def extract_strategy(self, solution_text, problem):
"""Identify high-level strategy from solution"""
prompt = f"""
Analyze this {problem['domain']} solution:
{solution_text}
What high-level strategy does it use? (e.g., algebraic, graphical, numerical, recursive)
Ignore surface-level variations in expression.
"""
strategy = self.classifier.classify(prompt)
return strategy
def cluster_solutions(self, solutions, problem):
"""Group solutions by strategy"""
strategy_groups = {}
for solution solutions:
strategy = .extract_strategy(solution[], problem)
strategy strategy_groups:
strategy_groups[strategy] = []
strategy_groups[strategy].append(solution)
strategy_groups
():
strategy_counts = {}
s all_strategies:
strategy_counts[s] = strategy_counts.get(s, ) +
frequency = strategy_counts.get(strategy, ) / (all_strategies)
rarity = / (frequency + )
normalized_rarity = / (rarity + )
normalized_rarity
Assign higher rewards to rare but correct solutions.
# Uniqueness-aware reward computation
class UniquenessReward:
def compute_reward(self, solution, is_correct, strategy, all_strategies, base_reward=1.0):
"""Compute reward based on correctness and strategy rarity"""
if not is_correct:
return 0.0
# Get rarity score for this strategy
rarity_score = SolutionStrategyClusterer().get_strategy_rarity(
strategy,
all_strategies
)
# Reward formula: base * (1 + rarity_bonus)
rarity_bonus = 0.5 * rarity_score # Up to 50% bonus for rare strategies
final_reward = base_reward * (1.0 + rarity_bonus)
return final_reward
def batch_compute_rewards(self, solutions, problem):
"""Compute rewards for all solutions in a rollout"""
# Determine correctness for each
correctness = [self.evaluate_correctness(sol, problem) for sol in solutions]
# Extract strategies
clusterer = SolutionStrategyClusterer(self.get_classifier())
all_strategies = [clusterer.extract_strategy(sol["text"], problem) for sol in solutions]
# Compute rewards
rewards = []
for solution, is_correct, strategy in zip(solutions, correctness, all_strategies):
reward = self.compute_reward(
solution,
is_correct,
strategy,
all_strategies
)
rewards.append(reward)
return rewards
def evaluate_correctness(self, solution, problem):
"""Check if solution is correct"""
# Domain-specific evaluation
if problem["domain"] == "math":
return self.verify_math_solution(solution, problem)
elif problem["domain"] == "physics":
return self.verify_physics_solution(solution, problem)
elif problem["domain"] == "medical":
return self.verify_medical_diagnosis(solution, problem)
return False
def get_classifier(self):
"""Initialize strategy classifier model"""
# Would load actual model in practice
pass
Analyze rollout diversity and re-weight samples.
# Rollout analysis for diversity
class RolloutAnalyzer:
def analyze_rollout(self, solutions, problem):
"""Assess diversity and correctness of rollout"""
clusterer = SolutionStrategyClusterer(self.get_classifier())
# Cluster solutions by strategy
strategy_groups = clusterer.cluster_solutions(solutions, problem)
# Compute diversity metrics
num_strategies = len(strategy_groups)
total_correct = sum(
1 for sol in solutions
if clusterer.extract_strategy(sol, problem) and sol["is_correct"]
)
diversity_score = num_strategies / len(solutions)
correctness_score = total_correct / len(solutions)
return {
"diversity_score": diversity_score,
"correctness_score": correctness_score,
"num_strategies": num_strategies,
"strategy_distribution": {
strategy: len(sols) for strategy, sols in strategy_groups.items()
}
}
def should_reweight_rollout(self, analysis):
"""Determine if rollout needs re-weighting"""
# Re-weight if collapse detected
if analysis["diversity_score"] < 0.5:
return True
# Don't re-weight if good diversity
return False
def reweight_samples(self, solutions, analysis):
"""Re-weight solutions to encourage diversity"""
reweighted = []
clusterer = SolutionStrategyClusterer(self.get_classifier())
for solution in solutions:
strategy = clusterer.extract_strategy(solution, None)
# Higher weight for underrepresented strategies
base_weight = 1.0 / analysis["strategy_distribution"].get(strategy, 1)
# Multiply by correctness score
weight = base_weight * (1.0 if solution["is_correct"] else 0.5)
reweighted.append({
"solution": solution,
"weight": weight
})
# Normalize weights
total_weight = sum(w["weight"] for w in reweighted)
for item in reweighted:
item["weight"] /= total_weight
return reweighted
Integrate uniqueness rewards into RL training.
# RL training with uniqueness rewards
class UniquenessRL:
def __init__(self, policy_model):
self.policy = policy_model
self.exploration_history = []
def train_step(self, problem, num_rollouts=10):
"""Training step with uniqueness-aware rewards"""
all_solutions = []
all_rewards = []
# Generate multiple solutions
for _ in range(num_rollouts):
solution = self.policy.generate(problem)
all_solutions.append(solution)
# Compute uniqueness-aware rewards
reward_computer = UniquenessReward()
rewards = reward_computer.batch_compute_rewards(all_solutions, problem)
all_rewards.extend(rewards)
# Analyze rollout diversity
analyzer = RolloutAnalyzer()
analysis = analyzer.analyze_rollout(all_solutions, problem)
self.exploration_history.append(analysis)
# If collapse detected, apply re-weighting
if analyzer.should_reweight_rollout(analysis):
solutions = analyzer.reweight_samples(all_solutions, analysis)
rewards = [item["weight"] * reward for item, reward in zip(solutions, all_rewards)]
# Policy gradient update
total_loss = 0.0
for solution, reward in zip(all_solutions, all_rewards):
log_prob = self.policy.get_log_prob(solution)
loss = -log_prob * reward
total_loss += loss
self.policy.backward(total_loss / len(all_solutions))
self.policy.optimize()
return {
"loss": (total_loss / len(all_solutions)).item(),
"diversity": analysis["diversity_score"],
"avg_reward": sum(all_rewards) / len(all_rewards)
}
def evaluate_exploration(self, test_problems, num_rollouts=10):
"""Measure exploration quality across test set"""
exploration_metrics = {}
for problem in test_problems:
solutions = [self.policy.generate(problem) for _ in range(num_rollouts)]
analyzer = RolloutAnalyzer()
analysis = analyzer.analyze_rollout(solutions, problem)
exploration_metrics[problem["id"]] = analysis
# Aggregate metrics
avg_diversity = np.mean([a["diversity_score"] for a in exploration_metrics.values()])
avg_correctness = np.mean([a["correctness_score"] for a in exploration_metrics.values()])
return {
"avg_diversity": avg_diversity,
"avg_correctness": avg_correctness,
"per_problem": exploration_metrics
}