メニュー
Build community-driven evaluation platforms for scientific tasks using pairwise model comparisons and human voting. Assess foundation models on literature-grounded reasoning without automated metrics.
デフォルトでは、最初にソースを確認する Prompt が選択されています。直接コマンドに切り替えるか、ローカルコピーをダウンロードすることもできます。
インストールを決める前に、SKILL.md と SkillsMP に表示されている付属ファイルをお読みください。
Codex または Claude でインストール この Prompt をコピーして Codex、Claude、または他のアシスタントに貼り付けると、Skill ページを確認してインストールできます。
直接コマンドでは確認用 Prompt が省略されます。実行前にソースを確認してください。
npx skills add https://github.com/ADu2021/skillXiv --skill sciarena-evaluation-platformコマンドは1行のまま表示されます。コピー前に横へスクロールして全体を確認してください。
ローカルで確認しますか？SkillsMP が現在取得できるファイルをダウンロードできます。
SKILL.md を表示中
SOC 職業分類に基づく
| name | sciarena-evaluation-platform |
| title | SciArena: An Open Evaluation Platform for Foundation Models in Scientific Literature |
| version | 0.0.2 |
| engine | skillxiv-v0.0.2-claude-opus-4.6 |
| license | MIT |
| url | https://arxiv.org/abs/2507.01001 |
| keywords | ["Evaluation","Scientific Literature","Community Voting","Meta-evaluation","Foundation Models"] |
| description | Build community-driven evaluation platforms for scientific tasks using pairwise model comparisons and human voting. Assess foundation models on literature-grounded reasoning without automated metrics. |
Evaluating language models on scientific literature tasks is challenging because correct answers aren't verifiable through simple string matching or automated metrics. A paper's technical contribution is nuanced; a model's explanation of a methodology can be correct in multiple ways; reasoning about research significance requires human judgment. Automated evaluation metrics (BLEU, ROUGE) fail because they don't capture semantic correctness of scientific reasoning. The traditional solution—hiring annotators to score outputs—is expensive and doesn't scale to many models.
SciArena solves this through community voting: researchers evaluate model outputs on scientific tasks by comparing two models side-by-side and voting for which response is better. This crowdsourced approach leverages existing expertise (the research community) and scales to 47 models. The platform also provides SciArena-Eval, a meta-evaluation benchmark measuring how well automated systems can predict which model researchers would prefer.
The key insight is that pairwise comparisons are easier than absolute scoring. Instead of asking "rate this explanation on a scale of 1-5," ask "which explanation better explains the research methodology?" Humans naturally prefer comparisons because they can identify relative strengths. By collecting many pairwise votes (20,000+ across the platform), you build a leaderboard that reflects community consensus.
The approach has three components:
The evaluation platform consists of:
Step 1: Prepare task dataset with scientific questions
Create a pool of questions grounded in scientific literature that require deep understanding.
def create_scientific_task_dataset(papers, num_questions_per_paper=3):
"""
Generate evaluation questions from research papers.
Each question requires understanding the paper's methodology, results, or contributions.
"""
tasks = []
for paper in papers:
# Extract key aspects from paper
abstract = paper['abstract']
methods = paper['methods']
results = paper['results']
title = paper['title']
# Generate different types of questions
questions = [
{
'type': 'methodology',
'question': f"Explain the key methodology in this paper: {title}. "
f"What are the main technical contributions?",
'context': methods,
'paper_id': paper['id']
},
{
'type': 'interpretation',
'question': f"What do the experimental results tell us about {title}? "
f"What are the key findings?",
'context': results,
'paper_id': paper['id']
},
{
'type': 'significance',
'question': f"Why is {title} important to the field? "
f"How does it advance our understanding?",
'context': abstract,
'paper_id': paper['id']
}
]
for q in questions:
tasks.append(q)
return tasks
Step 2: Implement the pairwise voting interface
Structure the evaluation to collect human preferences between model pairs.
class PairwiseVotingTask:
"""
Represents a single pairwise comparison task.
One research question, two model outputs, human vote.
"""
def __init__(self, task_id, question, model_a_output, model_b_output):
self.task_id = task_id
self.question = question
self.model_a_output = model_a_output
self.model_b_output = model_b_output
self.vote = None # 'a', 'b', or 'tie'
self.voter_expertise = None
self.confidence = None
def to_dict(self):
"""Format for JSON serialization or database storage."""
return {
'task_id': self.task_id,
'question': self.question,
'model_a': self.model_a_output,
'model_b': self.model_b_output,
'vote': self.vote,
'expertise': self.voter_expertise,
'confidence': self.confidence
}
def generate_pairwise_tasks(models, task_dataset, pairs_per_task=3):
"""
Create pairwise comparison tasks from all model outputs.
Each question is evaluated by multiple model pairs for statistical robustness.
"""
# First, generate outputs from all models
model_outputs = {}
for model in models:
model_outputs[model['name']] = []
for task in task_dataset:
output = model['generate_fn'](task['question'], task['context'])
model_outputs[model['name']].append(output)
# Now create pairwise tasks
pairwise_tasks = []
for task_idx, task in enumerate(task_dataset):
# Select pairs of models to compare (round-robin or random)
model_names = list(model_outputs.keys())
pairs = []
# Each task compared by multiple pairs for robustness
for i in range(pairs_per_task):
# Random pair selection
pair = random.sample(model_names, 2)
pairs.append(pair)
for model_a, model_b in pairs:
pairwise_task = PairwiseVotingTask(
task_id=f"{task_idx}_{model_a}_vs_{model_b}",
question=task['question'],
model_a_output=model_outputs[model_a][task_idx],
model_b_output=model_outputs[model_b][task_idx]
)
pairwise_tasks.append(pairwise_task)
return pairwise_tasks
def collect_vote(task, voter_id, vote, confidence=0.5):
"""
Record a vote with optional confidence score.
Confidence helps weight reliable voters more heavily.
"""
task.vote = vote # 'a', 'b', or 'tie'
task.voter_expertise = get_voter_expertise(voter_id)
task.confidence = confidence # 0.0 to 1.0
# Store in database
store_vote_in_database(task.to_dict(), voter_id)
Step 3: Aggregate votes into model rankings
Convert pairwise votes into a leaderboard using ranking algorithms.
def aggregate_votes_bradley_terry(pairwise_votes, models, max_iterations=100):
"""
Use Bradley-Terry model to convert pairwise comparisons into scalar scores.
This is the standard method for ranking from pairwise preferences.
"""
# Initialize scores
scores = {model: 1.0 for model in models}
for iteration in range(max_iterations):
# Update each model's score based on votes
new_scores = {}
for model in models:
# Count wins and losses
total_comparisons = 0
weighted_wins = 0
for vote in pairwise_votes:
# Did this model appear in this comparison?
if vote['model_a'] == model:
opponent = vote['model_b']
is_winner = vote['vote'] == 'a'
elif vote['model_b'] == model:
opponent = vote['model_a']
is_winner = vote['vote'] == 'b'
else:
continue
# Weight by voter confidence and expertise
weight = vote['confidence'] * vote['expertise_weight']
total_comparisons += weight
if is_winner:
weighted_wins += weight
# Bradley-Terry: account for opponent strength
weighted_wins += weight * scores[opponent] / (scores[model] + scores[opponent])
else:
# Loss: decrease proportionally to opponent strength
weighted_wins -= weight * scores[opponent] / (scores[model] + scores[opponent])
# Update score: empirical win rate adjusted for opponent strength
if total_comparisons > 0:
new_scores[model] = scores[model] * (1 + 0.01 * weighted_wins / total_comparisons)
else:
new_scores[model] = scores[model]
# Check for convergence
score_change = sum(abs(new_scores[m] - scores[m]) for m in models) / len(models)
scores = new_scores
if score_change < 0.001:
break
# Normalize to 0-100 scale
min_score = min(scores.values())
max_score = max(scores.values())
normalized_scores = {
model: 100 * (scores[model] - min_score) / (max_score - min_score)
for model in models
}
return normalized_scores
def compute_leaderboard_with_confidence(votes, models):
"""
Compute leaderboard with 95% confidence intervals using bootstrap.
This shows ranking uncertainty.
"""
# Bootstrap: resample votes with replacement
bootstrap_scores = []
for bootstrap_iteration in range(1000):
resampled_votes = [random.choice(votes) for _ in range(len(votes))]
scores = aggregate_votes_bradley_terry(resampled_votes, models)
bootstrap_scores.append(scores)
# Compute confidence intervals
leaderboard = []
for model in models:
model_scores = [scores[model] for scores in bootstrap_scores]
mean_score = np.mean(model_scores)
ci_low = np.percentile(model_scores, 2.5)
ci_high = np.percentile(model_scores, 97.5)
leaderboard.append({
'model': model,
'score': mean_score,
'ci_low': ci_low,
'ci_high': ci_high
})
# Sort by score
return sorted(leaderboard, key=lambda x: x['score'], reverse=True)
Step 4: Create meta-evaluation benchmark
Use human votes as ground truth to evaluate automated metrics.
def evaluate_automated_metric(metric_function, human_votes, models):
"""
Test how well an automated metric predicts human preferences.
Compute correlation between metric rankings and human consensus.
"""
# Get human consensus ranking
human_scores = aggregate_votes_bradley_terry(human_votes, models)
human_ranking = sorted(models, key=lambda m: human_scores[m], reverse=True)
# Get automated metric ranking
metric_scores = {}
for model in models:
score = metric_function(model)
metric_scores[model] = score
metric_ranking = sorted(models, key=lambda m: metric_scores[m], reverse=True)
# Compute Spearman correlation between rankings
human_ranks = {model: i for i, model in enumerate(human_ranking)}
metric_ranks = {model: i for i, model in enumerate(metric_ranking)}
correlation = spearmanr(
[human_ranks[m] for m in models],
[metric_ranks[m] for m in models]
)[0]
# Also compute accuracy: how often metric predicts the winner?
metric_correct = 0
total_comparisons = 0
for vote in human_votes:
model_a, model_b = vote['model_a'], vote['model_b']
human_preference = vote['vote']
metric_preference = 'a' if metric_scores[model_a] > metric_scores[model_b] else 'b'
if metric_preference == human_preference:
metric_correct += 1
total_comparisons += 1
metric_accuracy = metric_correct / total_comparisons
return {
'spearman_correlation': correlation,
'prediction_accuracy': metric_accuracy,
'is_reliable': correlation > 0.75
}
| Aspect | Recommended Value | Notes |
|---|---|---|
| Votes per task | 3-5 | Balances coverage and robustness |
| Total tasks | 100-500 | 100 minimum for statistical significance |
| Total votes collected | 5,000-20,000 | More votes = more stable rankings |
| Expert voter weight | 1.5-2.0x | Upweight domain experts |
| Confidence threshold | 0.3+ | Filter out uncertain votes |
When to use community-driven evaluation:
When NOT to use community voting:
Common pitfalls:
SciArena: An Open Evaluation Platform for Foundation Models in Scientific Literature https://arxiv.org/abs/2507.01001