learning-scientific-taste-with-rl

name	learning-scientific-taste-with-rl
title	AI Can Learn Scientific Taste
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2603.14473
keywords	["Scientific Taste","Reinforcement Learning","Community Feedback","Citation Prediction","Research Quality Assessment"]
description	Learn to predict and generate high-impact research ideas by training models on community feedback signals. Apply reinforcement learning to align research generation with citation-based indicators of scientific impact.

Learning Scientific Taste with RL: Improving AI's Judgment of Research Quality

Training AI systems to generate research ideas is table stakes, but teaching them to judge which ideas matter remains underexplored. This skill demonstrates how to build systems that learn scientific taste—the ability to recognize and propose high-impact research—by leveraging large-scale community feedback signals rather than expensive expert annotation.

The core insight is elegantly simple: citation patterns represent implicit community consensus about research value. By modeling this signal as a preference learning problem, you can train models to develop judgment that generalizes to unseen domains, future years, and even peer-review preferences.

Core Concept

Scientific taste learning operates as a two-stage preference modeling system:

1. Judge Training — Model research quality using historical citation patterns as weak supervision
2. Thinker Fine-tuning — Use the trained judge as a reward signal to generate higher-impact ideas via RL

The key innovation is treating this as preference alignment rather than classification. Instead of binary "good/bad" labels, the system learns a continuous preference signal from comparative citation evidence.

Architecture Overview

• Citation Dataset Construction — Pair papers published in the same year/field, where higher-citation papers represent preferred outcomes
• Judge Model — Transformer-based preference model trained on 700K+ paper pairs to distinguish high-citation from low-citation research
• Reward Signal — Judge outputs logit scores as continuous rewards for RL training
• Thinker Model — LLM fine-tuned via policy gradient to maximize judge-assigned rewards
• Evaluation Framework — Test generalization across unseen years, domains, and peer-review datasets

Implementation Steps

The first step is constructing your citation dataset. You need temporal separation between training and evaluation to measure generalization.

# Pair papers by publication date and field for contrastive learning
import numpy as np
from collections import defaultdict

def construct_citation_pairs(papers, year_cutoff=2020):
    """Create high/low citation paper pairs for preference learning."""
    pairs = []
    papers_by_year_field = defaultdict()

    
     paper  papers:
        key = (paper[], paper[])
        papers_by_year_field[key].append(paper)

    
     (year, field), candidates  papers_by_year_field.items():
         year > year_cutoff  (candidates) < :
            

        high_citation = (candidates,
                               key= x: x[],
                               reverse=)[:(candidates)//]
        low_citation = (candidates,
                              key= x: x[])[:(candidates)//]

         high_paper  high_citation:
             low_paper  low_citation:
                pairs.append({
                    : high_paper,
                    : low_paper,
                    : high_paper[] - low_paper[]
                })

     pairs