| name | research-idea-benchmark |
| description | This skill should be used when the user asks to "evaluate my research idea", "benchmark my research", "score my paper idea", "assess research quality", "check if my idea is publishable", "review my research design", or wants to evaluate a research idea against top journal standards. Provides a systematic 7-dimension rubric distilled from patterns in JFQA, JCF, Review of Finance, and leading finance/economics journals. |
| version | 0.1.0 |
Research Idea Benchmark
Evaluate research ideas against empirical standards distilled from 50+ papers published in top finance and economics journals (JFQA, JCF, Review of Finance, and leading ESG/sustainability venues).
When to Use
- Evaluating a new research idea before committing resources
- Assessing whether a research design is strong enough for a target venue
- Identifying the weakest dimension of a research proposal to strengthen
- Comparing multiple research directions to prioritize
Evaluation Workflow
Step 1: Gather the Research Idea
Request the user to provide their research idea in any format: a paragraph description, an abstract draft, a structured outline, or a verbal explanation. At minimum, extract:
- Research question: What is being asked?
- Proposed method: How will it be investigated?
- Target venue: Which journal or conference tier?
- Data availability: What data exists or needs to be collected?
If any of these are missing, ask targeted questions to elicit them before proceeding.
Step 2: Score Across 7 Dimensions
Evaluate the idea against each dimension using the detailed rubric in references/evaluation-rubric.md. The 7 dimensions and their weights:
| # | Dimension | Weight | What It Measures |
|---|
| 1 | Research Question Quality | 20% | Precision, gap significance, testability |
| 2 | Theoretical Foundation | 15% | Theory grounding, competing hypotheses |
| 3 | Identification Strategy | 25% | Causal inference credibility |
| 4 | Data & Variable Construction | 15% | Novelty, quality, proxy validity |
| 5 | Robustness Architecture | 10% | Alternative explanations, sensitivity |
| 6 | Contribution Clarity | 10% | Novelty, multi-dimensional impact |
| 7 | Narrative Quality | 5% | Logical flow, persuasiveness |
Critical rule: Identification Strategy (Dimension 3) is the single most important dimension for empirical finance journals. A score below 4 on this dimension is a near-certain rejection at top-5 finance journals.
Step 3: Generate the Evaluation Report
Produce a structured report with the following sections:
## Research Idea Benchmark Report
### Overview
[1-2 sentence summary of the research idea]
### Target Venue Assessment
[Venue name] | Estimated fit: [Strong/Moderate/Weak]
### Dimension Scores
| Dimension | Score | Key Strength | Key Weakness |
|-----------|-------|-------------|-------------|
| 1. Research Question | X.X/5 | ... | ... |
| 2. Theoretical Foundation | X.X/5 | ... | ... |
| 3. Identification Strategy | X.X/5 | ... | ... |
| 4. Data & Variables | X.X/5 | ... | ... |
| 5. Robustness | X.X/5 | ... | ... |
| 6. Contribution | X.X/5 | ... | ... |
| 7. Narrative | X.X/5 | ... | ... |
**Weighted Total: X.XX / 5.00**
### Referee Red Flags
[List any items from the "Referee Red Flag Checklist" that apply]
### Actionable Improvements
[Rank-ordered list of specific, concrete steps to strengthen the weakest dimensions]
### Comparable Published Papers
[2-3 published papers with similar research designs that succeeded,
with notes on what made them work]
Step 4: Provide Targeted Guidance
Based on the lowest-scoring dimensions, provide specific improvement strategies:
For weak Identification Strategy (< 4):
- Suggest specific natural experiments, policy shocks, or instrumental variables relevant to the topic
- Reference the identification arsenal in
references/evaluation-rubric.md
- Recommend specific econometric techniques
For weak Research Question (< 3.5):
- Help reframe the question as a paradox or puzzle
- Identify the specific gap in existing literature
- Suggest competing hypotheses to pit against each other
For weak Data/Variables (< 3.5):
- Suggest novel data sources or creative proxy constructions
- Reference exemplary proxy innovations from
references/paper-patterns.md
- Recommend validation approaches
For weak Contribution (< 3.5):
- Help position the paper against multiple literature streams
- Identify which contribution types (empirical, methodological, theoretical, policy) are achievable
- Suggest the "contribution statement formula" from
references/paper-patterns.md
Research Design Patterns
Consult references/paper-patterns.md for 7 recurring research design patterns extracted from top journals:
- Black Box Opening — identifying the mechanism in an established relationship
- Resolution Paper — introducing a dimension that resolves contradictory findings
- Policy Shock — exploiting government policy for DiD identification
- Unintended Consequences — documenting surprising spillover effects
- Proxy Innovation — creating a new way to measure the unmeasurable
- Cross-Country Variation — leveraging institutional differences
- Mechanism Decomposition — breaking a main effect into sub-channels
When evaluating an idea, identify which pattern(s) it most closely follows and assess whether the idea fulfills that pattern's requirements.
Scoring Interpretation
| Score Range | Assessment | Action |
|---|
| 4.5 - 5.0 | Exceptional | Proceed to execution |
| 4.0 - 4.4 | Very Strong | Minor refinements, then proceed |
| 3.5 - 3.9 | Strong | Strengthen 1-2 dimensions before proceeding |
| 3.0 - 3.4 | Moderate | Significant redesign in weak areas |
| 2.5 - 2.9 | Weak | Fundamental rethinking needed |
| < 2.5 | Not Ready | Return to ideation stage |
Minimum thresholds for top finance journals:
- Identification Strategy >= 4.0
- Research Question >= 3.5
- No single dimension below 2.5
The "Prior-Changing" Test (from Edmans, RF Editor)
The single most important question for evaluating any research idea:
"Does this paper change the reader's prior?"
If a reader spends hours going through the paper, they should learn something they didn't know or couldn't have guessed. Apply this test ruthlessly:
- If the result can be predicted by chaining two known findings → Convex combination → Reject
- If it adds "just another" item to a known list → Marginal contribution → Reject
- If extending to new country/period without institutional reason → Context extension → Reject
- If documenting only one side of a tradeoff → Incomplete answer → Reject
See references/editor-insights.md for the full taxonomy of rejection patterns with editor quotes.
Additional Resources
Reference Files
references/evaluation-rubric.md — Full 7-dimension rubric with scoring anchors, diagnostic questions, examples from published papers, plus editor rejection taxonomyreferences/paper-patterns.md — Recurring research design patterns, data source catalog, contribution statement formulas, and section-by-section benchmarksreferences/editor-insights.md — Editor decision-making insights from Edmans (1,000 Rejections at RF), Cochrane (Writing Tips), Mitton (Methodological Variation, RFS), and Armstrong et al. (Causality Redux). Covers the 6 deadly contribution sins, IV pitfalls, p-hacking quantification, and exposition standards.
Training Data
The rubric was built from analysis of papers and methodological guides:
- Papers: 50+ papers from JFQA, JCF, Review of Finance, ESG/sustainability venues in
/Users/yw/Desktop/001processing/60_github_mirrors/training_data/topjournal/
- Editorial wisdom: Edmans "Learnings From 1,000 Rejections" (RF editor, 998 rejections analyzed)
- Methodological standards: Mitton "Methodological Variation" (RFS 2022), Armstrong et al. "Causality Redux" (2022)
- Writing craft: Cochrane "Writing Tips for PhD Students" (2005), Jackwerth "Tips for the PhD" (2022)
- Topics: political economy, corporate governance, ESG, executive compensation, innovation, causal inference methodology
