evidence-grading

name	evidence-grading
description	Evaluate the strength of evidence behind scientific claims based on study design, replication status, venue quality, sample size, and recency. Use when deciding how much weight to put on a finding, or when calibrating how confidently to write about a result.
always	false

Evidence Grading

When to Use

• User asks "how strong is the evidence for [claim]?"
• Before building on a published result — assess how solid the foundation is
• Deciding between citing a claim as "established" vs. "promising" vs. "preliminary"
• Ranking a set of papers by evidentiary weight
• When two papers contradict each other — which has stronger evidence?

Grading Factors

Evidence is evaluated across 5 dimensions:

Dimension	Strong	Weak
Study design	Controlled experiment, ablations, comparisons	Single-condition, no ablations
Replication	Replicated by independent groups	Single study, preprint only
Sample/scale	Large dataset, multiple seeds, statistical tests	Small dataset, single seed
Venue	Top-tier venue (NeurIPS/ICML/Nature/Science)	Workshop / preprint / self-published
Recency	Recent (last 2 years)	Older work that may have been superseded

Grade Schema

For ML/CS Research:

Grade	Label	Criteria
A	Established	Multiple top-venue replications; widely accepted; in textbooks
B+	Strong	Published top venue + ≥1 independent replication
B	Good	Top-venue paper, solid methodology, not yet replicated independently
C+	Moderate	Smaller venue or preprint + single replication
C	Preliminary	Single top-venue paper, no independent replication
D	Weak	Workshop paper, preprint, limited evaluation
F	Insufficient	Single experiment, no baselines, no ablations

Hedging Language by Grade:

Grade	Suggested language
A	"It is established that..." / "[Method] has been shown to..."
B+	"[Method] consistently demonstrates..."
B	"[Paper] shows that... (though independent replication is limited)"
C+	"[Paper] suggests that... with moderate evidence"
C	"[Paper] provides preliminary evidence that..."
D	"[Paper] reports that... (preliminary; requires replication)"
F	Avoid citing as evidence; mention only for context

Workflow

Step 1: Identify the Claim and Source Paper

• State the specific claim being evaluated
• Use paper-read to get: venue, year, citation count
• Note citation count as a rough proxy (highly cited = more scrutiny + replication)

Step 2: Assess Study Design Quality

Read methodology (via paper-fetch or paper-read-pdf):

• Were multiple baselines included?
• Were ablation studies conducted?
• Were multiple random seeds used?
• Were results statistically significant (error bars, p-values)?
• Were results on standard benchmarks or custom/cherry-picked?

Step 3: Check Replication Status

Use paper-search and claim-tracker:

• Has the result been replicated by independent groups?
• Have surveys/meta-analyses included this finding?
• Has anyone failed to replicate it?

Step 4: Check Venue and Recency

• Venue tier: NeurIPS/ICML/ICLR/ACL/CVPR/NeurIPS (A*) > AAAI/EMNLP/ECCV (A) > arXiv preprint
• Recency: older results may be superseded by newer work — verify with paper-search --year-from [year]

Step 5: Assign Grade and Write Summary

Output Format

### Evidence Grade: [Claim]
- **Grade**: [A / B+ / B / C+ / C / D / F]
- **Source**: [Paper title] ([Year], [Venue])
- **Replication**: [Independent replications: N | Not yet replicated]
- **Study design**: [Controlled with ablations | Limited baselines | Single condition]
- **Scale**: [Large-scale, multi-seed | Small-scale, single seed]
- **Recency**: [Current | Potentially outdated — see [newer paper]]
- **Rationale**: [2–3 sentence explanation of the grade]
- **Recommended citation language**: "[Suggested phrasing]"
- **Caution**: [Any specific caveats about using this evidence]

Save Results

• Include evidence grades inline in survey or gap-analysis documents
• Update MEMORY.md when a key result is graded, especially if lower than expected (C or below)

Integration Tips

• Combine with claim-tracker — verify claim status first, then grade evidence strength
• Combine with contradiction-detection — contradicted claims automatically receive a grade penalty (max B-)
• Run before reproduce skill — prioritize reproducing B or lower results that are being relied upon
• Low-grade evidence (C/D) on a central assumption = a research opportunity (gap to fill)