Menu
Systematically evaluate scholarly work using the ScholarEval framework, providing structured assessment across research quality dimensions including problem formulation, methodology, analysis, and writing with quantitative scoring and actionable feedback.
Based on SOC occupation classification
| name | scholar-evaluation |
| description | Systematically evaluate scholarly work using the ScholarEval framework, providing structured assessment across research quality dimensions including problem formulation, methodology, analysis, and writing with quantitative scoring and actionable feedback. |
| license | MIT license |
| required_environment_variables | [{"name":"OPENROUTER_API_KEY","prompt":"OpenRouter API key for the skill's LLM-powered steps.","required_for":"optional features"}] |
| metadata | {"version":"1.1","skill-author":"K-Dense Inc.","openclaw":{"primaryEnv":"OPENROUTER_API_KEY","envVars":[{"name":"OPENROUTER_API_KEY","required":false,"description":"OpenRouter API key for the skill's LLM-powered steps."}]}} |
Apply the ScholarEval framework to systematically evaluate scholarly and research work. This skill provides structured evaluation methodology based on peer-reviewed research assessment criteria, enabling comprehensive analysis of academic papers, research proposals, literature reviews, and scholarly writing across multiple quality dimensions.
Use this skill when:
When creating documents with this skill, always consider adding scientific diagrams and schematics to enhance visual communication.
If your document does not already contain schematics or diagrams:
For new documents: Scientific schematics should be generated by default to visually represent key concepts, workflows, architectures, or relationships described in the text.
How to generate schematics:
python scripts/generate_schematic.py "your diagram description" -o figures/output.png
The AI will automatically:
When to add schematics:
For detailed guidance on creating schematics, refer to the scientific-schematics skill documentation.
Begin by identifying the type of scholarly work being evaluated and the evaluation scope:
Work Types:
Evaluation Scope:
Ask the user to clarify if the scope is ambiguous.
Systematically evaluate the work across the ScholarEval dimensions. For each applicable dimension, assess quality, identify strengths and weaknesses, and provide scores where appropriate.
Refer to references/evaluation_framework.md for detailed criteria and rubrics for each dimension.
Core Evaluation Dimensions:
Problem Formulation & Research Questions
Literature Review
Methodology & Research Design
Data Collection & Sources
Analysis & Interpretation
Results & Findings
Scholarly Writing & Presentation
Citations & References
For each evaluated dimension, provide:
Qualitative Assessment:
Quantitative Scoring (Optional): Use a 5-point scale where applicable:
To calculate aggregate scores programmatically, use scripts/calculate_scores.py.
Provide an integrated evaluation summary:
Transform evaluation findings into constructive, actionable feedback:
Feedback Structure:
Feedback Format Options:
Adjust evaluation approach based on:
Stage of Development:
Purpose and Venue:
Discipline-Specific Norms:
Detailed evaluation criteria, rubrics, and quality indicators for each ScholarEval dimension. Load this reference when conducting evaluations to access specific assessment guidelines and scoring rubrics.
Search patterns for quick access:
Python script for calculating aggregate evaluation scores from dimension-level ratings. Supports weighted averaging, threshold analysis, and score visualization.
Usage:
python scripts/calculate_scores.py --scores <dimension_scores.json> --output <report.txt>
User Request: "Evaluate this research paper on machine learning for drug discovery"
Response Process:
references/evaluation_framework.md for detailed criteriaThis skill integrates seamlessly with the scientific writer workflow:
After Paper Generation:
SCHOLAR_EVALUATION.md alongside PEER_REVIEW.mdDuring Revision:
Publication Preparation:
This skill is based on the ScholarEval framework introduced in:
Moussa, H. N., Da Silva, P. Q., Adu-Ampratwum, D., East, A., Lu, Z., Puccetti, N., Xue, M., Sun, H., Majumder, B. P., & Kumar, S. (2025). ScholarEval: Research Idea Evaluation Grounded in Literature. arXiv preprint arXiv:2510.16234. https://arxiv.org/abs/2510.16234
Abstract: ScholarEval is a retrieval augmented evaluation framework that assesses research ideas based on two fundamental criteria: soundness (the empirical validity of proposed methods based on existing literature) and contribution (the degree of advancement made by the idea across different dimensions relative to prior research). The framework achieves significantly higher coverage of expert-annotated evaluation points and is consistently preferred over baseline systems in terms of evaluation actionability, depth, and evidence support.
Deterministically query 78 public scientific, biomedical, materials science, regulatory, finance, and demographics databases through documented REST APIs. Use for reproducible lookups of compounds, genes, proteins, pathways, variants, clinical trials, patents, economic indicators, structures, astronomy objects, environmental records, or database-backed scientific facts when endpoints, filters, pagination, and provenance need to be explicit.
Hugging Face Transformers for loading Hub models, running pipeline inference, text generation, and Trainer fine-tuning on NLP, vision, audio, and multimodal tasks. Use when working with AutoModel, pipelines, tokenizers, or TrainingArguments—not for general ML outside the Transformers library.
Autonomously improve a real artifact (code, training recipe, agent harness, data pipeline, prompt) against an objective and an evaluator, using Hypothesis Tree Refinement (HTR) from the Arbor paper. Use this whenever someone wants to iteratively optimize something over many experiments without overfitting — e.g. "get my model's eval score up", "improve this agent/harness", "tune this pipeline", "beat the baseline on this benchmark", "run a search over approaches and keep the best", "do an MLE-bench / Kaggle-style optimization", or any long-horizon "make this artifact better and don't just memorize the dev set" task. Trigger it even when the user doesn't say "Arbor" or "hypothesis tree" but describes repeated experiment-and-evaluate loops, branching exploration of competing ideas, or worries about a dev/test gap. Runs Claude itself as the coordinator with subagent executors in isolated git worktrees; for the standalone `arbor` CLI tool see references/arbor-upstream.md.
Standard single-cell RNA-seq analysis pipeline. Use for QC, normalization, dimensionality reduction (PCA/UMAP/t-SNE), clustering, differential expression, visualization, and converting R-friendly single-cell formats such as Seurat or SingleCellExperiment RDS files into h5ad for Scanpy. Best for exploratory scRNA-seq analysis with established workflows. For deep learning models use scvi-tools; for data format questions use anndata.
Complete mass spectrometry analysis platform. Use for proteomics and metabolomics workflows—feature detection, peptide/protein identification, label-free and isobaric quantification, adduct/accurate-mass annotation, and complex LC-MS/MS pipelines. Supports extensive file formats and algorithms. For simple spectral comparison and small-molecule library matching use matchms.
Design experiments and studies BEFORE data is collected — choosing a design, randomizing, blocking, and laying out treatment combinations so the results will actually be interpretable. Use whenever someone is planning a study, asks how to assign subjects/samples to groups, mentions randomization, blocking, stratification, controls, factorial or fractional-factorial designs, design of experiments (DOE), screening many factors, response-surface optimization, crossover or repeated-measures or split-plot designs, cluster/group randomization, Latin squares, plate layouts, batch/run-order effects, replication vs. pseudoreplication, or sequential/adaptive/group-sequential designs. Trigger this even for informal phrasings like "how should I set up this experiment", "how do I avoid confounding", "what's the best way to test these 6 factors", or "assign these mice to conditions". For computing the sample size or power once the design is chosen, use statistical-power; for analyzing data already collected, use statistica