Multi-persona idea evaluation with quality gate. Evaluates ideas across 5 InnoEval dimensions (Clarity, Novelty, Validity, Feasibility, Significance) using 3 reviewer personas and a meta-review. Sits between inno-idea-generation and inno-code-survey in the Idea branch. Use after inno-idea-generation.
Multi-persona idea evaluation with quality gate. Evaluates ideas across 5 InnoEval dimensions (Clarity, Novelty, Validity, Feasibility, Significance) using 3 reviewer personas and a meta-review. Sits between inno-idea-generation and inno-code-survey in the Idea branch. Use after inno-idea-generation.
Inno Idea Eval
Directory structure
skills/inno-idea-eval/
├── SKILL.md ← this file
├── prompts/
│ ├── build_eval_query.md ← Per-persona evaluation query (all 5 dims)
│ ├── build_evidence_assembly.md ← How to compose evidence from pipeline artifacts
│ ├── build_meta_review_query.md ← Area-chair aggregation of 3 persona reviews
│ ├── build_novelty_queries.md ← Query extraction for novelty verification (Step 0.5a)
│ ├── build_novelty_analysis.md ← Similarity analysis for novelty verification (Step 0.5c)
│ └── build_refinement_feedback_query.md ← Structured feedback for refinement loop
└── references/
├── eval_agent_instructions.md ← Full eval agent system prompt + scoring rubrics
├── novelty_verification_config.md ← Novelty search config, threat levels, fast-fail protocol
└── reviewer_personas.md ← 3 persona definitions + evidence filter logic
How to use the resource files: Each prompt template in prompts/ documents
the exact parameters, the full text template, and usage notes (when it is a new
conversation vs. appended message, how to format evidence blocks, etc.).
The references/ directory contains the Eval Agent's complete system instructions
including its scoring rubrics, persona definitions, and evidence filter logic.
Consult these files for the authoritative details; the steps below provide a summary.
Inputs
Paths for Ideation/ideas and Ideation/references come from instance.json (instance.Ideation.ideas, instance.Ideation.references). They are absolute in Vibe Lab-created projects; use as-is. If relative, resolve with path.join(project_path, value).
Parameter
Required
Description
selected_idea
Yes
The idea to evaluate, read from Ideation/ideas/selected_idea.txt
references
No*
Pre-formatted string listing all source papers (from inno-prepare-resources)
prepare_res
No*
Full text response from the Prepare Agent (selected repositories and reasoning)
download_res
No*
Result log from downloading arXiv paper sources
data_module
No*
The imported metaprompt module (provides TASK field describing the ML task)
Strengths/weaknesses/suggestions (for refinement or downstream)
context_variables["idea_evaluation_result"]
Complete structured result dict
Cache file outputs
Each step produces two kinds of files:
.txt files (primary) -- the full markdown content of each review, written directly to Ideation/ideas/
.json files (derived) -- structured metadata under Ideation/ideas/logs/, whose text fields must be copied verbatim from the corresponding .txt files (never summarized)
Each file contains context_variables only (no messages). The review_text field holds the full text copied from the corresponding .txt file:
{"context_variables":{"ideas_path":"<instance.Ideation.ideas>","references_path":"<instance.Ideation.references>","persona":"senior_ml_researcher | domain_expert | methods_specialist","scores":{"clarity":{"score":0,"reason":"...","references":[]},"novelty":{"score":0,"reason":"...","references":[]},"validity":{"score":0,"reason":"...","references":[]},"feasibility":{"score":0,"reason":"...","references":[]},"significance":{"score":0,"reason":"...","references":[]}},"strengths":[],"weaknesses":[],"suggestions":[],"recommendation":"Accept|Reject|...","review_text":"<FULL text from eval_persona_{N}_review.txt>"}}
.json file format (meta-review)
{"context_variables":{"ideas_path":"<instance.Ideation.ideas>","references_path":"<instance.Ideation.references>","aggregated_scores":{"clarity":{"avg":0,"scores":[0,0,0]},"novelty":{"avg":0,"scores":[0,0,0]},"validity":{"avg":0,"scores":[0,0,0]},"feasibility":{"avg":0,"scores":[0,0,0]},"significance":{"avg":0,"scores":[0,0,0]}},"overall_avg":0,"decision":"strong_accept|accept|borderline_accept|borderline_reject|reject","report_text":"<FULL text from eval_report.txt>","strengths":[],"weaknesses":[],"suggestions":[],"idea_evaluation_result":{"...complete structured result..."}}}
.json file format (novelty verification)
{"context_variables":{"step":"novelty_verification","search_config":{"num_queries":4,"sources":["arxiv","semantic_scholar","openalex"],"max_results_per_query":10,"year_from":"<current_year - 3>"},"queries":[{"type":"core_method","query":"...","rationale":"..."},{"type":"problem_domain","query":"...","rationale":"..."},{"type":"key_component","query":"...","rationale":"..."},{"type":"broad_approach","query":"...","rationale":"..."}],"idea_summary":"...","search_results":{"total_raw":0,"total_unique":0},"triage":[{"title":"...","year":0,"relevance":"high|medium|low|irrelevant","is_inspiration_source":false,"assessment":"..."}],"detailed_analysis":[{"title":"...","year":0,"overlap":"...","differences":"...","threat_level":"..."}],"novelty_threat_level":"critical_overlap|high_overlap|moderate_overlap|low_overlap|novel","genuine_novel_contributions":["..."],"report_text":"<FULL text from novelty_grounding_report.txt>","fast_fail_triggered":false,"user_decision":null}}
review_text and report_text must contain the complete markdown from the .txt file -- never a summary or abbreviation.
IMPORTANT: Each persona .json grows independently; the meta-review .json aggregates all three.
Proactively search the literature to verify whether the idea (or key components) already exists. This step runs before persona reviews so all 3 reviewers have the prior art report as evidence.
Sub-steps:
0.5a — Extract search queries (LLM call using build_novelty_queries.md):
Input: selected_idea + known source_papers (inspiration)
