sciatlas-idea-evaluate

# SciAtlas Idea Evaluate Use this skill to run the repository automated review workflow. The workflow builds idea context, searches KG/S2 evidence, creates a manifest, grounds claims to paper paragraphs, builds rubric evidence, samples reviewer backgrounds, generates reviewer reports, and synthesizes a final report. ## Operating Contract - Run only `sciatlas idea-evaluate` or `python run_sciatlas.py idea-evaluate` for this skill. - Own the end-to-end novice flow: install or locate the CLI, guide registration, configure `.env` or shell variables, run the workflow, inspect artifacts, and synthesize the final review result. - Ask the user only for human-only values: missing idea/PDF path, email, verification code, SciAtlas token, LLM/S2/KG credentials that are not already configured, or one necessary scope clarification. - Do not ask the user to run shell commands when tool access is available. - Use `--workflow flash` by default. - Use `--workflow full` for a broader reviewer/rubric/evidence pass or when the user wants a more comprehensive review. - Never disclose full API keys or tokens. - Read saved artifacts before answering. ## Zero-Start Bootstrap 1. Check whether the repository command works: ```bash python run_sciatlas.py idea-evaluate -h ``` If needed, fall back to `sciatlas idea-evaluate -h` after installing the full checkout. 2. This dedicated workflow requires a full SciAtlas checkout. If it is missing, clone the repository, change into it, then run `python -m pip install -e ./sciatlas` and `python -m pip install -r requirements-workflows.txt`. Do not use the GitHub `#subdirectory=sciatlas` package-only installation for this workflow. 3. Run `python scripts/check_rubric_setup.py`. If it reports missing models, assets, packages, or keys, fix them yourself following [Local Model Setup](#local-model-setup) before proceeding — the workflow cannot run without the local embedding/rerank models. 4. Check current environment and `.env` for `SCIATLAS_API_KEY`, LLM settings, S2 settings, and KG settings before asking the user. 5. If no SciAtlas token is configured, guide the user to `http://sciatlas.openkg.cn/register`; ask for email, verification code, and returned `sciatlas_xxx` token only when needed. 6. If LLM/S2/KG credentials are required and missing, ask only for the missing values. Use the user's provider values without printing them back. 7. Configure the current shell or `.env` yourself, then run the workflow. Configure workflow credentials in `.env` or the shell: ```bash SCIATLAS_API_BASE_URL=http://sciatlas.openkg.cn SCIATLAS_API_KEY=<sciatlas-token> NEO4J_URI=<local-or-hosted-neo4j-uri> NEO4J_USER=<neo4j-user> NEO4J_PASSWORD=<neo4j-password> S2_API_KEY=<semantic-scholar-key> OPENAI_API_KEY=<llm-key> OPENAI_BASE_URL=https://api.deepseek.com LLM_MODEL=deepseek-v4-flash ``` The workflow also accepts `DMX-API-KEY`, `DMX_API_KEY`, `LLM_API_KEY`, `LLM_BASE_URL`, `LLM_API_URL`, `SEARCH_LLM_API_KEY`, `SEARCH_LLM_API_URL`, and `SEARCH_LLM_MODEL`. ## Local Model Setup The rubric and grounding stages run on two local models that you must download once — do this yourself with tool access, never ask the user to run download commands, and never substitute a remote embeddings service. First check what is missing (the script prints the exact fix commands): ```bash python scripts/check_rubric_setup.py ``` Download both models (~1.3GB each) into `<repo>/models/`; the workflow auto-detects them there, so no environment variables are required: ```bash pip install -r requirements-rubric.txt export HF_ENDPOINT=https://hf-mirror.com # mainland-China mirror; omit when huggingface.co is reachable huggingface-cli download BAAI/bge-large-en-v1.5 --local-dir <repo>/models/bge-large-en-v1.5 huggingface-cli download BAAI/bge-reranker-large --local-dir <repo>/models/bge-reranker-large python scripts/download_rubric_assets.py # FAISS index + precomputed NC-paper artifacts (~160MB) ``` ModelScope alternative (mainland China, no mirror needed): ```bash pip install modelscope modelscope download --model AI-ModelScope/bge-large-en-v1.5 --local_dir <repo>/models/bge-large-en-v1.5 modelscope download --model BAAI/bge-reranker-large --local_dir <repo>/models/bge-reranker-large ``` Verification checklist before running the workflow: - `python scripts/check_rubric_setup.py` exits 0 — re-run it after every download or config change. - Directory names are exactly `bge-large-en-v1.5` / `bge-reranker-large` under `<repo>/models/` (auto-detection is name-based). Models kept elsewhere must be pinned with **absolute paths**: `RUBRIC_EMBED_MODEL_PATH` / `RUBRIC_RERANK_MODEL_PATH` (rubric retrieval), `INNOEVAL_EMBEDDING_MODEL_PATH` / `INNOEVAL_RERANKER_MODEL_PATH` (KG search / author profiling), `SCIATLAS_EMBEDDING_MODEL_PATH` / `SCIATLAS_RERANKER_MODEL_PATH` (grounding). - Each model directory contains `config.json` plus weights (`model.safetensors` or `pytorch_model.bin`). - The embedding model must be exactly `bge-large-en-v1.5` (1024 dims) — the rubric FAISS index was built with it; other embedding models will not match. - On shared GPU machines set `INNOEVAL_CUDA_DEVICES` to free GPU ids; CPU also works (slower). - The asset pack is always required: it provides the FAISS index, `nc_meta.json`, and per-paper precomputed artifacts (skipping Europe PMC fetches and per-paper LLM calls). When default dataset paths do not exist, index and metadata are auto-detected from the pack. ## Run Plan Flash path for an idea: ```bash python run_sciatlas.py idea-evaluate \ --idea "<research idea>" \ --workflow flash ``` Full path: ```bash python run_sciatlas.py idea-evaluate \ --idea "<research idea>" \ --workflow full ``` For a paper PDF: ```bash python run_sciatlas.py idea-evaluate \ --pdf path/to/paper.pdf \ --workflow full ``` Useful overrides: - `--top-k N`, `--kg-top-k N`, `--s2-top-k N`, `--manifest-top-k N` tune evidence breadth. - `--max-reviewers N` controls reviewer fan-out. - `--grounding-final-top-k N` controls paragraph evidence depth. - `--short-report` requests compact meta-review output in full mode. - `--disable-review-llm` uses deterministic fallback when LLM review is unavailable. - `--smoke` runs a stubbed pipeline for structural validation. ## Workflow Modes `flash` compresses nonessential stages: - smaller KG/S2/manifest budgets; - one reviewer by default; - compact evidence-card budgets; - grounding refinement disabled; - short meta-review enabled. `full` keeps the comprehensive path: - broader KG/S2/manifest budgets; - reviewer fan-out and reviewer-background branch; - rubric-source and rubric-LLM branches; - paragraph grounding, reviewer reports, and full report synthesis. ## Artifacts To Read Read the run directory: - `summary.json`: wrapper status, workflow mode, subprocess command, and artifact pointers. - `result.json`: review pipeline final payload and stage statuses. - `report.md`: user-facing final report or compact meta-review. - `idea_context/idea_context.json`: normalized idea/PDF context and structured idea extraction. - `search/result.json`: KG/S2 search evidence. - `manifest/manifest.json`: papers selected for paragraph extraction. - `grounding/result.json`: paragraph grounding and experiment grounding. - `rubric_sources/result.json`, `rubric_llm/result.json`: rubric evidence when available. - `review/reviewer_reviews.index.json`: reviewer-level evaluations. - `report/final_report.md`, `report/final_report.json`: native final outputs. - `logs/*.log` and `logs/review_pipeline.*.txt`: stage logs when anything is partial or failed. Treat `partial_error` as usable when the final report exists, but disclose which branch failed or was skipped. ## Deliverable Return: - exact command used, with credentials omitted; - workflow mode and artifact paths; - overall go/revise/no-go recommendation; - novelty, feasibility, soundness, and differentiation assessment; - closest-prior-art risk; - strongest supporting evidence and biggest missing evidence; - concrete revision advice and next SciAtlas query. Keep judgments tied to retrieved papers, grounding snippets, reviewer reports, or rubric artifacts.