papers-collect-from-github-repo

name	papers-collect-from-github-repo
status	active-local-collect
description	Collects paper candidates from any GitHub repository README or docs — awesome lists, survey companion repos, lab paper lists, conference accepted-paper repos, benchmark leaderboards, etc. Agent fetches the README (or specified doc), analyzes its format, writes a one-off parser, and outputs rows aligned with paper_list.csv. No pre-built scripts — agent adapts to each repo's format on the fly.

Papers Collect From GitHub Repo

Purpose

Extract paper candidates from any GitHub repository and output rows aligned with obsidian-vault/paper_list.csv.

Supported repo types (non-exhaustive):

Type	Example
Awesome / curated list	Foruck/Awesome-Human-Motion
Survey companion repo	ChenHsing/Awesome-Video-Diffusion-Models (CSUR)
Lab / group paper list	showlab/showlab.github.io
Conference accepted papers	ICLR/ICLR-2026-accepted-papers
Benchmark / leaderboard repo	open-compass/opencompass
Topic-specific collection	soraproducer/Awesome-Human-Interaction-Motion-Generation
Any repo with a structured paper list in README or docs	—

Because repo formats vary widely (Markdown tables, bullet lists, mixed HTML, nested docs, multiple files, etc.), this skill does not include fixed scripts. The agent writes one-off parsing logic for each target.

Input

User provides:

• GitHub repo URL: e.g. https://github.com/Foruck/Awesome-Human-Motion
• target file (optional): defaults to README.md; can be a specific path like docs/papers.md or a directory to scan
• include/exclude keywords (optional): e.g. include motion generation, exclude survey
• target category (optional): e.g. Motion_Generation_Text_Speech_Music_Driven
• venue/year hint (optional): e.g. CVPR 2025 — used as default when venue cannot be parsed

Output Format

Output must align with the column format of obsidian-vault/paper_list.csv:

state,importance,paper_title,venue,project_link_or_github_link,paper_link,sort,pdf_path

Field notes:

Column	Initial default	Description
state	Wait	Later can be changed manually to Skip, or remain Wait for download
importance	empty	Later manually labeled as S/A/B/C
paper_title	parsed from repo	paper title
venue	parsed from repo, fallback Unknown	e.g., CVPR 2025, ICLR 2026; if only on open platforms, use arXiv YYYY
project_link_or_github_link	project page or GitHub link	prefer project page; if confirmed no open source, use N/A
paper_link	arXiv / OpenReview link	prefer arXiv abs link
sort	section/category heading in source	join words with _, e.g. Motion_Generation
pdf_path	empty	filled later by papers-download-from-list

See obsidian-vault/paper_list.csv in the repository for examples.

Workflow

1. Fetch Source Content

• Determine what to fetch:
  • Single README: https://raw.githubusercontent.com/<owner>/<repo>/<branch>/README.md
  • Specific file: user-specified path within the repo
  • Multiple files: if repo organizes papers across multiple .md files (e.g. papers/2025.md, papers/2026.md), fetch all relevant files
• Optionally save raw content to obsidian-vault/analysis/processing/github_awesome/<repo_slug>/ for debugging
  • Note: the github_awesome directory name is kept for backward compatibility; it stores artifacts from all GitHub repo sources, not just awesome lists

2. Analyze Format

Agent reads the content and determines structure:

• Markdown table, bullet list, nested sections, or mixed format?
• Single file or multi-file organization?
• Where paper titles appear and how links are organized?
• Whether venue metadata is inline, in section headings, or in separate columns?
• Whether the repo uses badges, emoji markers, or other conventions?

3. Write One-Off Parser

Based on the structure analysis, the agent writes a one-off Python script (or processes directly in conversation) to extract paper information.

Script requirements:

• Output CSV with the same column order as paper_list.csv
• Handle common link patterns: arXiv abs/pdf, OpenReview, GitHub project page, conference proceedings (ACL Anthology, IEEE Xplore, etc.)
• Parse venues from text patterns like (CVPR 2025) / [ICLR 2026] / section headings like ## NeurIPS 2025
• Deduplicate repeated appearances of the same paper

4. Filter and Deduplicate

• Apply user include/exclude keyword filtering
• Deduplicate against existing paper_list.csv entries (fuzzy match by paper_title)
• Report number of newly added candidate rows

5. Append to paper_list.csv

• Append new rows to obsidian-vault/paper_list.csv
• Do not modify existing rows
• Report to user: number of new candidates and source sections

6. Completeness Check and Suggestion

After append, scan newly added rows for missing fields:

Missing field	Detection condition	Completion method	Fallback
venue	empty or Unknown	lookup by paper_link via arXiv metadata / Semantic Scholar API	if unpublished at venue but public: arXiv YYYY (e.g., arXiv 2025)
paper_link	empty	search arXiv / Google Scholar by paper_title	if not found: keep empty
project_link_or_github_link	empty	search GitHub / Papers With Code by paper_title	if confirmed no open source: N/A
sort	empty	infer from source section heading or ask user	keep empty for manual fill

Then report and suggest:

"Added N new candidates: X missing venue, Y missing paper_link, Z missing project_link. Do you want multi-source completion? (complete all / venue only / links only / skip)"

After user confirmation:

• complete all: use arXiv API, Semantic Scholar, Google Scholar, Papers With Code in sequence
• specific fields only: search only matching sources
• skip: keep current state for later manual handling

Constraints for completion:

• Keep at least 3 seconds between searches (respect API rate limits)
• Match search results to paper_title; only auto-fill when similarity > 0.8, otherwise mark [needs confirmation]
• Write completion results back to matching rows in paper_list.csv

Environment Detection

If Python scripts are needed, follow Environment Detection rules from code-context-paper-retrieval:

• prioritize environment detection from codebase environment/environment.yml / environment/requirements.txt etc.
• if unresolved, proactively ask user

Constraints

• No prebuilt parser scripts; each repo format is adapted live
• One-off scripts may be saved to obsidian-vault/analysis/processing/github_awesome/<repo_slug>/parse_<timestamp>.py for future reference
• Do not auto-download PDFs (that is papers-download-from-list)
• Do not auto-generate analysis notes; that belongs to the formal analysis chain (scripts/run_local_paper_analysis.py) or the batch analysis manager.

Typical Usage

"Collect papers from https://github.com/Foruck/Awesome-Human-Motion, only motion generation related"

"Organize papers from https://github.com/ChenHsing/Awesome-Video-Diffusion-Models into paper_list.csv, focus on controllable generation sections"

"Collect ICLR 2026 accepted papers from https://github.com/xxx/iclr2026-papers"

"Extract papers from https://github.com/xxx/lab-publications, filter by 2025-2026"