enrich-paper

# Enrich a Paper This skill fills in the two things a freshly imported paper most often lacks: 1. **Code links** — the paper's *own* official GitHub repository and project page URL. 2. **Hugging Face artifacts** — official model, dataset, and Space URLs, synchronized into normalized artifact links. 3. **Taxonomy tags** — `tasks` and `methods` that already exist in the database, matched to what the paper is actually about. 4. **Thumbnail** — an authoritative, recognizable paper-card image, not merely a reachable URL or generic browser capture. It is the agent-driven replacement for the Gemini URL-extraction cron (`backend/genai_steps/extract_github_and_project_page_url.py` + `backend/database/repos_and_pages_setup/batch_extract.py`). Instead of calling the Gemini API, **you** (the coding agent) fetch a small amount of evidence and reason over it yourself. The whole point is to find *authentic* URLs at low token cost, so prefer cheap structured sources first and only escalate to heavier ones (PDF text, project-page HTML) when something is still missing. Writes go to the **live database** via existing services / the admin API, exactly like `add-external-paper`. Keep writes idempotent and scoped to the requested paper. ## Core principles - **Authenticity over recall.** A GitHub or project-page URL is only valid if it is the paper's *own* artifact. Papers routinely cite *other* repos (baselines, datasets, frameworks, related work). A `github.com/...` on page 1 is **not** automatically the paper's repo. When in doubt, leave it out and report it — a false positive is worse than a miss. - **Cheapest evidence first.** Each step below is more expensive than the last. Stop as soon as you confidently have both URLs. Never dump a whole PDF into your context — the first 2 pages (or the HF markdown) carry the code links. - **Only existing taxonomy.** Tag with `tasks`/`methods` rows that already exist. Do not invent new ones (that's `add-task` / `add-method`). If the right concept is missing, pick the closest existing tag and report the gap. - **Complement the crons, don't fight them.** Method tags are *also* added by the Semantic Scholar citation cron (papers that cite a method's source paper) and by the HF import keyword matcher. Your job is content-based tagging for *this* paper; a few high-confidence tags beat keyword stuffing. - **Leave the paper card looking right.** Thumbnail completion is part of enrichment. A reachable image can still be incomplete when it is blank, generic, says “Untitled resource,” shows stale content, or uses the wrong arXiv-version title. Use the sibling `create-thumbnail` skill for the audit, source selection, composition, cache-safe publication, and live visual verification. ## Pre-flight Run everything from the repo root with the project environment so the venv and secrets load: ```bash uv run --env-file backend/keys.env ... ``` Confirm the API is up and grab the admin token + HF token: ```bash curl -s -o /dev/null -w "%{http_code}\n" http://localhost:8000/api/v1/papers/recent?limit=1 ADMIN_TOKEN=$(grep '^PWC_ADMIN_API_TOKEN=' backend/keys.env | cut -d= -f2-) HF_TOKEN=$(grep '^PWC_HF_TOKEN=' backend/keys.env | cut -d= -f2-) ``` If the API is unavailable, use direct database verification where supported or report the blocker. Do not start or restart deployed services during curation. ## Step 0 — Resolve the paper row You need the DB `paper_id` (and `arxiv_id` if any). Accept an arXiv id, a `paperswithcode.co/paper/<id>` URL, or an external paper. Read what's already linked so you can **merge** rather than clobber. ```bash uv run --env-file backend/keys.env python - <<'PY' from sqlalchemy import text from backend.database.connection import get_engine ident = "2509.14033" # arxiv_id, numeric paper id, or title fragment with get_engine().connect() as conn: row = conn.execute(text(""" SELECT id, arxiv_id, title, source, source_url, urls_extracted FROM papers WHERE arxiv_id = :ident OR CAST(id AS TEXT) = :ident OR title ILIKE :like ORDER BY is_latest_version DESC NULLS LAST, id DESC LIMIT 1 """), {"ident": ident, "like": f"%{ident}%"}).mappings().first() print(dict(row) if row else "NOT FOUND") PY ``` Then read existing tasks/methods/repos/project pages so you can merge: ```bash curl -s "http://localhost:8000/api/v1/papers/<paper_id>?include_resources=true" \ | uv run python -c "import json,sys; p=json.load(sys.stdin); print('tasks',[(t['id'],t['slug']) for t in p.get('tasks') or []]); print('methods',[(m['id'],m['slug']) for m in p.get('methods') or []]); print('repos',[(r.get('url'),r.get('is_official')) for r in p.get('repositories') or []]); print('project_pages',[pp.get('url') for pp in p.get('project_pages') or []])" ``` If the paper has no arXiv id (external paper), the arXiv-specific steps below don't apply — use the paper's `source_url`, linked GitHub README, or HF model card as the evidence surface instead, and skip straight to URL validation + tagging. If the paper row itself does not exist yet, this is the wrong skill — use `add-external-paper` first, then come back to enrich. ## Part A — GitHub repository & project page URL Walk the layers in order. Carry forward whatever you've found; only escalate for what's still missing. After each candidate URL, **validate authenticity and liveness** (see "Validating URLs"). ### Layer 1 — Hugging Face papers API (cheapest, most authoritative) If the paper is on HF, authors/maintainers often already linked the repo and project page. Use the HF token to avoid rate limits. ```bash curl -s -H "Authorization: Bearer $HF_TOKEN" \ "https://huggingface.co/api/papers/<arxiv_id>" \ | uv run python -c "import json,sys; d=json.load(sys.stdin); print('githubRepo:', d.get('githubRepo','')); print('projectPage:', d.get('projectPage',''))" ``` `githubRepo` and `projectPage` here are author-curated and can be trusted with light validation. If both are present and live, you're done with Part A. The same surface has author/community comments that frequently contain the links (HF often surfaces the official repo here even when the abstract doesn't): ```bash curl -s -H "Authorization: Bearer $HF_TOKEN" \ "https://huggingface.co/api/papers/<arxiv_id>?field=comments" \ | uv run python -c "import json,sys; d=json.load(sys.stdin); [print(c.get('author',{}).get('name'),'=>',c.get('data',{}).get('latest',{}).get('raw','')[:400]) for c in d.get('comments',[])]" ``` Prefer links posted by an author of the paper or its official org; treat random third-party comments with more suspicion. ### Layer 2 — Regex over abstract + HF markdown (cheap text) The HF markdown rendering of a paper is small and link-rich. Read it and the abstract, and scan for the paper's own GitHub/project links: ```bash # Markdown version of the paper page (note the .md suffix) curl -sL -H "Authorization: Bearer $HF_TOKEN" "https://huggingface.co/papers/<arxiv_id>.md" -o /tmp/<arxiv_id>.md # or: uv run --env-file backend/keys.env hf papers read <arxiv_id> > /tmp/<arxiv_id>.md ``` Patterns the existing pipeline uses (reuse them rather than reinventing): - GitHub: `\bhttps?://github\.com[^\s]*\b` - Project page (github.io is a strong signal but not required): `\bhttps?://[a-zA-Z0-9.-]+\.github\.io[^\s]*` You can also call the repo's own helpers for the abstract-regex + HF-API combo: ```bash uv run --env-file backend/keys.env python - <<'PY' import asyncio from backend.utils.paper_utils import ( get_abstract, find_github_url, get_github_and_project_page_urls_from_daily_papers, ) async def main(): arxiv_id = "<arxiv_id>" gh_api, pp_api = await get_github_and_project_page_urls_from_daily_papers(arxiv_id) abstract = await get_abstract(arxiv_id) gh_abs = await find_github_url(arxiv_id=arxiv_id, abstract=abstract) print("hf_api github:", gh_api, "| hf_api project:", pp_api, "| abstract github:", gh_abs) asyncio.run(main()) PY ``` A GitHub URL stated in the abstract ("code available at …", "we release … at …") is almost always authentic. A bare `github.com` link buried in markdown body text needs the authenticity check before you trust it. ### Layer 3 — First 2 pages of the PDF (only if still missing) Many papers put the repo/project link in a first-page footnote or under the abstract, not in the HF metadata. Download the PDF and read **only the first two pages** to keep cost down: ```bash curl -sL "https://arxiv.org/pdf/<arxiv_id>.pdf" -o /tmp/<arxiv_id>.pdf pdftotext -l 2 -layout /tmp/<arxiv_id>.pdf - 2>/dev/null | head -c 12000 ``` If `pdftotext` is unavailable, use the repo helpers (`extract_text_from_pdf` reads the first 2 pages; `extract_hyperlinks_from_pdf` pulls embedded hyperlinks, which often contain the clickable repo link even when the visible text is truncated): ```bash uv run --env-file backend/keys.env python - <<'PY' from backend.utils.pdf_utils import extract_text_from_pdf, extract_hyperlinks_from_pdf print(extract_text_from_pdf("/tmp/<arxiv_id>.pdf", num_pages=2)[:8000]) print("LINKS:", [l["url"] for l in extract_hyperlinks_from_pdf("/tmp/<arxiv_id>.pdf") if "github.com" in l["url"] or "github.io" in l["url"]]) PY ``` Reason over this text yourself (this is the part that used to be the Gemini call). Decide which link, if any, is the paper's own repo/project page. Do **not** feed the whole PDF to anything — first two pages only. ### Layer 4 — Project-page HTML → GitHub (the Step-2 fallback) Some papers only print a project-page URL and hide the real GitHub link inside the project page's HTML — this is the single highest-value step, because the layers above genuinely can't see it. If you have a project page but **no** GitHub yet, fetch the project page and look for the repo: ```bash curl -sL "<project_page_url>" | grep -oE 'https?://github\.com/[A-Za-z0-9_.-]+/[A-Za-z0-9_.-]+' | sort -u | head ``` (Or use `WebFetch` on the project page and read off the "Code"/"GitHub" link.) **Critical: project pages are built from website templates, so the HTML almost always contains a template's GitHub repo that is NOT the paper's code.** Grepping blindly here is the most common false-positive source. Reject these on sight: - `github.com/eliahuhorwitz/Academic-project-page-template` - `github.com/nerfies/nerfies.github.io` (and other `*/<name>.github.io` *page-source* repos) - any `*-project-page`, `*-template`, `*.github.io` repo that is the page's own source rather than the method's code - starter kits, analytics, or library repos referenced by the page's `<head>`/footer Accept a repo only when it is plausibly the paper's *own* code. Strong signals: the owner matches the project-page subdomain owner (e.g. page `https://yimingc9.github.io/physiformer` → repo `github.com/yimingc9/PhysiFormer` ✓) or a paper author; the repo name matches the method/paper name; the page links it under a "Code"/"GitHub" button rather than a tiny "website template" credit in the footer. When the page exposes a "Code coming soon" placeholder and only the template repo is present, the correct answer is **no GitHub** — link the project page and move on. Examples seen in the wild: a page whose only GitHub link is `eliahuhorwitz/Academic-project-page-template` ⇒ no repo; a page linking both `owner/Method` and `owner/Method.github.io` ⇒ take `owner/Method`, not the `.github.io` page-source repo. ### Validating URLs (authenticity + liveness) Before saving, every candidate must pass both: 1. **Liveness** — the URL resolves (HTTP 200). Reuse the repo helper or curl: ```bash uv run --env-file backend/keys.env python -c "import asyncio; from backend.utils.verify_url import verify_url; print(asyncio.run(verify_url('<url>')))" ``` For GitHub, also confirm the repo exists and isn't a 404/empty redirect: `gh repo view <owner>/<repo> --json url,isArchived,description` (or `curl -sI`). 2. **Authenticity** — it is the paper's own artifact. Strong positive signals: stated in the abstract or a first-page footnote; linked as `githubRepo`/`projectPage` on the HF API; posted by an author/official-org comment; owner/org matches the authors' affiliation or the project page domain; project page and repo cross-link each other. Red flags that mean **reject**: the link sits in References / Related Work / a baseline-comparison table; it points to a well-known third-party project (a dataset, a framework, a different paper's repo); it is a **website-template / page-source repo** found in project-page HTML (`eliahuhorwitz/Academic-project-page-template`, `nerfies/nerfies.github.io`, any `*.github.io` page-source or `*-template` repo — see Layer 4); the owner is clearly unrelated to the authors and the project page. A `github.com` literal is required for a GitHub URL; a project page must *not* be a GitHub repo URL, and HF model/dataset/collection pages are **not** project pages (an HF Space can be only if explicitly presented as the project page). When unsure, prefer no write and list the candidate under "needs review". ### Saving Part A results GitHub repos and project pages are saved as **official** (`is_official=true`) and tagged with `source='ai_extraction'` to match the cron's provenance. Use the services directly (they normalize the GitHub URL and skip if an official one already exists): ```bash uv run --env-file backend/keys.env python - <<'PY' from backend.database.connection import get_database_session from backend.services.repository_service import RepositoryService from backend.services.project_page_service import ProjectPageService from backend.utils.github_utils import parse_github_url paper_id = <paper_id> github_url = "<validated github url or ''>" project_page_url = "<validated project page url or ''>" session_gen = get_database_session() db = next(session_gen) try: if github_url: owner, name = parse_github_url(github_url) RepositoryService(db).create_repository_for_paper( paper_id=str(paper_id), url=github_url, owner=owner, name=name, num_stars=0, is_official=True, source="ai_extraction", skip_if_official_exists=True, ) if project_page_url: ProjectPageService(db).create_project_page_for_paper( paper_id=str(paper_id), url=project_page_url, is_official=True, source="ai_extraction", skip_if_official_exists=True, ) db.commit() print("saved", {"github": github_url, "project_page": project_page_url}) finally: try: next(session_gen) except StopIteration: pass PY ```