add-external-paper

# Add External Paper This skill creates or updates a `papers` row with `source = 'external'` for research outputs that are paper-like but are not backed by an arXiv ID. The goal is a complete local paper page: title, abstract or short summary, publication date, authors when available, appropriate task and method tags from the current taxonomy, official GitHub repository links, project pages, explicit Hugging Face artifacts, and a paper thumbnail. Use the existing application paths first. Direct SQL is acceptable for fields that have no admin API surface, but keep writes idempotent and scoped. ## Pre-flight Run commands from the repo root and load the project environment: ```bash uv run --env-file backend/keys.env ... ``` Confirm the API is reachable and the admin token exists: ```bash curl -s -o /dev/null -w "%{http_code}\n" http://localhost:8000/api/v1/papers/recent?limit=1 grep -E '^PWC_ADMIN_API_TOKEN=' backend/keys.env | sed 's/=.*/=<redacted>/' ``` If the API is unavailable, use the direct database path where the operation supports it or report the blocker. Do not start or restart deployed services during data curation. Use `ADMIN_TOKEN=$(grep '^PWC_ADMIN_API_TOKEN=' backend/keys.env | cut -d= -f2-)` for API writes with `X-Admin-Token`. ## Scope Decision 1. If the source has a real arXiv ID for the same release, do not create an external duplicate. Use the normal arXiv indexing flow, then add repos/artifacts/tags to that arXiv paper. 2. If the source is a paper-like external release with no arXiv ID, create or reuse an external row keyed by canonical `source_url`. 3. If the source is only a related code repo or model card for an existing paper, attach it to the existing paper instead of making a new row. Be strict: citations, base-model references, forks, or benchmark mentions are not enough to claim the source introduces the paper. An external release may acquire an arXiv paper later. The daily importer then canonicalizes it onto the arXiv row and leaves the old numeric route as a redirect. Verify that this merge preserves explicit HF arrays and normalized `paper_hf_artifacts` links along with evaluations, repositories, project pages, tasks, and methods; otherwise the paper can silently lose its HF trending component after canonicalization. Canonical `source_url` should be the most durable primary surface for the release: - **The paper PDF whenever one exists**, even when a project page or landing page also exists. Make the PDF the `source_url` and put the project/landing page in `project_pages`. Example: paper 123404 uses `https://sunovivid.github.io/correction-training/paper.pdf` as `source_url` and the project page as a project link. - Technical report or blog post over a GitHub README when no PDF exists. - HF model card when the model card is the release surface. - GitHub repo when the README is the only technical report. ## Collect Source Metadata Gather evidence before writing. Save temporary source text under `/tmp/<slug>/` when useful. For GitHub repositories: ```bash mkdir -p /tmp/<slug> gh repo view <owner>/<repo> --json nameWithOwner,description,url,homepageUrl,createdAt,pushedAt,repositoryTopics,stargazerCount > /tmp/<slug>/repo.json gh api repos/<owner>/<repo>/readme -H "Accept: application/vnd.github.raw" > /tmp/<slug>/README.md ``` For Hugging Face models: ```bash mkdir -p /tmp/<slug> uv run --env-file backend/keys.env hf models info <repo_id> > /tmp/<slug>/hf-info.json uv run --env-file backend/keys.env hf models card <repo_id> --text > /tmp/<slug>/card.md ``` Check HF metadata for `arxiv:` tags. If a tag clearly points to the source paper and the local DB has that arXiv paper, use the arXiv row rather than creating an external row. For webpages, PDFs, blogs, and vendor reports, try the built-in metadata extractor: ```bash uv run --env-file backend/keys.env -m backend.genai_steps.extract_external_resource_metadata --url "<source_url>" ``` The extractor returns title, abstract/summary, authors, date, GitHub URLs, and HF artifact URLs. Treat its output as a draft: verify against the source text and fill gaps manually from the page, README, card metadata, release notes, or visible dates. For PDFs, if extraction returns no readable text or only a generated summary, download the PDF and extract text with `pdftotext -layout` (or another structured PDF text extractor already available in the environment) before writing metadata. ## Metadata Rules - `title`: use the actual release/report title, not browser chrome like `GitHub - owner/repo`. - `abstract`: use the source's explicit abstract verbatim whenever one is present. For PDFs, inspect the extracted first pages before summarizing; do not replace an abstract-like opening block with a generated summary just because the metadata extractor failed. If there is no explicit abstract, executive summary, TL;DR, or README overview, write a concise 2-5 sentence summary from the source and make it clear in the report that the abstract was manually summarized. - `date_published`: required for external rows. Use the release/report publication date, not the scrape date. For GitHub-only releases, prefer a dated release/tag or README announcement date over repository creation time. If no credible date can be found, stop and ask the user. - `authors`: include human authors only. Do not store organization names such as "OpenAI", "Sapient", or "Hugging Face" as authors. - `github_repository_urls`: official GitHub repositories linked by the source or provided by the user. Only include repos that the source presents as official code/tools for the release; passing a URL here marks the paper-repository relationship as official. If additional GitHub URLs are merely dependencies, citations, examples, or benchmark/framework references, either leave them unlinked or link them separately with `is_official = false` rather than mixing them into this field. - `hf_artifact_urls`: full `https://huggingface.co/...` URLs for model, dataset, and space artifacts. Store absolute URLs, not bare repo ids. - `project_pages`: official non-GitHub landing pages, demos, docs, reports, or blog/project pages connected to the release. Do not duplicate the canonical `source_url` as a project page unless it is useful as a project link in addition to being the paper source. For model-family releases, add the family/collection page and technical report as project pages, but keep individual HF model cards in `hf_artifact_urls`; the frontend groups project pages behind one "Project page" action and expands HF artifacts separately. ## Create Or Update The External Row Prefer a small Python script that uses `ExternalResourceService`, because it normalizes GitHub URLs and classifies HF artifacts into `hf_models`, `hf_datasets`, and `hf_spaces`. ```bash uv run --env-file backend/keys.env python - <<'PY' from datetime import date from backend.database.connection import get_database_session from backend.services.external_resource_service import ( ExternalResourceInput, ExternalResourceService, ) payload = ExternalResourceInput( source_url="<canonical source URL>", title="<clean title>", abstract="<explicit abstract or short source-grounded summary>", authors=["<human author>", "..."], date_published=date(YYYY, MM, DD), github_repository_urls=["https://github.com/<owner>/<repo>"], hf_artifact_urls=[ "https://huggingface.co/<org>/<model>", "https://huggingface.co/datasets/<org>/<dataset>", "https://huggingface.co/spaces/<org>/<space>", ], ) session_gen = get_database_session() db = next(session_gen) try: result = ExternalResourceService(db).add_external_resource(payload) print(result.paper.id, result.paper.title) finally: try: next(session_gen) except StopIteration: pass PY ``` This is idempotent by `(source='external', source_url)`: re-running updates mutable fields and merges links. If the page can be fetched and the automatic extractor has enough metadata, `add_from_url` is also fine: ```python result = await ExternalResourceService(db).add_from_url( source_url="<canonical source URL>", date_published=<manual date if extractor cannot infer it>, title_override="<optional clean title>", abstract_override="<optional summary>", github_repository_urls_extra=["https://github.com/<owner>/<repo>"], hf_artifact_urls_extra=["https://huggingface.co/<org>/<model>"], ) ``` ## Generate Thumbnail After the external row exists, generate a thumbnail before finishing unless `papers.thumbnail_url` is already set and the cached file still serves. External paper cards and social previews look incomplete without one. Pick the render source URL in this order: 1. A direct PDF URL linked by the source (for example `https://aclanthology.org/Q19-1026.pdf` when the canonical `source_url` is the HTML landing page). 2. The canonical `source_url` when it is itself a PDF. 3. The canonical `source_url` for HTML pages, blogs, GitHub README surfaces, or HF model cards. `ThumbnailService` falls back to a first-viewport screenshot when no PDF is available. Prefer PDF first-page rendering over HTML screenshots whenever a credible PDF exists. ```bash uv run --env-file backend/keys.env python - <<'PY' import asyncio from backend.database.connection import get_database_session from backend.models.database import Paper as DBPaper from backend.services.thumbnail_service import ( ThumbnailService, build_custom_thumbnail_url, ) paper_id = <paper_id> thumbnail_source_url = "<PDF or page URL to render>" async def main() -> None: service = ThumbnailService() filename = service.get_external_filename(paper_id) existing_url = service.get_external_thumbnail_url(paper_id) if existing_url: session_gen = get_database_session() db = next(session_gen) try: paper = db.get(DBPaper, paper_id) if paper is not None and not paper.thumbnail_url: paper.thumbnail_url = existing_url db.commit() print("thumbnail_url", existing_url, "status=cached") finally: try: next(session_gen) except StopIteration: pass return await service.generate_external_thumbnail( paper_id, thumbnail_source_url, force=False, ) thumbnail_url = build_custom_thumbnail_url(filename) session_gen = get_database_session() db = next(session_gen) try: paper = db.get(DBPaper, paper_id) paper.thumbnail_url = thumbnail_url db.commit() print("thumbnail_url", thumbnail_url, "status=generated") finally: try: next(session_gen) except StopIteration: pass asyncio.run(main()) PY ``` Verify the image is reachable: ```bash curl -s "http://localhost:8000/api/v1/papers/<paper_id>" | uv run python -c "import json,sys; p=json.load(sys.stdin); print(p.get('thumbnail_url'))" curl -s -o /dev/null -w "%{http_code}\n" "http://localhost:8000/api/thumbnails/custom/external_<paper_id>.jpg?fit=fit4" ``` If generation fails because headless Chrome or PDF tooling is unavailable, note the failure in the report. The `preload_thumbnails` cron may backfill later, but do not skip an inline attempt during ingestion. ## Tag With Existing Tasks And Methods Only use tasks and methods already present in the taxonomy unless the user explicitly asks to expand the taxonomy. If the right concept is missing, choose the closest existing tag and report the gap. Inspect candidate tasks: ```bash uv run --env-file backend/keys.env python - <<'PY' from sqlalchemy import text from backend.database.connection import get_engine terms = ["text generation", "language model", "reasoning", "biology"] with get_engine().connect() as conn: for term in terms: rows = conn.execute(text(""" SELECT id, name, slug, level FROM tasks WHERE name ILIKE :q OR slug ILIKE :q ORDER BY level NULLS LAST, name LIMIT 20 """), {"q": f"%{term}%"}).fetchall() print(term, [dict(r._mapping) for r in rows]) PY ``` Inspect candidate methods: ```bash uv run --env-file backend/keys.env python - <<'PY' from sqlalchemy import text from backend.database.connection import get_engine terms = ["transformer", "latent reasoning", "hierarchical reasoning", "diffusion"] with get_engine().connect() as conn: for term in terms: rows = conn.execute(text(""" SELECT id, name, slug, keywords FROM methods WHERE name ILIKE :q OR slug ILIKE :q OR keywords ILIKE :q ORDER BY name LIMIT 20 """), {"q": f"%{term}%"}).fetchall() print(term, [dict(r._mapping) for r in rows]) PY ``` Apply tags through the admin API when you have an authorized browser session. These endpoints replace the full set, so first read existing tags and merge unless intentionally retagging: ```bash curl -s "http://localhost:8000/api/v1/papers/<paper_id>?include_resources=true" | uv run python -m json.tool curl -s -X PUT "http://localhost:8000/api/v1/papers/<paper_id>/tasks?task_ids=<id>&task_ids=<id>" \ -H "X-Admin-Token: $ADMIN_TOKEN" curl -s -X PUT "http://localhost:8000/api/v1/papers/<paper_id>/methods?method_ids=<id>&method_ids=<id>" \ -H "X-Admin-Token: $ADMIN_TOKEN" ``` If the local API rejects `X-Admin-Token` because the endpoint requires an HF admin session, use the CLI instead of hacking auth. It replaces the whole set, so pass the final list of ids: ```bash uv run --env-file backend/keys.env pwc paper update <paper_id> tasks <task_id> [<task_id> ...] uv run --env-file backend/keys.env pwc paper update <paper_id> methods <method_id> [<method_id> ...] ``` (`PaperService.update_paper_tasks` / `update_paper_methods` no longer exist; do not call them.) Prefer a few high-confidence tags over broad keyword stuffing. A good external paper page should usually have 1-4 task tags and 1-5 method tags. ## Add Repositories And Project Pages `ExternalResourceService` already links GitHub repositories passed in the payload as official. If you need to adjust or add more after creation, use the API: ```bash curl -s -X PUT "http://localhost:8000/api/v1/papers/<paper_id>/repositories" \ -H "Content-Type: application/json" \ -H "X-Admin-Token: $ADMIN_TOKEN" \ -d '{"repositories":[{"url":"https://github.com/<owner>/<repo>","is_official":true}]}' curl -s -X PUT "http://localhost:8000/api/v1/papers/<paper_id>/project-pages" \ -H "Content-Type: application/json" \ -H "X-Admin-Token: $ADMIN_TOKEN" \ -d '{"project_pages":[{"url":"https://example.com/project","is_official":true}]}' ``` For project pages, include only useful official release surfaces: demo pages, report landing pages, docs, blog/project homepages, and model family pages. Avoid adding every linked dependency, unrelated citation, or social URL. When a release has multiple official surfaces, put them all in `project_pages`; users see them under one fixed "Project page" action. For HF model families, put the HF collection in `project_pages` and every concrete model/dataset/space card in `hf_artifact_urls` so users can click each artifact instead of being sent only to the first model. ## Add Or Fix HF Artifacts `ExternalResourceService` synchronizes explicit URLs into both the paper arrays and the normalized `hf_artifacts` / `paper_hf_artifacts` tables during ingestion. For an existing row, use the repository CLI instead of updating the arrays directly: ```bash uv run --env-file backend/keys.env pwc paper update <paper_id> hf-artifacts \ --model "https://huggingface.co/<org>/<model>" \ --dataset "https://huggingface.co/datasets/<org>/<dataset>" \ --space "https://huggingface.co/spaces/<org>/<space>" \ --json ``` The command is replace-style, so read and preserve existing artifacts unless