food-fetch

# Food-Fetch — Lawful Full-Text Acquisition Turn a list of references or DOIs into **read full text**, so `food-research`, `food-deep-research`, and `food-review` build their knowledge base from the actual papers rather than abstracts. Original work; the routing/manifest architecture is informed by the open-source `nature-downloader` skill (see the repo README Acknowledgements). Legal-access-only, and **it never fabricates** — a paper that was not obtained is recorded as not obtained, never summarized as if read. ## First-run setup — ask once, remember, remind if skipped On the **first** run that needs full text, help the user set up access — this is what decides whether the review reads real papers. Check `python3 scripts/food_fetch_setup.py status` (exit 3 = not set up): - If **not set up**, surface a **highlighted request** offering the choices, then save the answer so it isn't asked again: > 📚 **Set up full-text access (one-time).** To read papers, not just abstracts, > pick one: > 1. **Reference-manager library** — send the path to your **EndNote `.Data` > folder** (or Zotero `storage/` / Mendeley). Best coverage for cited work. > 2. **Institutional access** — I'll reach entitled full text through **your own > logged-in library session** in the browser when needed. > 3. **Open access only** — no setup, but see the warning below. > > ⚠️ **Without access to non-open-access articles, the accuracy of the results is > substantially limited** — most published papers are paywalled, so the knowledge > base and every claim's grounding would rest on abstracts + the ~half of literature > that is open access. Save the choice: `food_fetch_setup.py set --library <path>` / `--institutional` / `--open-access-only`. - If **already set up** (exit 0), use the saved mode silently — **do not re-ask**. - If the saved mode is **`open_access_only`**, **remind** at the start of each full-text run — briefly, once — that non-OA access is not configured and that this **substantially limits accuracy**, and offer to switch (options 1–2). Don't block; proceed at open-access + abstract level. The store holds only a folder path and a mode — **never** a password, cookie, or token. ## Access ladder (each article, in order) — see `references/access-routing.md` 1. **Open access** — resolve a legal free PDF (Unpaywall → OpenAlex; PMC/Europe PMC full text; arXiv/preprints) and **download it**. `oa_fetcher` runs `scripts/fetch_oa.py`. **Open-access full text is always fetched and read — never left at abstract-level.** 2. **The user's reference-manager library** — `library_fetcher` reads the PDFs the user already downloaded in **EndNote** (`<Library>.Data/PDF/`), **Zotero** (`storage/`), or **Mendeley**, mapping each reference by DOI/title. Read-only. 3. **The user's entitled institutional access** — `institutional_fetcher` reuses the user's **own logged-in library session in the browser** (Claude-in-Chrome) to open the article through their library resolver / EZproxy / Shibboleth-CARSI / WebVPN and save the entitled PDF. **User-driven login only.** See `references/institutional-access.md`. 4. **User-supplied PDFs** — anything the user drops in the project folder. If none reaches it, the article is `oa_not_found` / `library_no_permission` — the user is told plainly, and the source stays unread (not summarized). ## Subagents (dispatch via the Agent tool) 1. **`fetch_coordinator`** — takes the reference list / DOIs, confirms the **Supporting Information** choice once, runs the batch, and returns the manifest + a plain summary of read vs unread. 2. **`access_router`** — per article, normalize the DOI/title, detect open-access vs paywalled and publisher, and choose the ladder step (`references/access-routing.md`). 3. **`oa_fetcher`** — download every open-access PDF via `scripts/fetch_oa.py`; verify each is a real PDF (`%PDF`); record source + SHA-256. 4. **`library_fetcher`** — match references to PDFs in the user's reference-manager library folder and read them (read-only). 5. **`institutional_fetcher`** — with the user's own browser session, reach the library-entitled full text and save the PDF; hand off to the user for any login/CAPTCHA/2FA (`references/institutional-access.md`). 6. **`content_reader`** — normalize each obtained article (XML/HTML/PDF) into clean sectioned full text, **preferring structured JATS XML over scraped PDF** (`references/format-reading.md`); never treat an abstract or a login page as the paper. ## Deliverable — a coverage manifest (`references/manifest-and-status.md`) Always return one manifest listing **every** requested reference with a typed status (`open_access_downloaded` · `library_pdf_read` · `institutional_downloaded` · `oa_not_found` · `library_no_permission` · `pdf_fetch_failed` · `not_a_pdf`), its access route, and file/SHA-256 where downloaded — plus a one-line count of how many were read in full vs not. This is what makes the review's grounding honest and auditable. Run `scripts/privacy_scan.py` on anything delivered. ## Boundaries (non-negotiable) — `references/boundaries.md` Lawful access only: legal OA, the user's own library files, and the user's own authorized institutional session. **Never** bypass a paywall, DRM, or 2FA; never read, type, export, or ask for passwords / cookies / OTP codes / session tokens; never scrape against a site's terms; no bulk/indiscriminate downloading — only the confirmed list. For a login or verification challenge, hand the browser to the user. ## Called by `food-research` (`screener_appraiser`/`data_extractor`), `food-deep-research` (`investigator`/`source_verifier`), and `food-review` (`knowledge_builder`) call food-fetch to obtain full text before extracting; the `agri-*` skills via delegation.