develop-web-translator

## Prerequisites Fetch and read the Zotero translator documentation: - https://www.zotero.org/support/_export/raw/dev/translators - https://www.zotero.org/support/_export/raw/dev/translators/coding Also read `index.d.ts` in the repo root for type definitions. Give more weight to **recently created** translators when looking for examples. ## Step 1: Gather information Collect from the user: 1. **Label**: The translator name (usually the site name) 2. **Creator**: The author's name 3. **Target URL(s)**: One or more example URLs from the target site From the URLs, derive the **target regex**. ## Step 2: Analyze the site **DO NOT** fetch site pages with WebFetch, curl, or any HTTP tool. Use the tools instead: ``` node .bin/capture-har.mjs "<example url>" ``` **Read the generated YAML file.** It contains full API schemas. This is your source of truth. ``` node .bin/inspect-page.mjs "<example url>" ``` This gives you meta tags, accessibility tree, and screenshot. ### Difficult sites (anti-bot walls) The browser tools (`capture-har`, `inspect-page`, `create-test`, `run-tests`) run **headless** by default. If a site is behind Cloudflare, a captcha, or another anti-bot wall, add `--headed`; the tool waits for any challenge to clear, then continues. (`--interact` and `--keep-open` also run headed.) `--headed` is not just a fallback for solving captchas by hand — on Cloudflare-protected publishers it is the only thing that works, because headless Chrome announces itself as `HeadlessChrome` and is refused outright. Headed usually loads those pages with no challenge at all. Reach for it as soon as a site returns "Just a moment…", a "Validate User" page, or content that isn't there. Do not try to defeat these walls by faking the browser: passing `--user-agent` makes Chrome drop its `Sec-CH-UA` client hints, which is a *worse* signal than the one it hides. Playwright's bundled Chromium can't get through at all, headed or not, which is why the tools drive real Chrome. A solved challenge is cached in a reused browser profile at `.tmp/browser-profile`, so the next run carries it over. Don't clear that profile routinely — you throw the clearance away and have to solve everything again; the tools already clear the stale *extension* state on their own. Clear it only if a run hangs on a profile lock or the cached state is clearly bad: ``` rm -rf .tmp/browser-profile ``` ## Step 3: Choose an approach Check the `inspect-page` meta tags first: 1. **Embedded Metadata (EM)** — if the page has Highwire Press tags (`citation_title`, `citation_author`, `citation_doi`, etc.), Dublin Core (`DC.title`, etc.), or good JSON-LD with bibliographic data, use EM. This is the most common approach (~180 translators use it): ```js async function scrape(doc, url = doc.location.href) { let translator = Zotero.loadTranslator('web'); translator.setTranslator('951c027d-74ac-47d4-a107-9c3069ab7b48'); // EM translator.setDocument(doc); translator.setHandler('itemDone', (_obj, item) => { // fix up fields EM gets wrong item.complete(); }); await translator.translate(); } ``` Call `await translator.getTranslatorObject()` only if you need to customize EM before translation (e.g. setting `itemType`). 2. **DOI search** — if the page doesn't have rich metadata but you can extract a DOI, use a search translator to look it up via DOI Content Negotiation: ```js async function scrape(doc, url = doc.location.href) { let doi = doc.querySelector('a[href*="/doi/"]')?.href.match(/10\.\d{4,}\/[^\s]+/)?.[0]; if (!doi) return; let translate = Zotero.loadTranslator('search'); translate.setSearch({ DOI: doi }); translate.setHandler('error', () => {}); translate.setHandler('itemDone', (_obj, item) => { item.complete(); }); await translate.translate(); } ``` 3. **API-based** — the site has a clean JSON API visible in the YAML. Call it with `requestJSON()`. 4. **HTML scraping** — no useful APIs or metadata. Parse the DOM directly. Last resort. 5. **Hybrid** — combine any of the above. ## Step 4: Initialize and write code ``` node .bin/init-translator.mjs --label "<Label>" --creator "<Creator>" --target "<regex>" --type web ``` This scaffolds the file from the web translator template at `.bin/templates/web.js`. That file is the canonical structure a web translator should follow — read it when you need to know the expected shape of `detectWeb`/`getSearchResults`/`doWeb`/`scrape`, or when a task asks you to make an existing translator better conform to the template. Implement `detectWeb(doc, url)`, `getSearchResults(doc, checkOnly)`, `doWeb(doc, url)`, and `scrape(doc, url)`. ## Step 5: Create tests ``` node .bin/create-test.mjs "<Label>.js" --url "<example url>" ``` Include at least one single-item test and one multiple-item test (if supported). ## Step 6: Verify and submit **Update `lastUpdated` every time you modify translator code.** Zotero uses it to determine when to push updates to users. ``` node .bin/update-metadata.mjs "<Label>.js" npm run lint -- "<Label>.js" node .bin/run-tests.mjs "<Label>.js" ``` All tests must pass. Then create a branch and PR.