connector-docs-import

name

connector-docs-import

description

(Re-)import Jacob's connector-doc web — the recursive network of *.jacobcole.net Google Docs (systematicawesome / "Systematic Awesome" universe) — into WikiHub at @jacobcole/systematic-awesome, as faithfully (1:1) as possible, with a backup snapshot taken FIRST. Use when the user says "re-run the systematic awesome import", "refresh my connector docs on wikihub", "re-import the jacobcole.net docs", or "sync my Google Docs web into the wiki".

Connector-docs → WikiHub import

Faithfully mirror the *.jacobcole.net Google-Docs web into @jacobcole/systematic-awesome. Goal: as 1:1 as possible. Do not restructure, rename, or summarize unless the user explicitly asks.

Where everything lives (the reference Jacob asked for)

Thing	Location
Source scrape (HTML, one file per doc)	noos-prod:~/noos/backups/google-docs/html/*.html (ssh noos-prod, AWS Lightsail 3.216.129.34)
Scrape manifest (filename → source URL + docId)	noos-prod:~/noos/backups/google-docs/export-results.json (also export-results-v2.json, dead-links-report.json)
Last scrape date	recorded as SCRAPE_DATE in the import script (2026-01-30 as of writing)
Local working copy used by the importer	/tmp/sa-scrape/ (*.html + export-results.json)
Scraper (refreshes from Google)	scripts/scrape_gdocs.py
Importer (HTML → markdown → WikiHub)	scripts/import_systematicawesome.py
Import API key	~/.config/wikihub/jacobcole-import-key.txt
Destination	wiki @jacobcole/systematic-awesome on https://wikihub.md

How the importer works (so "1:1" is understood)

scripts/import_systematicawesome.py:

• one wiki page per HTML file; filename stem → page path (<stem>.md)
• cleans HTML (drops <style>/class=, unwraps Google redirect URLs), then pandoc -f html -t gfm → markdown
• converts internal *.jacobcole.net links to [[wikilinks]] when the target exists in the scrape
• writes frontmatter: title, source_url, source_gdoc, scraped_at, imported_by: wikihub-q7gz, visibility: public
• idempotent + safe: updates its own prior imports in place (matches on imported_by: wikihub-q7gz); if a page exists that is NOT one of our imports, it writes to <stem>.import.YYYY-MM-DD.md and flags it for review — it never silently clobbers human edits.

Known fidelity gaps (call out to the user, don't silently accept):

• inline images (data: URIs) are omitted (size limit) — replaced with a note
• pandoc GFM normalizes some formatting
• WikiHub's Cloudflare WAF rejects page writes containing shell one-liners (curl … | bash, npm install -g …); such pages fail with an HTML 403. Workaround: write those via the origin directly (the GCP wikihub-prod box's local API on 127.0.0.1:5100, bypassing Cloudflare) — see docs/deploy.md for SSH.

Procedure

0. SNAPSHOT FIRST — never import without a fresh backup

A nightly backup already runs (wikihub-backup.timer → GCS bucket wikihub-backups-932822f5). Before a big import, force an on-demand one:

gcloud compute ssh wikihub-prod --project=wikihub-prod --zone=us-east1-b \
  --tunnel-through-iap --command='sudo systemctl start wikihub-backup.service \
    && journalctl -u wikihub-backup.service --no-pager -n 5'

Rollback companion if needed: scripts/restore.sh (see docs/backup-and-restore.md).

1. (Optional) refresh the scrape from Google

Only if you want newer-than-last-scrape content. Needs Google auth; run on noos-prod:

ssh noos-prod 'cd ~/noos && python3 <path>/scrape_gdocs.py'   # see scripts/scrape_gdocs.py

If you skip this, you re-import the existing 2026-01-30 scrape verbatim.

2. Sync the scrape locally

mkdir -p /tmp/sa-scrape
rsync -az noos-prod:~/noos/backups/google-docs/html/ /tmp/sa-scrape/
scp noos-prod:~/noos/backups/google-docs/export-results.json /tmp/sa-scrape/
ls /tmp/sa-scrape/*.html | wc -l   # sanity: expect ~70+ docs

3. Dry-run, then import

python3 scripts/import_systematicawesome.py --dry-run      # prints per-page plan
python3 scripts/import_systematicawesome.py                # ~5 req/s, idempotent
python3 scripts/import_systematicawesome.py --only=<stem>  # single page (good for testing)

4. Verify

• counts line at the end: created / updated / skipped / flagged / errors
• spot-check a few pages render and that [[wikilinks]] resolve
• review anything under *.import.YYYY-MM-DD.md (collisions with non-import pages)

Testing the skill cheaply

Use --only=<stem> (e.g. --only=thoughtfulweb) for a single-page round-trip before a full run. Combine with --dry-run first.