design-google-ingest

name	design-google-ingest
version	1.0.0
description	Crawl design.google end-to-end via Edge headless (SPA-aware) — discover every /library article + site page, scrape post-hydration DOM via scripts/edge-mass-scrape.ts, then run the design-google-curator agent to assemble docs/DESIGN.md as the canonical aphrody design reference.
when_to_use	User types "/design-google-ingest", says "scrape design.google", "refresh DESIGN.md", "rebuild the design reference", "ingest design.google", or asks to keep the DESIGN.md in lockstep with what Google publishes at design.google. Pair with /loop weekly ("/loop 7d /design-google-ingest") to keep the cache fresh.

design-google-ingest — One-shot SPA crawl + curate

Mode /goal permanent : décider seul, ne pas demander confirmation, finir le refresh + commit.

Crawls design.google end-to-end, materialises the post-hydration DOM of every discoverable URL into the local edge-cache, then dispatches the design-google-curator agent to assemble docs/DESIGN.md from the raw HTML. The result is a single Markdown file that aphrody contributors can read in 5 minutes to know what Google's design org has shipped + the canonical references each article exposes.

Operating contract

• One invocation = one full refresh. Re-running is idempotent: cached URLs are skipped unless --force is passed.
• SPA-aware crawl. design.google is a Single Page Application; most subroutes ship only the empty shell when fetched without JS. The skill uses scripts/edge-mass-scrape.ts (Microsoft Edge --headless=new --dump-dom --virtual-time-budget=15000) so every page receives the full client-side hydration before snapshot.
• No mocks. The curator only ever cites text that actually appears in the captured DOM. If a page failed to scrape, the curator says so in the audit footer rather than inventing content.
• Two-output split.
  • var/data/edge-cache/<sha256-of-url>.html — raw HTML per URL (gitignored, regenerable).
  • docs/DESIGN.md — distilled Markdown ledger (checked into git).

The pipeline, one tick

1. Discover URLs (scripts/design-google-ingest.ts):
     - Re-scrape `https://design.google/library` for the freshest
       /library/* href list.
     - Union with the canonical seed list
       (home, about, events, products, library).
     - Drop duplicates, normalise to lowercase, write
       `scripts/design-google.urls.json` (schema v1).

2. Run mass-scrape:
     bun run scripts/edge-mass-scrape.ts
       --urls=scripts/design-google.urls.json
       --cache=var/data/edge-cache
       --concurrency=4
       --virtual-time=15000
       --retry=2 --timeout=60000
     -> manifest.json (engine=edge) lands next to the HTML files.

3. Dispatch the `design-google-curator` agent:
     prompt = "Read every URL in scripts/design-google.urls.json,
               match it to its cached HTML, extract title + meta
               description + first prose paragraph + every #RRGGBB
               color stop appearing in <code> or inline style, then
               assemble docs/DESIGN.md per the SKILL.md output schema."

4. Verify:
     - `docs/DESIGN.md` exists, non-empty.
     - Every URL in scripts/design-google.urls.json maps to either a
       successful section or an explicit `> _Scrape failed_` block
       in DESIGN.md.
     - Run `bun run scripts/m3-coverage-audit.ts` to update the
       coverage table that DESIGN.md links to.

5. Commit (single-deliverable):
     scripts/design-google.urls.json + docs/DESIGN.md +
     docs/audits/m3-coverage.md (regenerated).

6. ScheduleWakeup({ delaySeconds: 604800, prompt: "/design-google-ingest",
                   reason: "weekly refresh design.google cache" })
     — only when invoked under /loop.

docs/DESIGN.md output schema

<!-- SPDX-License-Identifier: Apache-2.0 -->
<!-- GENERATED by .claude/skills/design-google-ingest — DO NOT EDIT BY HAND. -->

# Aphrody Design Reference

Last refreshed: <ISO-8601 UTC timestamp>
Source: design.google + scripts/edge-mass-scrape.ts (Edge headless, virtual-time=15000)
Articles ingested: N / M (M = URLs requested)

## 1. Quick-reference index
| Section | Source URL | Bytes |
|---|---|---|
| ... | ... | ... |

## 2. Foundations
(Articles in the design-systems / typography / accessibility / color categories.)

## 3. Material 3
(Articles tagged m3, material-design-3, design-tokens.)

## 4. Gemini visual identity
(gemini-ai-visual-design + related.)

## 5. Brand assets
(google-sans-flex-font, open-source-custom-fonts-google, transparent-screens.)

## 6. Other library articles
(Everything else, alphabetical.)

## 7. Site pages
(home, about, events, products.)

## 8. Raw failures
(URLs that returned <= 5 KB or an empty shell — needs re-run.)

## 9. Audit
- M3 coverage snapshot (token=X% bridge=Y% catalogue=Z% html=W% overall=V%)
- Aphrody bridge mappings already in lock-step.
- Open follow-ups.

Files this skill touches

Path	Direction	Role
scripts/design-google.urls.json	write	fresh URL list
scripts/design-google-ingest.ts	invoke	URL discovery + crawl orchestration
scripts/edge-mass-scrape.ts	invoke	SPA-aware scrape engine
var/data/edge-cache/<sha256>.html	write	per-URL DOM cache (gitignored)
var/data/edge-cache/manifest.json	write	run manifest with per-URL status
docs/DESIGN.md	write	distilled Markdown reference
docs/audits/m3-coverage.md	refresh	coverage table linked from DESIGN.md

When NOT to use

• For single-URL scrapes (use scripts/edge-mass-scrape.ts directly).
• For Linux/Mac native max-power scrape — use Chrome headless.

Related skills / agents / references

• design-google-curator agent (.claude/agents/design-google-curator.md) — the agent the skill dispatches to read the cache + write DESIGN.md.
• scripts/edge-mass-scrape.ts — the underlying SPA-aware scraper.
• scripts/m3-coverage-audit.ts — coverage snapshot consumed by §9.
• docs/audits/2026-05-17-edge-headless-spa-fallback.md — why we use Edge instead of a static scraper for design.google.
• docs/audits/2026-05-17-gemini-clone-and-google-sans-flex.md — prior application of design.google intel to crates/m3-tokens.