site-mirror

name

site-mirror

description

Mirrors/clones websites locally by downloading HTML, CSS, JS, and images. Use this skill whenever the user shares a URL and wants to: download a site, clone a site, mirror a page, scrape a website offline, save a site locally, copy a webpage, or archive a URL. Trigger even if the user says things like "baixa esse site", "salva essa página", "clona esse link", "quero esse site offline" or similar. Always use this skill when the user provides a URL and wants it saved locally.

Site Mirror Skill

Mirror any website to a local folder — HTML, CSS, JS, images, and fonts downloaded with full URL rewriting so the site opens offline in your browser.

---

REGRAS DE SEGURANÇA DE CONTEXTO (OBRIGATÓRIO)

Estas regras evitam estourar a janela de contexto. NUNCA as viole.

1. NUNCA ler arquivos de OUTPUT_DIR no contexto (html, css, js, json, img, zip).
2. Ler APENAS o bloco ===*SUMMARY=== / ===END SUMMARY=== do stdout de cada script.
3. Se um script falhar sem imprimir o summary, ler apenas as últimas 5 linhas do stderr.
4. Summary blocks são limitados a 100 linhas pelos próprios scripts.
5. Reportar ao usuário apenas números e paths do summary — nunca conteúdo de arquivos.
6. Se o usuário pedir análise de conteúdo, gerar um script analisador que retorne só um resumo — nunca ler o arquivo diretamente.

---

Step 1 — Install dependencies

pip install requests beautifulsoup4 lxml

If pip isn't available, try pip3. If Python isn't installed, tell the user to install Python from python.org first.

---

Step 2 — Ask the user these questions (all at once, in one message)

Ask concisely:

1. Output folder name — where to save the files (default: a slug based on the URL)
2. Crawl depth — how many link levels to follow (default: 2; use 0 for single page)
3. Paths to exclude — URL path prefixes to skip (default: /blog, /blogs, /news, /tag, /tags)
4. Follow only same-section links? — e.g. /architecture/X stays within /architecture/ (default: yes)
5. Create ZIP? — pack everything into a zip when done (default: no)

For beginners, suggest defaults and ask them to confirm with "ok".

---

Step 2.5 — Detect site type (static vs SPA)

Before generating the mirror script, run the pre-check:

python scripts/detect_site_type.py <TARGET_URL>

Read the one-line result:

• TYPE: static → proceed with mirror_site.py as normal
• TYPE: spa → inform the user the site is JavaScript-rendered, ask if they want Playwright mode (Step 3b). Playwright captures fully rendered HTML but is 5–10× slower.
• TYPE: unknown → proceed with static mode, warn that results may be incomplete if the site is JS-heavy.

Never skip this step. It takes under 5 seconds and uses no extra dependencies.

---

Step 3 — Generate and run the mirror script (static sites)

Use the bundled template at scripts/mirror_site.py as your base. Fill in the user's choices and write the final script to the working directory as mirror_<slug>.py, then run it.

Key parameters to fill in:

• TARGET_URL — the URL the user provided
• OUTPUT_DIR — folder name from Step 2
• EXCLUDED_PATHS — list of path prefixes to skip
• MAX_DEPTH — crawl depth from Step 2
• SECTION_FILTER — path prefix to restrict link-following, or None
• CREATE_ZIP — True or False

---

Step 3b — Playwright mode (for SPA / JavaScript-rendered sites)

If the pre-check returned TYPE: spa and the user agrees, first install:

pip install playwright
playwright install chromium

Then use scripts/mirror_site_playwright.py instead of mirror_site.py. Fill in the same parameters. Warn the user that depth > 1 may take a long time (each page takes 2–5 seconds to render).

---

Step 4 — Report results

After the script finishes, read only the ===MIRROR SUMMARY=== block from stdout and tell the user:

• Where the files were saved (output_dir)
• How many pages were mirrored
• How many assets and CSS files were rewritten
• Any errors
• How to open the site: value of open_with

---

Step 4.5 (Optional) — Serve the mirrored site locally

If the user wants to preview the result in a browser, serve it with Python's HTTP server.

Before starting the server — check if the port is free

On Windows, run:

netstat -ano | findstr ":8080"

If any process is listed, the port is occupied. Try another port (8081, 8082, etc.) or tell the user to free it.

Start the server

python -m http.server 8080 --directory <OUTPUT_DIR>

Run with run_in_background: true. Wait 2 seconds, then validate it started correctly:

curl -s -I http://127.0.0.1:8080/ | head -5

Check that the response contains Server: SimpleHTTP (Python's server). If the header shows a different server (Apache, nginx, IIS), another process is serving on that port — the Python server failed silently. Choose another port and retry.

Tell the user the URL

Always use http://127.0.0.1:PORT/... — never http://localhost/... on Windows. On Windows, localhost may resolve to ::1 (IPv6) instead of 127.0.0.1 (IPv4). If the Python server listens only on IPv4, the browser will connect to a different process and see a wrong or 404 response.

Troubleshooting

Symptom	Cause	Fix
Server exits immediately (code 0)	Port already in use	Check with netstat, pick another port
curl returns 200 but browser shows 404	Multiple processes on same port	Verify Server: SimpleHTTP header
Browser 404 via localhost but 200 via 127.0.0.1	IPv6/IPv4 mismatch on Windows	Always use IP literal 127.0.0.1

---

Step 5 (Optional) — Extract design tokens

After mirroring, ask: "Want me to extract the design tokens (colors, fonts, CSS variables)?"

If yes:

python scripts/extract_tokens.py <OUTPUT_DIR>

Read the ===TOKEN SUMMARY=== block and present the colors, fonts, and CSS variables to the user. Do not read tokens.json — it's for the user to open directly.

---

Step 6 (Optional) — Extract a specific component

Ask the user for a CSS selector (e.g. nav.main-nav, .hero-section, #product-card).

python scripts/extract_component.py <OUTPUT_DIR> "<CSS_SELECTOR>"

Read the ===COMPONENT SUMMARY=== block. Report the paths to component.html and component.css. Do not read those files — tell the user to open them in a browser.

---

Step 7 (Optional) — ZIP packaging

If the user wants to share or archive the result:

python scripts/package_site.py <OUTPUT_DIR>

Read the ===POLISH SUMMARY=== block and report the ZIP path and size.

---

Edge cases

• Site blocks scrapers: The scripts use a realistic User-Agent. If still blocked, the site may require cookies or a login — inform the user.
• JavaScript-heavy sites (not detected): If pages look empty after mirroring, suggest re-running with Playwright mode.
• Large sites: If depth > 2 or thousands of links, warn the user it may take a long time. Suggest increasing REQUEST_DELAY to 1.0.
• HTTPS errors: Handled natively by requests. No extra packages needed.
• Google Fonts / web fonts: mirror_site.py automatically downloads font files and rewrites CSS to use local paths.