Use when fetching a web page/URL or converting a saved .html/.htm/.mhtml/.webarchive to clean Markdown — an Obsidian web-clipper and a universal HTML acquisition + HTML→Markdown step that also feeds the pdf and docx skills. Triggers include "html to markdown", "url to markdown", "download this page", "save the html", "web page to obsidian", "webarchive/mhtml to markdown", "clip this article".
Use when fetching a web page/URL or converting a saved .html/.htm/.mhtml/.webarchive to clean Markdown — an Obsidian web-clipper and a universal HTML acquisition + HTML→Markdown step that also feeds the pdf and docx skills. Triggers include "html to markdown", "url to markdown", "download this page", "save the html", "web page to obsidian", "webarchive/mhtml to markdown", "clip this article".
tier
2
version
1.1
license
LicenseRef-Proprietary
html skill
Purpose: Convert a web URL or a downloaded .html/.htm/.mhtml/.webarchive
into clean Markdown — with YAML frontmatter and a shared _attachments/ folder —
for two consumers: (1) an Obsidian web-clipper (self-contained note), and
(2) a universal HTML→Markdown step any agent workflow can call.
1. Red Flags (Anti-Rationalization)
"I'll just paste the HTML and convert it in my head" → WRONG. The script
reuses the docx-mastered turndown core (GFM tables, rowspan→flat grid) and the
pdf-mastered cleaner (reader-mode, SPA-chrome strip); reimplementing in prose
regresses on every edge case.
"I'll fetch the page with curl and strip tags with regex" → WRONG. Use the
script — it has SSRF protection, dual-output, sha1-deduped attachments.
2. Capabilities
URL → Markdown via a resilient fallback ladder (--engine lite|chrome|auto|jina|remote):
httpx+trafilatura lite fetch (also yields title/date/author) with retry + backoff +
429/Retry-After and a 403 → browser-UA escalation. --engine auto (default) is
local-first (lite → chrome → remote last-resort); --engine jina|remote is
remote-first with automatic local fallback. No single tier is a point of failure:
if a remote reader is down / rate-limited / quota-exhausted, the run falls back to the next
provider then to the local engines; only when every viable tier is exhausted does it fail
with one typed FetchFailed (kind=all_engines_failed) carrying a details.tried trace.
Vendor-agnostic remote reader (--engine jina|remote): the remote tier is a pluggable
provider layer — jina (r.jina.ai) is the built-in default, but HTML_READER_URL /
HTML_READER_PROVIDERS point it at a self-hosted Jina or any compatible reader, so
resilience does not depend on any single vendor. --engine remote REQUIRES a configured
provider (never a silent fall-back to jina.ai). --no-remote disables the remote tier
entirely. --remote-format markdown trusts the reader's own clean Markdown; --target-selector
extracts just the article block. --rate-limit throttles fetches.
Web search → Markdown (--search "QUERY" [OUTPUT_DIR] [--max-results N]): a
vendor-agnostic search provider (s.jina.ai default; HTML_SEARCH_URL /
HTML_SEARCH_PROVIDERS override) returns the top results; each result URL is fetched
through the same fallback ladder (so every result inherits per-result fallback) and
written as one note (frontmatter query: + source:). A failed result is skipped, not
fatal; a healthy zero-result search exits 0.
Authenticated (login-gated) Chrome (--engine chrome + auth): read pages behind a login
(X Articles/threads, paywalled/members, private docs) by replaying a human-minted session.
Mint once: (headful; 2FA ok). Then convert with
(portable, , read-only →
concurrency-safe), (cookie-only), or
(local persistent profile).
pulls lazy content (replies). The Chrome tier is (private / off-target-public
redirects refused; non-public sub-resources aborted); a stale session → . Auth
is strictly — with none configured, behaviour is unchanged (no crash). The target
URL + session stay (no third party). See (Hermes deploy).
3. Execution Mode
Mode: script-first.
Why this mode: HTML→Markdown is a deterministic, edge-case-heavy pipeline
(fetch → clean → turndown → emit) reusing hardened docx/pdf code. Inline agent
conversion regresses on tables, SPA chrome, encodings, and image handling, and
has no SSRF protection.
4. Script Contract
Two operations + a combined one-shot. The skill exposes the pipeline as composable
verbs so a fetched page can feed Markdown, pdf, or docx:
python3 scripts/html fetch INPUT [OUTPUT_DIR] — OP1: download to an on-disk
<slug>.html + <slug>.meta.json sidecar (+ localized _attachments/). The HTML is
sanitized (file:/javascript: refs stripped) so it is safe to render; an
authenticated fetch's body is written 0600. Keep the HTML to feed the pdf skill.
python3 scripts/html md INPUT [OUTPUT_DIR] — OP2: convert a fetched artifact /
local HTML / URL → Markdown. A local .html with a sibling .meta.json recovers full
frontmatter from the sidecar.
python3 scripts/html get URL (OUTPUT_PATH | --stdout) — OP3: download a URL to
raw bytes, verbatim, through the same SSRF-guarded ladder. No conversion, no
sanitizing, no content sniffing — the caller decides what the bytes are. Use it when
you need the file itself (a PDF to hand to the pdf skill, an image, any binary):
OP1/OP2 refuse a %PDF- payload on purpose, because turndown blows its call stack on
binary input, and before OP3 existed a caller needing those bytes had nothing to shell
out to and fell back to an unguarded raw HTTP GET. OP3 bypasses no guard — it calls
_assert_safe_target + _http_get_bytes, the same two the text path uses, and omits
only the content-type layer above them. Egress is direct unless HTML_PROXY is set (§5).
Flags: --stdout (bytes to stdout, no file — for a caller that wants a string and
no temp-file lifecycle) · --max-bytes N (default 64 MiB — a finite default, unlike
the conversion flags', because an unbounded arbitrary-binary download is a DoS
foot-gun; the body is buffered, so this is also the memory bound) · --timeout S
(per operation, default 60, in (0, 300]) · --deadline S (total wall clock,
default 300, in (0, 3600]) · --retries N (0..10) ·
(repeatable — needed for content negotiation, e.g. ) ·
· .
Environment (optional):HTML_READER_URL / HTML_READER_PROVIDERS (remote reader base(s)), HTML_READER_TOKEN (generic reader auth), JINA_API_KEY (jina quota), HTML_SEARCH_URL / HTML_SEARCH_PROVIDERS (search provider base(s)), HTML_CHROME_STORAGE_STATE / HTML_CHROME_COOKIES_FILE / HTML_CHROME_USER_DATA_DIR (Chrome auth — server-deployable secrets), HTML_SSRF_ALLOW_NETS (SSRF carve-out CIDR list — no code default; unset/empty → none; .env.example ships 198.18.0.0/15 for RFC-2544/.eth.limo mappings; 0.0.0.0/0 disables IPv4 protection), HTML_PROXY (egress proxy — the ONLY way to proxy: trust_env=False, so HTTP_PROXY/HTTPS_PROXY and the macOS System Configuration proxy are ignored. Setting it re-opens the DNS-rebinding window, see §5, and prints a one-time notice). All optional; the CLI auto-loads <skill>/.env at startup (an in-process import caller does not — call _load_skill_env() yourself). See .env.example.
INPUT: a URL, or a local ////.
5. Safety Boundaries
Allowed scope: only the input + the named OUTPUT_DIR (and its _attachments/).
Never writes elsewhere. Exception — OP3 get: it writes the single caller-named
OUTPUT_PATH, creating parent directories, and overwrites an existing file (atomically,
via a sibling .part + rename, so a truncated artifact is impossible). A symlink or a
directory at OUTPUT_PATH is refused (exit 2) — get never writes through a link.
⚠️ The bytes it writes are unsanitized and fully remote-controlled — every other write
path in this skill runs sanitize_untrusted_html and derives its own slug. Do not point
get at a path that will subsequently be rendered (html get URL page.html produces a live
script/file:-bearing document on disk). The artifact is written 0600 — it is created
via mkstemp and os.replaced into place, so it is never world-readable, not even in
flight, and a crash cannot leave a truncated file under the caller's name.
Image reads are confined: a malicious <img src="../../etc/passwd"> /
file:///… / absolute path is refused — local image reads are confined to the
input's base dir (CWE-22/73 guard).
SSRF protection (lite path): every fetch hop (initial + redirects) is refused if
it resolves to a loopback / private / link-local / cloud-metadata (169.254.169.254)
address; body is streamed with a --max-bytes abort; --max-images bounds remote
fetches; non-http(s) top-level INPUT is treated as a local path, never fetched.
Remote-reader tier sends the target URL to an external service (r.jina.ai or a
configured reader fetches it server-side). In --engine auto the remote tier is an
automatic last-resort escalation for public targets (so a Cloudflare/anti-bot page
recovers without manual intervention) — meaning a public URL may leave the machine on
escalation. Guards: a private/internal/loopback/metadata target is NEVER forwarded to a
reader (a public-IP gate runs before any remote request); --no-remote disables the
remote tier entirely (no external egress); CR/LF/control chars in the target/query are
refused (request-splitting guard). Do not point , or against
sensitive URLs, at internal hosts you don't want proxied; use for fully
local conversion. The local hop to the reader passes the SSRF gate.
./scripts/.venv/bin/python -m unittest discover -s scripts/html2md/tests and
-s scripts/tests — full unit + E2E suite (file/archive/url mocked + real
tmp/ fixtures when present).
bash scripts/tests/test_e2e.sh — runs the suite + the diff -q replication gate.
CI signal: python3 .claude/skills/skill-creator/scripts/validate_skill.py skills/html — exits 0.
Whole-page Markdown on stdout; failures as a single-line JSON envelope.
8. Architecture & Replication (for maintainers)
html (formerly html2md) is the repo's first two-master skill (CLAUDE.md §2). It carries
byte-identical replicas — do not edit them here, diff -q gated:
The pdf render.py/chrome_engine.py/package __init__.py (weasyprint/playwright
carriers) are never replicated; web_clean/__init__.py is an html-owned thin
facade. See scripts/.AGENTS.md.
9. License
Proprietary, All Rights Reserved — see LICENSE / NOTICE. This skill embeds
byte-identical copies of proprietary docx/pdf code; it is a derived work and is
not Apache-2.0.
10. Resources
references/html-to-markdown.md — decision tree (URL/archive/file; reader vs whole;
lite vs chrome) + honest scope.
examples/basic-usage.md — copy-paste examples.
html login URL --save-state state.json
--chrome-storage-state state.json
server/Hermes-deployable
--chrome-cookies-file cookies.txt
--chrome-user-data-dir DIR
--chrome-scroll [--chrome-scroll-passes N]
SSRF-gated
auth_required
opt-in
local
references/html-to-markdown.md
Site-specific clean-source endpoints (proactive, auto/lite): Wikipedia/wiki/<Title> → the Parsoid REST page/html endpoint (the canonical page is
chrome-only and strips to empty); arXiv/abs/ or /pdf/<id> → the full-text
/html/<id> rendering (PDF-only papers return an actionable "use the pdf skill" hint);
HackerNoon/<slug> → /lite/<slug>.
Empty-extraction guard: a substantial source page that converts to a near-empty
body is a typed EmptyExtraction (exit 11) — never a silent exit 0 with an empty
note.
Obsidian emit: YAML frontmatter; --download-images → _attachments/
(sha1-dedup, relative links) — covers remote http(s)and content-sized inline data:
images (base64/percent-encoded decoded to files; tiny icon blobs dropped); dual-output
(<slug>.md + <slug>.reader.md); --reader-only emits a SINGLE
<slug>.md = the reader extraction (whole-page fallback if empty) — for note pipelines.
Math → Obsidian-native: Pandoc/MathJax \(…\) / \[…\] (and class="math" spans)
→ $…$ / $$…$$ with raw, un-escaped TeX, so formulas render in Obsidian/KaTeX. Escaped
plain-text brackets ([word], citations) and code spans are left untouched.
Agent step: --stdout (Markdown to stdout — inline data: blobs stripped to keep the
stream clean) + --json-errors envelope.
--header 'KEY: VALUE'
Accept: application/pdf,*/*
--browser-ua
--json-errors
★ Wall-clock: size your subprocess(timeout=…) against --deadline, not
--timeout.--timeout is per operation and bounds nothing in total: a redirect
chain multiplies it by max_redirects + 1 inside every retry pass, and a slow-drip
body resets the read timeout on each chunk while --max-bytes caps only SIZE. Measured:
a 5-hop chain stalling below the timeout ran 2.4x the per-op budget, and a body
dripping one byte per 1.6 s returned OK after 12.85 s against a 2 s timeout —
time-unbounded in principle. At the shipped defaults the redirect case alone would be
60 x 6 x 4 = 1440 s. --deadline is enforced inside the ladder at every hop and every
chunk, so it is a real bound; exceeding it is exit 10 with details.kind == "deadline".
Exit map: 0 ok · 2 usage — including a non-http(s) URL, so a caller's typo
is distinguishable from a security refusal · 10 FetchFailed (SSRF refusal, redirect
cap, over-cap body, deadline) · 1 internal · 141 broken pipe (--stdout only: the
consumer closed the pipe and holds a PREFIX — not reported as success, so truncation
stays detectable). Under exit 10 read details.kind to tell the classes apart:
refused (a security block — SSRF / scheme / control chars) · deadline · otherwise a
transport failure; details.max_bytes marks the over-cap case. Asserting only on the
exit code cannot distinguish a security refusal from an unreachable host.
--browser-ua and --header 'User-Agent: …' are mutually exclusive (the header
would silently win and make the flag a no-op). The artifact is written 0600.
On failure nothing is written, and a pre-existing OUTPUT_PATH is left untouched
— not removed (get never unlinks a file it did not create). On success the write is
atomic (.part + rename), so a truncated artifact the caller cannot detect is
impossible.
Capability probe (for programmatic consumers, so an older install fails CLOSED):
python3 scripts/html --help | grep -q 'html get URL'. ⚠️ A bare grep -q getalways succeeds — it matches --target-selector — and html get --help exits 0 on
a build without the verb too (argparse reads get as INPUT and prints the top-level
help). Neither is a valid probe.
python3 scripts/html2md.py INPUT [OUTPUT_DIR] — combined (fetch → md → delete the
intermediate HTML): the classic web-clip — you get just <slug>.md (+ .reader.md) +
_attachments/, no leftover HTML. This is the bare/back-compat one-shot.
Pipelines:
download → pdf:html fetch URL out/ && python3 ../pdf/scripts/html2pdf.py out/<slug>.html out.pdf --untrusted
(always pass --untrusted — it refuses file:// at the renderer, defense-in-depth over
the fetch-time sanitizer). Requires html fetch ran with images (the default) so the
PDF has them (pdf is offline — it never fetches remote <img>).
OUTPUT_DIR: directory to write <slug>.md (+ <slug>.reader.md by default) and
_attachments/ into. Omit → defaults to ./tmp/html_out/ (created on demand,
in the working directory). --stdout opts into stdout mode: YAML frontmatter +
whole-page Markdown (the reader variant and image files are skipped — not the
reader-extracted text).
Defaults: --engine auto, dual-output ON (--no-reader to suppress),
--download-images ON (--no-download-images keeps remote URLs), attachments dir
_attachments, --archive-frame main.
Outputs: <slug>.md + <slug>.reader.md + _attachments/<sha1>.<ext>; or
Markdown on stdout. <slug> is derived from the input filename / URL path
(deterministic); the human title lives in frontmatter.
Failure semantics / exit codes: 0 ok · 1 BadInput/ConvertFailed/internal ·
2 usage (incl. --search+URL, --engine remote with no provider, --max-results≤0) ·
3 EngineNotInstalled (Chrome explicitly requested, Playwright absent — in auto/
remote-first this is a silent fall-through, not exit 3) · 6 SelfOverwriteRefused ·
10 FetchFailed (unreachable / blocked / over --max-bytes; details.kind ∈ bot_blocked/
auth_required/not_found/rate_limited/server_error/unreachable/pdf/binary/arxiv_no_html/
refused/offsite_redirect/all_engines_failed) · 11 EmptyExtraction (substantial source →
near-empty body). auth_required from the chrome path = a stale/expired session (re-mint).
On a total-ladder failure, details.tried lists each tier attempted + its failure kind
(URL-free). --json-errors emits {v:1, error, code, type?, details?} on stderr.
Idempotency: same input → same output filenames + deduped attachments. URL
fetches reflect live content (not idempotent across server changes).
--engine jina|remote
auto
--no-remote
Authenticated Chrome (TASK 024): auth replays a human-minted session (no password/2FA
automation). The Chrome tier is now SSRF-gated — _assert_public_http before navigation,
context-level route guard aborting non-public sub-resources/fetch/beacon, and an
off-target public-redirect refusal (final origin must equal the target's eTLD+1) so a
session is never carried to another site. Session files (storage_state/cookies.txt) are
bearer credentials: passed by path only (never argv), required mode 0600 (group+world
rejected), symlinks refused, values never logged/redacted. The target + session stay local.
Egress is DIRECT by default (trust_env=False) so the connection pin is authoritative.
The ambient environment — HTTP_PROXY/HTTPS_PROXY/ALL_PROXY, .netrc, SSL_CERT_* — is
ignored; proxying is an explicit opt-in via HTML_PROXY, and setting it prints a
one-time stderr notice. This is not tidiness: until it changed, the pin was decorative.
Measured — pinning example.com to a blackholed 192.0.2.1 returned HTTP 200 under an
ambient proxy (pin ignored) and ConnectTimeout direct (pin honoured); the machine had no
proxy env vars, yet urllib.request.getproxies() still returned one from macOS System
Configuration. Check yours with
python3 -c "import urllib.request;print(urllib.request.getproxies())".
Honest-scope residuals: DNS-rebinding (resolve-then-connect TOCTOU) is closed on the lite
path (the connection is pinned to the validated IP) — but re-opens whenever HTML_PROXY is
set, because the proxy then resolves the target itself. The per-hop _assert_public_http
pre-check still runs in that mode, so it is a TOCTOU window, not an open door. ⚠️ Note also that
under a fake-IP resolver (Clash/V2Ray-style, which is what the shipped
HTML_SSRF_ALLOW_NETS=198.18.0.0/15 default exists for) the pin binds the synthetic address;
the synthetic→real mapping lives in the proxy tool and is outside this skill's control.
Rebinding also remains on the Chrome tier (Playwright manages its own sockets, and it keeps
using the SYSTEM proxy regardless of HTML_PROXY); storage_state localStorage is origin-restored (readable
by same-origin scripts the page loads); the login-wall heuristic is best-effort/per-site; _registrable is last-2-labels
(multi-level suffixes like co.uk over-match); a reader follows its own server-side redirects.
Run untrusted conversions in an egress-restricted sandbox. See references/html-to-markdown.md
and docs/KNOWN_ISSUES.md (HTML2MD-10).
No global installs: deps live in scripts/.venv + scripts/node_modules.
