web-markdown-fetch

name

web-markdown-fetch

description

Retrieve web page content in clean markdown/text for research and summarization. Use when Codex needs to read a URL, bypass noisy HTML, or handle Cloudflare-prone pages. Always use this fallback chain in order: markdown.new, then defuddle.md, then r.jina.ai, then Scrapling. Persist markdown/image assets under the current Codex session directory, and when image understanding is needed, read downloaded local images first.

Web Markdown Fetch

Core Rule

When reading any target URL, apply these methods in strict order:

1. https://markdown.new/<full-url>
2. https://defuddle.md/<full-url>
3. https://r.jina.ai/<full-url>
4. Scrapling fallback (scrapling extract ...)

Do not jump to Scrapling before trying the three mirror prefixes. For successful mirrors, keep evaluating in-order and select the best complete candidate (do not stop at the first partial success).

Treat asset handling as mandatory:

1. Persist fetched markdown and discovered images under the current Codex session directory.
2. Keep image assets enabled by default (--no-save-assets is allowed only if the user explicitly requests disabling image persistence).
3. For image analysis requests, read local downloaded images from that session asset directory before any external vision fallback.

Workflow

1. Normalize the input URL. If scheme is missing, prepend https://.
2. Try markdown.new first.
3. If output is empty, blocked, timeout, or wrong content, try defuddle.md.
4. If still failing, try r.jina.ai.
5. For usable mirror outputs, score candidates and pick the most complete one.
6. If all three fail, run Scrapling CLI fallback.
7. Save fetched markdown and discovered markdown-image assets to a temp directory named by Codex session id (auto-detect from CODEX_THREAD_ID and fallback env keys).
8. Explicitly report the resolved session asset directory path and the image save count (images_saved/images_discovered).
9. If the user asks for visual details (diagram/chart/screenshot content), read image files from that saved asset directory first using local image-reading capability.
10. Do not call external vision services first for image interpretation; only use them as fallback when local image reading is unavailable or explicitly requested by the user.
11. After each successful fetch, report a completeness score (0-100) and whether it is full (is_full=true/false).
12. Return extracted content and explicitly state which method succeeded.
13. If Scrapling reports SSL certificate verification errors, retry Scrapling with --no-verify.

Quick Commands

Test mirrors directly:

curl -L "https://markdown.new/https://example.com"
curl -L "https://defuddle.md/https://example.com"
curl -L "https://r.jina.ai/https://example.com"

Run bundled helper:

python3 scripts/fetch_markdown.py "https://example.com" --output content.md

Disable asset persistence only when explicitly needed:

python3 scripts/fetch_markdown.py "https://example.com" --output content.md --no-save-assets

Failure Policy

If all methods fail:

1. Report each attempted method with the failure reason.
2. Suggest next action: authenticated cookies, proxy, or site API.
3. Do not fabricate content.

If image analysis is requested but local image files are unavailable or unreadable:

1. Report whether assets were saved and list the missing/unreadable files.
2. Re-fetch with asset saving enabled if needed.
3. Only then propose external vision fallback.

SSL Note

Some environments fail in Scrapling with curl: (60) SSL certificate problem.

• Manual Scrapling fallback should include --no-verify in that case.
• scripts/fetch_markdown.py already tries this automatically during Scrapling fallback.

Session Asset Directory

• Default root: /tmp/codex-web-markdown-fetch
• Session id: auto-detected from CODEX_THREAD_ID (then CODEX_SESSION_ID, OPENAI_SESSION_ID, SESSION_ID)
• Layout: <assets-root>/<session-id>/<url-sha1-12>/
• Files: content.md, image_urls.txt, manifest.json, downloaded images (image_01.*, ...)
• manifest.json includes completeness scoring: score_percent, is_full, grade, missing_components
• Override root: --assets-root /path/to/root
• Override session id (rare): --session-id <id>

Default expectation for this skill:

1. Use the resolved Codex session-id directory as the canonical place for fetched images.
2. Reuse those local files for image interpretation in the same session.

Resources

• Use scripts/fetch_markdown.py for deterministic retries and structured fallback.
• Read references/workflow.md for mirror URL rules and troubleshooting.