crypto-recon-fetch

name	crypto-recon-fetch
description	Download all JS/WASM/CSS assets from a target URL using cloudscraper. Saves to output/[domain]/. Called by crypto-recon-orchestrator.

Crypto Recon — Asset Fetcher

Purpose

Fetch the target URL, discover all JavaScript and WebAssembly assets, download them all, and organize into output/[domain]/. This is always Phase 1 before any analysis begins.

Execution Steps

Step 1 — Initial Request

Run:

python tools/fetch.py <url>

This script:

1. Uses cloudscraper (bypasses Cloudflare, handles 302/301/meta-refresh)
2. Fetches the URL
3. Detects content type (HTML vs JSON vs binary)

Step 2 — Asset Discovery

If response is HTML, extract ALL of these:

Script tags:

<script src="/assets/js/main.js">
<script type="module" src="...">
<script crossorigin src="...">

Link preloads:

<link rel="modulepreload" href="...">
<link rel="preload" as="script" href="...">

Inline script analysis — search raw HTML for:

/assets/js/*.js
*.chunk.js
*.bundle.js
/static/js/*.js

Dynamic import hints — regex in HTML:

import("..."), require("...")

Source maps (for minified JS):

//# sourceMappingURL=*.map

Step 3 — JS Inside JS (Second Pass)

After downloading initial JS files, scan each one for:

• import( dynamic imports
• require( CommonJS
• fetch('/assets/js/, fetch("/static/
• Additional chunk URLs embedded as strings

If new JS URLs found → download those too.

Step 4 — Download

Run:

python tools/download.py <domain> <js_url_list>

Storage structure:

output/
└── [domain]/
    ├── index-5ef15b56.js
    ├── common.modules-6d4292d1.js
    ├── page-login-abc123.js
    ├── chunk-vendor-xxx.js
    └── [...]

Rules:

• Keep original filenames (strip query params)
• If filename collision → append _2, _3
• Save raw text (no modification, no minification)
• Download in parallel (max 8 threads)
• Timeout per file: 30s

Step 5 — WASM Handling

If .wasm files found:

• Download them to output/[domain]/wasm/
• Do NOT attempt to analyze content
• Note in summary: "WASM found — requires binary RE tools (Ghidra, wabt)"

Step 6 — Summary Report

After all downloads, print and return:

Asset Discovery Summary
=======================
Base URL:      http://example.com
Final URL:     https://example.com (after redirects)
JS files:      18
WASM files:    1
Failed:        0
Total size:    1.2 MB
Saved to:      output/example.com/

JS files:
  index-5ef15b56.js           (84.3 KB)
  common.modules-6d4292d1.js  (212.1 KB)
  page-login-abc123.js        (31.2 KB)
  ...

Special Cases

SPA with no static JS links:

• The HTML may have only one entry index.js that lazy-loads everything
• Check inside that file for dynamic chunk references
• Look for webpack chunk manifest: __webpack_require__.e, webpackChunks

API endpoint input (no HTML):

• If user gave an API URL directly (returns JSON)
• Try stripping path to get base URL, fetch that
• Report: "API endpoint detected. Fetched base URL: [url]"

Cloudflare challenge:

• cloudscraper handles JS challenges automatically
• If still blocked: report "Cloudflare 403 — manual cookie extraction needed"
• Ask user for browser cookies

Content Security Policy:

• Note any Content-Security-Policy header in the summary
• It may reveal additional asset domains

Output to Orchestrator

Return:

{
  "domain": "[domain]",
  "folder": "output/[domain]",
  "js_files": ["index-5ef15b56.js", "common.modules-6d4292d1.js"],
  "wasm_files": ["runtime.wasm"],
  "total_size_kb": 1247,
  "status": "ok"
}