reverse-api-engineer

name	reverse-api-engineer
description	Reverse engineer undocumented or hidden web APIs by observing browser traffic and building a live client. Use when: the user wants to extract API endpoints from a website (e.g. moba.example.gg, internal dashboards, third-party apps), recover a hidden list endpoint by hooking fetch/XHR in a real browser, build a Python/Node client that calls the real API for a webapp, or document a target site. Triggers: reverse api, api endpoint, แกะ api, ดึง api, undocumented api, hidden endpoint, scrape api, internal api, traffic capture, session replay, เลียนแบบ client.
argument-hint	<target-url> [purpose]

Reverse API Engineer

End-to-end playbook for turning a website into a working live API client. Verified against moba.example.gg (Next.js + Cloudflare + Laravel-style backend at back.moba.example.gg) — 5 endpoints, 132 heroes, 100% live data, no scraping required for the working endpoints.

When to Use

• User wants API endpoints extracted from a specific website
• "ดึงข้อมูลจากเว็บ / แกะ API / ทำ client เลียนแบบ"
• Building a webapp that needs fresh data from a 3rd-party site
• Documenting an internal/undocumented API for team use
• Recovering the structure of a "hidden" aggregated list endpoint

When NOT to Use

• Public, well-documented APIs (use the official SDK / OpenAPI)
• TOS-violating scraping of personal data — see legal-guardrails.md
• Mobile APK reverse engineering (use a different skill)

Outcome

A documented set of live REST endpoints + a runnable client + the data shape, ready to drop into a webapp. Not a one-shot HTML scraper.

---

Core Philosophy (load these rules first)

1. Observe before you guess. Always hook the browser BEFORE speculating about endpoints. The moba.example.gg case: 5 endpoints were real, ~25 guessed paths were 404.
2. Live API > scraping. Real endpoints return fresh data and survive HTML changes. Scrape only as last resort.
3. Hook both transports. Modern SPAs mix fetch and XMLHttpRequest. Hook both — counters/builds/forum endpoints each used different transports in moba.example.gg.
4. Browser-like headers are mandatory. Cloudflare in front of most public APIs accepts the stdlib User-Agent, but matching Chrome User-Agent + Origin + Referer prevents 403 if WAF rules tighten later.
5. Retry + backoff is not optional. Burst traffic to Cloudflare-fronted APIs causes RemoteDisconnected ~2-5% of the time. Exponential backoff + small delay solves it.

---

Workflow — 5 Phases

Phase 1: Recon — find the data sources

Goal: identify the frontend stack, backend host, and what data is on screen.

1. Open the target site in a real browser session (Playwright is fine)
2. From the HTML, extract:
   • Backend host: look for X-Powered-By, server headers, RSC payload URLs, asset CDN domains
   • Framework: Next.js? React+Vite? Look for _next/, __NEXT_DATA__, RSC streaming
   • CDN/WAF: Cloudflare (cf-ray, __cf_bm), Akamai, DataDome?
3. List the page's tabs/filters/sections. Each is a potential endpoint to discover.

Example (moba.example.gg): Next.js + Cloudflare, backend at back.moba.example.gg/api/v1, page tabs = Overview / Counters / Synergy / Builds → 5 endpoints to find.

Not sure what framework this is? See frameworks.md for a 5-second detection cheatsheet (Next.js App/Pages, Nuxt 3, Vue 3 SPA, React SPA, Angular, Remix).

Phase 2: Observe — hook real browser traffic

Goal: capture every fetch + XMLHttpRequest to the backend while clicking through the UI.

See observe-browser.md for the complete hook template (Playwright + vanilla JS).

Key steps:

1. Open target page in browser
2. Install fetch + XHR hooks before any user interaction. If you see live updates (chat, prices, scores), also install the WebSocket hook (see same file)
3. Click every tab / change every filter / submit every form
4. Record { url, method, body } for each call
5. For Next.js RSC sites, also parse the RSC payload for SSR data (see same file)

Result (moba.example.gg): clicking Counters, Synergy, Builds on /heroes/104 revealed 3 hidden endpoints in seconds.

Phase 3: Test the endpoints

Goal: confirm which captured calls are publicly callable without browser session, cookies, or unusual headers.

For each URL captured in Phase 2:

1. Try curl / Invoke-WebRequest / requests.get() with no headers
2. If 401/403, add User-Agent: Mozilla/5.0 ... Chrome/..., Origin: https://<site>, Referer: https://<site>/
3. If still failing, the endpoint likely needs a session cookie → check Set-Cookie on the page load
4. If endpoint returns 404 with no auth, the endpoint is server-only (Next.js SSR/RSC) → see "Hidden endpoints" below

Phase 4: Build the client

Goal: a runnable client with retry, backoff, and proper headers. See client-impl.md.

Template (Python, dependency-free):

• Set browser-like default headers
• Wrap every call in retry-with-exponential-backoff
• For bulk fetches: add delay parameter (0.3s is safe default for Cloudflare-fronted APIs)
• Write JSON to stdout via sys.stdout.buffer to avoid Windows console codec errors

Phase 5: Document the data shape

Always document:

• Which endpoint returns what
• The JSON schema (record the actual field types — string vs number matters)
• The auth/cookie/session requirements
• Known hidden endpoints (with workaround)

---

Decision Tree

Is the target a single GraphQL endpoint (POST with `query` body)?
├── YES → See [graphql.md](./references/graphql.md) — introspection, body-content hooks, anti-abuse
└── NO → continue with the REST tree below

Is the endpoint public (no auth, returns 200 from curl)?
├── YES → Add to client, document
└── NO (401/403) → 
    ├── 401 Unauthorized? → Check Set-Cookie; you may need a session
    ├── 403 Forbidden? → Check User-Agent / Origin / Referer / TLS fingerprint
    │   └── If 403 persists → endpoint is WAF-protected, may need curl_cffi or browser session
    └── 404 Not Found → 
        ├── Are you guessing? → STOP, go observe real traffic in browser
        └── Was the URL captured from browser? → Endpoint is server-only (Next.js RSC) 
            ├── Try same URL with `RSC: 1` header
            └── If still 404, the data is in the RSC payload — parse that instead

Hidden endpoints (server-side only)

When the page shows data but no public endpoint returns it, the data is fetched server-side and embedded in the page payload. For Next.js App Router:

# Get the raw RSC payload
curl -s "https://moba.example.gg/statistics?rank_filter=Mythic" \
     -H "RSC: 1" \
     -H "Next-Router-State-Tree: %5B%22%22%2C...%5D" | head -c 200

The response is a stream of self.__next_f.push() calls; parse the line that contains "heroes":[{...}] for the aggregated data. See observe-browser.md §RSC parsing for the full pattern.

Anti-bot / WAF countermeasures

See anti-bot.md — only applies if the simple Python urllib/requests approach gets blocked. Most public APIs (including moba.example.gg) work fine with browser-like headers.

---

Quick reference: tools to use

Need	Tool	When
Hook browser fetch/XHR	mcp_playwright_browser_evaluate	Phase 2 (always)
Parse RSC payload	Python re.findall on the raw HTML	When endpoint is server-only
Try endpoint quickly	PowerShell Invoke-WebRequest	Phase 3
Production client	Python urllib with retry wrapper (see client-impl.md)	Phase 4
Response validation	jsonschema + genson (Python) or ajv + json-schema-to-typescript (TypeScript) — see client-impl.md §Response validation	Phase 4+ — catches upstream shape changes
WAF bypass (last resort)	curl_cffi (Python), impers / wreq-js (Node)	When 403 with proper headers
GraphQL endpoints	Introspection query + body-content hooks (see graphql.md)	When target has a single /graphql endpoint
WebSocket capture	Hook window.WebSocket constructor (see observe-browser.md §WebSocket capture)	When the page updates live (chat, prices, scores) without page refresh
Framework detection	`curl	grep` for tells (see frameworks.md)

---

Quality criteria — done means

• All visible data on the target page has at least one live endpoint that returns it
• Each endpoint documented with: URL, method, params, response shape, auth requirements
• Client has retry + delay, works 100× in a row without manual intervention
• Hidden endpoints are flagged with workarounds (RSC parse or per-resource fan-out)
• Run the client at least once end-to-end; the data matches what's on the page

---

Quick start environment

For most of the skill you only need Python 3.10+ and a browser. Install the optional extras as you need them:

# Phase 2 (Observe) — one of:
pip install playwright && playwright install chromium      # Python Playwright
# OR use the Chrome MCP / Playwright MCP tool already available in the agent

# Phase 4 (Build) — schema validation
pip install jsonschema genson

# Optional — replay / regenerate from HAR
pip install haralyzer

# Optional — WAF bypass when stdlib gets 403'd
pip install curl_cffi
# Node alternative:
npm install wreq-js

A pre-baked Dockerfile with Playwright + curl_cffi pre-installed is a planned future addition. For now, the install list above is the supported path.

---

Optional integrations (from kalil0321, additive — never required)

The patterns below are imported from kalil0321/reverse-api-engineer. Use them when the situation calls for it. None of them are required for the skill to work — they extend the workflow, they don't change it.

Save capture as HAR (for replay)

Default: window.__apiCalls array. Optional: save as HAR 1.2 file for replay/regenerate/share. See observe-browser.md §HAR replay and the capture_save.py helper.

When worth it: when the client might need to be regenerated later (e.g. site added endpoints, you want to compare today's capture with last week's).

Output structure convention

Recommended layout for the agent's output (helps user find files):

scripts/<name>/
├── api_client.py      # importable + CLI
├── README.md          # endpoint catalog
├── example_usage.py   # 3-5 minimal examples
└── captures/          # optional HAR files

See client-impl.md §Output structure.

Collector mode (one-time dump to JSONL/CSV)

When the user wants data, not a reusable client. See client-impl.md §Collector mode.

When worth it: "scrape all X from this site" — single-use, no need to maintain a client.

Multi-language output (JS/TS)

Default is Python. For Node/Next.js webapps, translate to TypeScript. See client-impl.md §Multi-language.

When worth it: the webapp itself is TypeScript (Next.js, Remix, Hono). Avoid Python↔TS impedance mismatch.

Chrome MCP / Chrome DevTools Protocol (Cloudflare bypass)

If the simple Playwright MCP approach fails to load the page (Cloudflare JS challenge, Akamai sensor data), drop down to Chrome MCP for the observe phase only. The endpoint discovery still works the same way; only the browser driver changes.

When worth it: the target site has aggressive bot detection that blocks Playwright's default fingerprint. Keep using stdlib + browser headers for the actual API calls — Chrome MCP is just for observing.