| name | shadertoy-api |
| description | Fetching shaders from the Shadertoy.com REST API — Cloudflare TLS-fingerprint workaround (curl_cffi), API key, monthly budget, caches, and what is permanently blocked (/media/). Use when downloading shaders, debugging 403 errors from shadertoy.com, adding fetch code, or reasoning about API budget for the mass-test campaign. |
Shadertoy API access
The one thing you must know first
shadertoy.com sits behind Cloudflare bot protection (since ~June 2026). Two
different mechanisms, two different outcomes:
- REST API (
https://www.shadertoy.com/api/v1/...) blocks by TLS
fingerprint (JA3), not by cookies or a JS challenge. Plain requests/curl
gets HTTP 403 "Bad request" no matter what headers you send.
Working fix (already implemented): curl_cffi with
impersonate="chrome". See houdini/scripts/python/hshadertoy/api/shadertoy.py
(top of file) — it falls back to plain requests if curl_cffi is missing.
- Gotcha: do NOT set a custom
User-Agent when impersonating. Chrome TLS
fingerprint + non-Chrome UA = 403 again.
- Residual 403s still happen occasionally even with impersonation (1 in
~1000 campaign fetches:
XsXfR4). Distinguish by message: the client's
explicit "Blocked by Cloudflare bot protection..." text means curl_cffi
is missing/inactive; a generic "Shadertoy API request failed (HTTP 403)"
means impersonation WAS active and Cloudflare rejected once anyway —
that's a normal retryable ERROR in the campaign ledger (just re-fetch
later), NOT a reason to reinstall curl_cffi or escalate to Plan B.
curl_cffi==0.15.0 is pinned in requirements.txt. For Houdini's own
Python: "C:/Program Files/Side Effects Software/Houdini 21.0.440/python311/python.exe" -m pip install --user curl_cffi.
- Media assets (
/media/... textures, cubemaps, and the homepage) are behind
the full JavaScript managed challenge. No impersonation target works.
Do not try to download media headlessly. The project doesn't need to:
the HDA builder maps Shadertoy asset IDs to bundled local assets via
houdini/scripts/python/hshadertoy/builder/hda/assets.json.
download_media_file / download_shader_icon in the API client are
broken-but-unused.
- Plan B if Cloudflare ever escalates onto the API path: Houdini 21's
PySide6 ships
QtWebEngineWidgets — fetch inside a QWebEngineView and read
the JSON out of the page. Do NOT try to extract its cookies for requests
(the TLS fingerprint still won't match).
API key and limits
- App key:
NdHjRR (hardcoded at tests/campaign/campaign.py:75 and used by the
API client). Keys are per-app, obtained at https://www.shadertoy.com/howto#q2.
- Budget: 1500 API calls/month. There is NO quota-query endpoint. The
campaign tracks its own tally in
tests/campaign/api_budget.json. The
authoritative "you're out" signal is the API responding
{"Error":"Too many requests"} — campaign.py fetch detects this, stops
immediately, and leaves the current id unfetched (safe to re-run next month).
- Only shaders published as Public + API are reachable. The API returns
"Shader not found" for BOTH genuinely missing ids and ids not set Public+API —
you cannot distinguish them.
- Quick connectivity test shader:
lsGSDG.
Where fetched shaders live (check cache before spending API calls)
tests/campaign/cache/<id>.json — campaign cache, the inner shader dict
({"ver":…, "info":…, "renderpass":[…]}).tests/shaders/api/json/ and tests/shaders/api/json_sp/ — older Phase-2
caches; campaign.py fetch checks these first (cache hits cost 0 API calls).- The full shader index
shader_all.json (41,854 ids, newest→oldest) is what
campaign sampling is built from — never re-fetch it casually; it's a huge call.
Always prefer python tests/campaign/campaign.py fetch --ids <id> over ad-hoc
scripts — it is cache-first, budget-tracked, and rate-limit-safe.
JSON shape gotcha
The raw API response is {"Shader": {inner}}. The campaign cache stores only
{inner}. Houdini-side tools (e.g. builder_test_headless.py) expect the
outer {"Shader": {...}} shape — wrap cache files before feeding them to
Houdini tooling (see the houdini-testing skill).
Related
- Mass-test fetch stage:
.claude/skills/mass-test-campaign/SKILL.md
- Rendering originals headlessly (wgpu-shadertoy, same Cloudflare rules apply to
its API module):
.claude/skills/render-compare/SKILL.md
