website-to-cli

name

website-to-cli

description

Build a local CLI that reproduces an authenticated website workflow by inspecting browser traffic and replaying private or semi-private web APIs. Use when Codex needs to turn a website feature into commands, capture cookies or bearer tokens through a real browser/manual login flow, design API replay commands, implement polling/download/export flows, or document a repeatable browser-assisted CLI integration for another site.

Website To CLI

Core Approach

Use the website as the source of truth. Keep login and bot checks in a real browser when needed, then move stable API calls into a CLI.

When the website depends on an obfuscated browser runtime, bytecode VM, request signer, anti-abuse token, or browser-only client algorithm that must become a pure local implementation, use the reverse-browser-runtime skill for the trace-driven reverse-engineering workflow, then return here for CLI productization.

Prefer this pattern:

1. Launch an isolated browser profile.
2. Let the user log in manually.
3. Capture cookies, local/session storage, and live request headers through CDP/Playwright.
4. Reconstruct the minimal API request bodies from captured fetch/HAR examples.
5. Store auth material outside the repo with restrictive permissions.
6. Implement one CLI command per user-visible action.
7. Add end-to-end tests against the real service only when the user accepts side effects.
8. Harden the CLI into a normal local tool: config, diagnostics, retries, default output paths, shell completion, and optional single-file binary builds.

For serious integrations, start with git before changing code. If the workspace is not already versioned and the user wants implementation, run git init, commit the existing code, then keep each milestone in its own commit. Never commit auth stores, captures, cookies, signed URLs, private outputs, or generated secrets.

Read references/checklists.md when planning a new integration or reviewing one before final handoff.

Workflow

1. Map The User Workflow

Write the target website workflow as user actions before coding:

• login
• create or select workspace/project
• upload or create resource
• start async operation
• poll status
• fetch or download result
• optional composition/export step

Name CLI commands after these actions, not after internal endpoint names. For example, prefer generate-video, wait-video, download-media, add-video-to-scene, export-scene.

2. Capture Auth Safely

Do not ask the user to paste secrets if a browser can supply them.

Use a dedicated browser profile and a local auth store such as:

~/.tool-name/profiles/<profile>/auth.json
~/.tool-name/browser-profiles/<profile>/

Capture:

• cookies for first-party endpoints
• live Authorization headers for API hosts
• CSRF/session/build hints from HTML, storage, or request headers
• token refresh endpoints if visible
• anti-abuse tokens generated in the browser context

Store full captures privately and print only redacted summaries.

3. Separate Auth Classes

Classify each host by authentication mechanism:

• first-party web API: usually cookies plus CSRF/referer/origin
• cross-site API: often bearer token plus browser metadata headers
• content CDN: often signed URL, not reusable auth
• anti-abuse service: often short-lived token that must be generated inside a browser

Implement refresh before replay when possible. If a cookie endpoint returns a fresh access token, update the saved cross-site API bearer and retry once after 401.

4. Replay Minimal Requests

Start with the smallest request that matches the captured body and works.

Keep browser-only headers out unless proven required. Usually keep:

• accept
• content-type
• origin
• referer
• cookie or authorization
• service-specific required headers

Avoid copying volatile headers blindly: sec-ch-ua, priorities, browser validation, and client data are often unnecessary.

5. Model Commands Around API Shapes

Implement commands in this order:

1. read-only account/session command
2. create/list commands
3. upload or input commands
4. async start command
5. status check command
6. wait command
7. download/export command
8. composed convenience command

For async workflows, always provide both:

• check-*: single status request
• wait-*: polling wrapper with --interval-ms and --timeout-ms

For multi-step workflows, keep low-level commands and add one high-level command that chains them.

6. Handle Outputs Explicitly

Support the common result forms:

• JSON result: print redacted JSON.
• Redirect result: request with redirect: "manual" and read Location.
• Signed URL: download binary with normal fetch and write to --output.
• Base64 payload in JSON: decode to bytes and write to --output.
• Operation job: print operation name and provide check-*/wait-*.

Never print huge binary/base64 payloads. Redact fields such as access_token, authorization, cookie, encodedVideo, imageBytes, recaptchaContext, sessionId, and token.

7. Verify With Real Calls

Test in increasing-risk order:

1. syntax/type checks
2. help/usage output
3. read-only session or credits call
4. token refresh
5. harmless create in a test project
6. upload using a small local fixture
7. async generation/export only after noting side effects or credits
8. status and download

Relay important command outputs in the final answer. The user usually cannot see raw command output.

8. Productize The CLI

After the first working replay, turn the script into a robust CLI before adding more feature surface:

1. Split the code by responsibility:
   • client: base URL, default params, cookies/headers, JSON fetch, retries, redaction.
   • auth: browser launch/capture, profile paths, auth summary.
   • signer: browser signer, local/headless signer, pure runtime signer when possible.
   • feature modules: one module per product workflow such as image/video/audio/export.
   • media or jobs: status, wait, queue, download, local job tracking.
   • config: user-level settings outside the repo.
2. Keep the executable entrypoint thin: parse flags, load config/auth, call modules, print structured JSON.
3. Preserve compatibility commands while introducing clearer grouped commands. For example, keep generate-video but add video create/status/wait/download/run.
4. Default to the most local stable implementation. Use browser signing only as a fallback when a local signer is unavailable or fails.
5. Add a doctor command that performs only read-only checks:
   • config readable
   • auth present and redacted
   • session/account endpoint works
   • quota/credits endpoint works when available
   • signer dry-run can produce required anti-abuse fields without submitting a generation request
6. Add config list|get|set for settings such as:
   • output_dir
   • retry_count
   • retry_delay_ms
   • default JSON/compact output behavior
7. Make downloads ergonomic:
   • allow explicit --output
   • support --output-dir
   • when safe, derive deterministic filenames from type, job/history id, index, and media extension
   • do not overwrite existing files unless an explicit overwrite flag exists
8. Add retry/backoff in the shared request layer for transient network errors and common retryable statuses such as 429, 500, 502, 503, and 504.
9. Add machine-facing affordances:
   • commands list for command registry output
   • completion zsh|bash|fish for shell completion
   • --compact for one-line JSON
   • --quiet for script-friendly silence
10. Add optional packaging:

• keep npm link support through bin
• add npm run build:binary for a current-platform executable when useful
• document that single-file binaries embed a runtime but still read auth/config from the user profile
• test the produced binary with --help, command registry, and doctor

Do not let productization hide the low-level primitives. A good website CLI has both composable commands (create, status, wait, download) and one convenience command (run) that chains the common path.

CLI Design Rules

Use stable, explicit flags:

--profile
--project-id
--media-id
--operation-name
--input-videos id1,id2
--output ./file.mp4
--interval-ms 5000
--timeout-ms 600000

For captured enum values, expose friendly aliases and preserve override flags:

--aspect portrait|landscape
--model <raw-model-key>
--user-tier <raw-tier>

Return a structured JSON result with:

• ok
• status
• endpoint
• parsed or redacted response
• request summary
• output file path and byte count when writing files

Expose stable grouped commands for user-facing workflows:

tool auth login|capture|status
tool config list|get|set
tool doctor
tool <resource> create|status|queue|wait|download|run
tool jobs list|sync|add|status
tool commands list
tool completion zsh|bash|fish

Use consistent result semantics:

• create: starts work and returns ids plus a redacted request summary.
• status or check: performs one read-only status fetch.
• queue: returns service queue/polling hints when available.
• wait: polls until success/failure/timeout.
• download: fetches one result item and reports output path and bytes.
• run: create, wait, and optionally download in one command.
• doctor: read-only health check; never consumes credits.

Safety Rules

• Do not commit auth stores, captures, cookies, tokens, signed URLs, or generated secrets.
• Do not overwrite existing user files without clear intent.
• Do not run credit-consuming or state-changing real tests without telling the user what will happen.
• Do run read-only verification when it materially improves confidence.
• Keep a backup before changing an existing CLI if the file looks hand-built or valuable.
• If a command returns secrets or huge payloads, add redaction before testing again.
• Treat local captures as private even when redacted. Commit only documentation and code unless the user explicitly asks to preserve sanitized captures.
• When testing generated outputs, use ignored output directories and report only paths, byte counts, dimensions, durations, or hashes.
• Make destructive or costly behavior opt-in. Prefer read-only doctor, status, and credits checks during verification.

Implementation Pattern

Use one shared request layer:

requestJson({ auth, url, method, body, contentType })
requestWithAutoRefresh({ paths, auth, url, method, body, contentType })

Use helpers for repeated protocol details:

savedHeadersForHost(auth, host)
refreshAccessToken({ paths, auth })
checkStatus(...)
waitStatus(...)
resolveSignedUrl(...)
downloadBinary(...)
saveBase64Payload(...)

Keep command parsing simple unless the project already has a CLI framework.

Recommended local files:

bin/<tool>.mjs
lib/client.mjs
lib/auth.mjs
lib/config.mjs
lib/signer.mjs
lib/<resource>.mjs
lib/media-or-jobs.mjs
scripts/build-binary.mjs

Recommended user data paths:

~/.tool-name/config.json
~/.tool-name/profiles/<profile>/auth.json
~/.tool-name/browser-profiles/<profile>/
~/.tool-name/jobs.jsonl
~/.tool-name/sdk/

Recommended package scripts:

{
  "bin": {
    "tool": "./bin/tool.mjs"
  },
  "scripts": {
    "check": "node --check bin/tool.mjs && node --check lib/client.mjs",
    "build:binary": "node scripts/build-binary.mjs"
  }
}

For Node CLIs, npm link is enough for local use. If the user wants not to think about Node, build a current-platform executable with Node SEA or an equivalent packager. Verify the binary separately from the source script because packaging can fail due to dynamic imports, embedded files, code signing, or runtime path assumptions.

Handoff

Update README or command docs with:

• login/capture steps
• auth storage path
• config path and useful defaults
• every command and required flags
• parameter meanings
• confirmed endpoints
• signing strategy and fallbacks
• real test results
• binary build/install instructions when implemented
• known side effects and remaining unknowns

Include exact commands that worked, but never include live tokens, cookies, signed URLs, or private file contents.

Before final handoff, run and report the highest-signal checks that apply:

npm run check
tool --help
tool commands list --compact
tool doctor --profile default
tool <resource> status --id <known-id>
tool <resource> download --id <known-id> --index 0 --output-dir outputs/check
npm run build:binary
./dist/tool-<platform>-<arch> --help
./dist/tool-<platform>-<arch> doctor --profile default
git status --short

If a check would consume credits or mutate remote state, say that explicitly before running it.