web-inspection-forensics

name

web-inspection-forensics

description

Use common CLI, Kali, and shell tools to inspect web Targets, captured assets, logs, HAR/PCAP files, packages, and generated artifacts, then turn observations into Evidence, Security Signals, and ranked Hypotheses. Use when the user asks for web bug-hunting inspection, digital records, URL maps, crawling, endpoint discovery, JS bundle/package analysis, light forensics, or command-line investigation workflows.

Web Inspection Forensics

Use this skill to turn a web Target or captured artifact set into reproducible Evidence, concrete Security Signals, and bounded follow-up Hypotheses. Prefer passive collection first, keep scope explicit, and avoid exploit payloads until a later approved Investigation Run. See GLOSSARY.md for shared term definitions.

Handoffs: Use bug-hunt-planner to structure findings into a staged plan. Use ctf-tool-selection when a user provides a broad CTF/security tool list or asks which established tool fits the job. Use reverse-engineering-analysis for JS bundles, WASM, or binary artifacts found during inspection. Use exploitation-sandbox to verify a Hypothesis with a PoC.

Quick Start

1. Define scope: allowed hosts, auth state, rate limits, start URLs, capture IDs, artifact paths, and stop conditions.
2. Create a working folder: mkdir -p evidence/{raw,normalized,reports,stats}.
3. Record provenance: date -Is, uname -a, tool versions, command line, URL/source, hashes, file sizes, and environment notes.
4. Collect passive Digital Records: headers, robots/sitemap/security.txt, TLS/DNS basics, HTML, linked assets, HARs, logs, and prior Web Captures.
5. Build a URL Map and candidate surface inventory: pages, forms, POST endpoints, APIs, JS bundles, source maps, package/version hints, technologies, protocols, parameters, object IDs, and third-party APIs.
6. Sweep the hunt spine: attack-surface mapping, vulnerability-class triage, business-logic/API review, secrets and sensitive-data review, then proof/report readiness.
7. Summarize outputs into Evidence, then queue ranked Hypotheses for any active testing or deeper review.

Command Logging

Wrap important commands so outputs are reproducible. Prefer the repo helper when available:

When an HTTP collection or replay step would normally use curl, prefer curl_cffi if it is installed in the approved Workspace Container. Preserve the same headers, cookies, timeout, redirect, rate-limit, provenance, and Evidence capture requirements. Fall back to curl when curl_cffi is unavailable, when the user specifically needs a shell-native transcript, or when a wrapper only supports a curl command.

./sandbox/skills/reverse-engineering-analysis/scripts/capture-command-stats.sh \
  "evidence/stats/curl-home.json" -- curl -sSIL https://example.test/

If no wrapper is available, capture at least:

{ date -Is; pwd; command -v curl; curl --version | head -1; } > evidence/stats/env.txt
curl -sSIL "$URL" > evidence/raw/headers.txt 2> evidence/raw/headers.err
printf '%s  %s\n' "$(sha256sum evidence/raw/headers.txt)" headers.txt >> evidence/stats/hashes.txt

Digital Records / Discovery / URL Map

• Headers and redirects: curl -sSIL -A bug-hunter-web-inspection/0.1 "$URL".
• HTML and assets: curl -sSL --compressed "$URL" -o evidence/raw/index.html; parse with rg, pup, xmllint, jq, or local scripts.
• Passive historical URLs: gau, waybackurls, vendor docs, existing Web Captures, and user-provided HARs.
• Same-origin crawl: katana, hakrawler, wget --spider, or app-local crawler code; keep depth, host allowlist, and rate limits explicit.
• Content discovery, only when approved and bounded: ffuf, feroxbuster, gobuster; use low rate, small wordlists, and stop on instability.
• DNS/TLS basics: dig, host, openssl s_client -connect host:443 -servername host, testssl.sh --fast, and nmap -sT -Pn -p 80,443 only within scope.
• Web vulnerability scanners: nuclei, OWASP ZAP active scan, Burp Scanner, W3AF, sqlmap, Commix, and XSSer require approval because they can send payloads, mutate state, or generate high request volume.

Normalize URL maps as TSV/JSONL with: method, url, status, content_type, bytes, source, auth_required, third_party, and notes.

Hunt Spine Inspection

Use the same repeatable sequence for broad web/API inspection, but keep each step tied to artifacts and approval boundaries:

1. Attack-surface mapping: merge scoped assets, DNS/certificate clues, live-looking hosts, route maps, passive historical URLs, crawl output, API specs, JS assets, source maps, packages, and observed technologies. Deduplicate before testing.
2. Vulnerability-class triage: group leads by input handling, access control/auth/session, information disclosure/debug, exposed files/directories, browser/API security posture, known-version risk, and SSRF-like egress clues. Scanner output and pattern matches are triage leads, not findings.
3. Business-logic/API review: map roles, accounts, object ownership, state transitions, GraphQL/RPC methods, rate limits, payment or workflow gates, and tenant boundaries. Prefer manual modeling and non-mutating observations before approval-gated validation.
4. Secrets and sensitive-data review: inspect JS bundles, source maps, config-looking files, logs, error messages, metadata endpoints, and uploaded artifacts for secrets, internal endpoints, private data, or credentials. Redact values in summaries; checking validity requires approval.
5. Proof and reporting: save screenshots, request/response summaries, HAR entries, command stats, and hashes as Evidence. Promote Signals only when reproduction, affected surface, impact, proof limit, and remediation are clear.

Advanced focus queue:

• Review health, metrics, debug, console, status, admin, source-map, backup, log, and config-looking paths when they appear in passive records or approved discovery output.
• Treat forwarding, original-URL, host, client-IP, cache, CORS, CSP, and content-negotiation headers as trust-boundary clues; changing them against live Targets is active probing.
• Extract duplicate parameters, array/object parameters, ID-like values, UUIDs, JWT/OAuth/SAML hints, cookies, and CSRF tokens as hypotheses for parser, authz, and session review.
• Preserve verbose errors, stack traces, framework banners, internal paths, SQL-looking messages, and access-denied details as Evidence before interpreting them.

Login, Captcha, And Guided Capture

• Do not bypass captcha, MFA, lockouts, payment flows, or anti-abuse controls.
• Prefer browser/HAR capture for authenticated flows; export cookies only to scoped files and redact them from reports.
• Reproduce benign requests with curl -b cookies.txt -c cookies.txt, httpie, or xh.
• Preserve CSRF and state handling as observations: token names, refresh behavior, cookie flags, SameSite, redirects, and auth boundaries.
• Mark flows needing human/browser guidance instead of forcing automation.

Web Tool Guidance

• Burp Suite / OWASP ZAP / mitmproxy: use for manual proxying, replay, authenticated capture, and passive scan summaries. Disable active scanning, spidering beyond scope, form submission, and attack plugins until the user approves the exact Target, auth state, rate limits, and stop condition.
• Postman / Insomnia / httpie / xh: use for repeatable API requests and collection cleanup. Export collections without secrets, or redact tokens before saving Evidence.
• sqlmap / Commix / XSSer: treat as active exploit tooling. Before use, capture a manual differential signal, choose the lowest practical risk/level, restrict URL/parameter scope, set delay/rate limits, and save full command output as an Artifact.
• nuclei / W3AF / scanner templates: prefer passive or low-impact templates first. Pin template versions and avoid broad template packs unless the approval explicitly includes them.
• RequestBin / collaborator services: use only when outbound callbacks are in scope. Prefer a local listener or user-controlled callback host for sensitive tests.
• Deprecated or niche browser add-ons: do not install old exploitation extensions by default. Prefer Burp/ZAP/DevTools unless a specific workflow requires the add-on and provenance is verified.

Candidate Target Identification

• POST/input handling: extract forms and HAR entries with jq '.log.entries[] | {method:.request.method,url:.request.url,status:.response.status}' capture.har.
• API references: rg -n "fetch\\(|axios\\.|XMLHttpRequest|/api/|/rest/|graphql|POST|PUT|PATCH|DELETE" evidence/raw.
• JS bundles/source maps: file, du -h, sha256sum, rg "sourceMappingURL|webpack|vite|rollup|parcel", npx prettier, js-beautify, source-map-explorer, or wakaru when useful.
• Package/version hints: inspect package.json, lockfiles, source maps, bundle banners, npm view, pnpm list, npm audit --json, retire, and osv-scanner.
• Technologies/protocols: whatweb, httpx -tech-detect, response headers, CSP, CORS, WebSocket URLs, GraphQL introspection hints, OAuth/OIDC/SAML/JWT strings, WASM imports, and third-party API hosts.
• Generated artifacts: review logs, screenshots, reports, traces, command outputs, and prior Evidence before expanding scope.

Rank candidates by reachability, sensitivity, controllability, change/drift, affected versions, auth boundary, and cleanup feasibility.

Broad Discovery Lanes

When the user asks for a broad target scan, vulnerability sweep, or first-pass bug hunt, optimize for breadth across Discovery Lanes before drilling into the first interesting issue. Start from passive records and URL/API inventory, then sample one or two representative routes per lane. Do not invent findings to fill a lane; unsupported lanes become open questions.

Use these Discovery Lanes as the default queue:

Discovery Lane	Passive Discovery Signals	Representative Next Step
Input handling	Search forms, query parameters, body parameters, reflected values, parse errors, template/script sinks, path or file parameters	Capture routes/parameters and propose the smallest non-mutating probe; active payloads require approval.
Access control and auth/session	Login/logout/reset/profile/account routes, ID-shaped objects, user-specific APIs, cookies, JWT/OAuth/SAML hints, anonymous-vs-auth boundary responses	Map auth boundaries and object ownership assumptions before any cross-account test.
Information disclosure and debug	Stack traces, verbose errors, framework/version banners, internal paths, debug endpoints, source maps, build metadata, noisy 4xx/5xx responses	Save the response as Evidence, extract disclosed components, and rank whether it enables another lane.
Exposed files and directories	Directory listings, backups, .bak/.old, config filenames, public uploads, source maps, logs, archive references, security/privacy files	Inventory filenames and sensitivity; request approval before high-volume content discovery.
Browser/API security posture	CSP/HSTS/X-Frame-Options/CORS headers, CSRF tokens, WebSocket/SSE endpoints, API versioning, rate-limit headers, cache headers, client-side route guards	Record policy/header gaps and API boundary clues; avoid state-changing requests without approval.

For a broad sweep report, aim for 3-5 distinct Evidence-backed Discovery Lanes when the target supports them within the budget. For each Signal include:

• Discovery lane: the lane name.
• Surface: route, parameter, header, object boundary, asset, or artifact.
• Observed behavior: concise fact from Evidence.
• Evidence: artifact path, capture ID, request/response hash, screenshot, HAR entry, or command stats.
• Confidence: confirmed / likely / speculative.
• Next step: passive follow-up or explicitly approval-gated validation.

If the agent has a fixed time/cost/tool budget, sample across lanes first, then spend remaining budget on the highest-impact lane. Stop once evidence is sufficient to report candidates and avoid repeated tool-schema repair or route-by-route probing that does not add new lanes.

Forensics And Artifact Triage

• Inventory files: find, stat, file, du -ah, wc -l, sha256sum, tree, tar -tf, unzip -l.
• Text extraction: strings, rg, jq, yq, xmllint, exiftool, xxd, hexdump -C.
• Media/stego triage from web artifacts: pngcheck, zsteg, stegsolve, stegseek/StegCracker, ImageMagick, Audacity, binwalk, and foremost; keep originals immutable and record dictionaries/time limits for brute-force steps.
• Logs: normalize timestamps, sort by request/session IDs, count status/method/path with awk, sort, uniq -c, and jq.
• PCAPs or local traffic captures: use tshark, tcpdump -r, zeek, and Wireshark only on owned or explicitly provided traffic.
• Redact secrets from strings, URLs, cookies, headers, HARs, screenshots, and summaries.

Output Shape

Produce a concise report with:

• Scope: Target, authorization, hosts, auth state, time window, and excluded actions.
• Digital Records: files, hashes, command stats, tool versions, and capture sources.
• URL Map: pages, assets, endpoints, methods, status codes, third-party APIs, and auth/captcha notes.
• Stats: counts and bytes by asset kind, JS bundle sizes, package/version hints, status-code distribution, and drift versus prior captures when available.
• Hunt Spine Progress: attack-surface mapping, vulnerability-class triage, business-logic/API review, secrets/sensitive-data review, and proof/report readiness.
• Discovery Lane Coverage: Evidence-backed lanes and unsupported lanes that remain open questions.
• Security Signals: concrete observations with evidence paths and confidence.
• Hypotheses: ranked next steps with rationale, first bounded command or browser action, expected Evidence, and stop condition.

Guardrails

• Stay authorized, scoped, and reproducible. Passive first; active probing requires explicit approval.
• Do not run credential harvesting, uncontrolled scanning, persistence, stealth, destructive checks, or high-rate fuzzing.
• Store large bodies as Artifacts and record durable summaries as Evidence.
• Prefer local reproduction, captures, diffs, and static analysis before network-impacting work.