profile-website-bot-detection

name	profile-website-bot-detection
description	Profile a website for bot detection vendors using stealth vs non-stealth Kernel browsers. Use when analyzing bot detection on a website, comparing stealth effectiveness, identifying anti-bot vendors and products, or detecting challenge types.

Profile Website for Bot Detection Vendors

Analyzes a target website to identify bot detection vendors, their specific products, and challenge types. Supports comparative analysis between stealth and non-stealth browser modes.

Prerequisites

• Kernel CLI installed and authenticated
• Bash, Node.js 22+, and jq installed
• KERNEL_API_KEY exported for the analysis script; never print it or ask the user to paste it into chat

Comparative Workflow (Recommended)

Compare bot detection behavior between stealth and non-stealth browsers while controlling the network path. Run Steps 1-7 in the same Bash process so the cleanup trap remains armed. Do not run Step 2 as a standalone non-interactive shell: its EXIT trap will delete the sessions when that shell returns.

Step 1: Prepare the Analyzer

Install dependencies before creating billable browser sessions:

cd scripts
npm install  # first run only
export TARGET_URL='https://example.com'
test -n "${KERNEL_API_KEY:-}" || { echo "KERNEL_API_KEY is not set" >&2; exit 1; }

Step 2: Create Both Browser Types and Arm Cleanup

Use JSON output to capture exact session IDs. The EXIT trap cleans up the first session if the second creation or a later analysis fails.

set -euo pipefail
STEALTH_ID=
NORMAL_ID=

cleanup() {
  local ids=()
  [[ -n "${STEALTH_ID:-}" ]] && ids+=("$STEALTH_ID")
  [[ -n "${NORMAL_ID:-}" ]] && ids+=("$NORMAL_ID")
  if ((${#ids[@]})); then
    kernel browsers delete "${ids[@]}" || true
  fi
}
trap cleanup EXIT
trap 'exit 130' INT
trap 'exit 143' TERM

STEALTH_ID=$(kernel browsers create --stealth --viewport 1920x1080@25 --timeout 300 --output json | jq -er '.session_id')
NORMAL_ID=$(kernel browsers create --viewport 1920x1080@25 --timeout 300 --output json | jq -er '.session_id')

# Isolate browser stealth behavior: route the stealth session directly, like the normal session.
kernel browsers update "$STEALTH_ID" --disable-default-proxy >/dev/null

A stealth session otherwise uses Kernel's default stealth proxy, while a normal session connects directly. Without the update, the result measures stealth plus proxy versus normal plus direct, not stealth alone. Apply --disable-default-proxy only to the stealth session; the API rejects it for non-stealth sessions.

Choose one network design before navigating:

• Browser-mode comparison (recommended): Keep the update above so both sessions connect directly.
• Platform-default comparison: Omit the update and report the stealth result as stealth + default proxy; do not attribute differences solely to stealth.
• Explicit-proxy comparison: Optionally validate a configured proxy with kernel proxies check "$PROXY_ID" --url https://example.com --output json, add --proxy-id "$PROXY_ID" to both create commands, and omit the update. This controls proxy configuration, though provider rotation can still produce different exit IPs.

Step 3: Run Analysis on Both Browsers

KERNEL_BROWSER_ID="$STEALTH_ID" TARGET_URL="$TARGET_URL" BROWSER_MODE=stealth npm run analyze
KERNEL_BROWSER_ID="$NORMAL_ID" TARGET_URL="$TARGET_URL" BROWSER_MODE=normal npm run analyze

Each analyzer run disconnects its local CDP client after writing the report; it leaves the Kernel session active for comparison and cleanup.

Step 4: Compare Results

Compare the newest report from each mode rather than concatenating multiple JSON documents:

# Derive the same hostname folder used by the analyzer (e.g., chase.com -> chase-com).
HOST=$(node -e 'const host = new URL(process.env.TARGET_URL).hostname; console.log(host.replace(/^www\./, "").replace(/\./g, "-"))')
latest_report() {
  local reports=("$1"/report-*.json)
  [[ -e "${reports[0]}" ]] || return 1
  printf '%s\n' "${reports[@]}" | sort | tail -n 1
}
STEALTH_REPORT=$(latest_report "output/$HOST/stealth")
NORMAL_REPORT=$(latest_report "output/$HOST/normal")

echo "=== STEALTH VERDICT ===" && jq '.summary.verdict' "$STEALTH_REPORT"
echo "=== NORMAL VERDICT ===" && jq '.summary.verdict' "$NORMAL_REPORT"

echo "=== STEALTH BLOCKED ===" && jq '.summary | {isBlocked, blockedPages, blockedVendors}' "$STEALTH_REPORT"
echo "=== NORMAL BLOCKED ===" && jq '.summary | {isBlocked, blockedPages, blockedVendors}' "$NORMAL_REPORT"

echo "=== STEALTH VENDORS ===" && jq '.summary.vendorNames' "$STEALTH_REPORT"
echo "=== NORMAL VENDORS ===" && jq  

Step 5: Interpret Comparison

Scenario	Stealth	Normal	Meaning
No vendors detected	0	0	No signals observed; do not assume the site has no bot detection
Same vendors, no blocks	N	N	Bot detection present, both pass
Normal blocked, stealth passes	0 blocks	Blocked	With a controlled network, stealth is effective
Both blocked	Blocked	Blocked	Both tested configurations are blocked
Different challenge types	Lighter	Harder	With a controlled network, stealth likely reduces suspicion

Step 6: Provide Summary

After running the comparative analysis, state the network design explicitly. Claim stealth effectiveness only when both sessions used the same network class.

Summary Report Template:

## Bot Detection Comparative Analysis: [TARGET_URL]

### Verdict
- **Network Design**: [both direct / same explicit proxy config / platform defaults]
- **Stealth Browser**: [verdict from summary.verdict]
- **Normal Browser**: [verdict from summary.verdict]
- **Stealth Effectiveness**: [Effective/Ineffective/Inconclusive; use Inconclusive for mixed network paths]

### Block Status
| Browser | Blocked | Block Type | Evidence |
|---------|---------|------------|----------|
| Stealth | [Yes/No] | [blockType or N/A] | [first evidence item] |
| Normal  | [Yes/No] | [blockType or N/A] | [first evidence item] |

### Detected Vendors
| Vendor | Stealth | Normal | Products |
|--------|---------|--------|----------|
| [vendor] | ✓/✗ | ✓/✗ | [product list] |

### Analysis
- [Explain what the results mean]
- [Note any differences between stealth and normal]
- [Recommend next steps if blocked]

### Key Findings
1. [Finding 1]
2. [Finding 2]
3. [Finding 3]

Use the JSON reports to populate this template:

• summary.verdict - The final verdict string
• summary.isBlocked - Whether the browser was blocked
• summary.blockedPages - Details about blocked pages
• summary.vendorNames - List of detected vendors
• vendorDetections - Detailed vendor/product information

Step 7: Cleanup

Delete both sessions immediately after collecting reports. browsers delete is non-interactive and accepts multiple IDs; it has no -y flag.

cleanup
trap - EXIT INT TERM

If the shell was interrupted or the variables were lost, use kernel browsers list to find the created sessions and delete each with kernel browsers delete <session-id>.

---

Interpreting Results

The analysis detects vendors and their specific products:

Vendor	Products Detected
Akamai	Bot Manager, Bot Manager Premier, mPulse RUM, Sensor Script, Edge DNS
Cloudflare	Bot Management, Turnstile, Challenge Platform, JS Challenge, Managed Challenge
DataDome	Interstitial Challenge, Slider Challenge, Device Check, Picasso Fingerprint
HUMAN/PerimeterX	Bot Defender, Sensor SDK, Press & Hold Challenge
Imperva/Incapsula	Advanced Bot Protection (utmvc), Advanced Bot Protection (reese84), WAF
Kasada	IPS (Initial Page Security), FP (Fingerprint Endpoint), Telemetry, POW Challenge
Google	reCAPTCHA v2, reCAPTCHA v3, reCAPTCHA Enterprise
hCaptcha	Widget, Enterprise
FingerprintJS	Fingerprint Pro, BotD
Arkose Labs	FunCaptcha

Detection methods:

• URL pattern matching for vendor scripts and endpoints
• Cookie analysis (e.g., _abck, __cf_bm, datadome, _px*)
• Header detection (e.g., cf-ray, x-kpsdk-*, x-d-token)
• Challenge detection from response status codes

Vendor-specific checks:

• DataDome: Hard IP block detection (dd.t === 'bv')
• Akamai: Cookie validity check (~0~ indicator)
• Kasada: Flow type detection (IPS vs FP)

Pages Analyzed

The script automatically analyzes:

1. Homepage - Initial page load and bot detection scripts
2. Login page - Automatically discovered via link detection or common paths (/login, /signin, etc.)

Login pages often have more aggressive bot detection due to credential stuffing prevention.

Output Files

Results are organized by target hostname in scripts/output/<hostname>/<mode>/:

• report-<timestamp>.json - Full JSON report with vendor detections
• screenshot-homepage-<timestamp>.png - Homepage screenshot
• screenshot-login-<timestamp>.png - Login page screenshot (if found)

Example structure for comparative test on chase.com:

output/chase-com/
├── stealth/
│   ├── report-*.json
│   ├── screenshot-homepage-*.png
│   └── screenshot-login-*.png
└── normal/
    ├── report-*.json
    ├── screenshot-homepage-*.png
    └── screenshot-login-*.png

The JSON report includes:

• summary: Quick access to verdict, block status, and vendor names
  • verdict: Human-readable result (e.g., "BLOCKED - homepage (Error Page)")
  • isBlocked: Boolean - true if any page was blocked
  • vendorNames: Array of detected vendor names
  • blockedPages: Details of blocked pages with evidence
• vendorDetections: Map of detected vendors with products, URLs, cookies, headers
• blockDetections: Detailed block analysis for each page
• vendorScriptsDetected: URLs of detected vendor scripts (not saved to disk)
• networkRequests/networkResponses: All requests with vendor matching
• cookies: All cookies with vendor attribution

Vendor-Specific Detection Notes

Akamai

• Cookies: _abck (core validation), bm_sz, bm_sv
• Cookie ~0~ in _abck value = valid session

Cloudflare

• Cookies: __cf_bm, cf_clearance
• Challenge: /cdn-cgi/challenge-platform/
• Turnstile: challenges.cloudflare.com/turnstile

DataDome

• Cookie: datadome
• dd.t === 'bv' = hard IP block (changing IP required, solving captcha won't help)

HUMAN/PerimeterX

• Cookies: _px2, _px3, _pxhd
• Press & Hold challenge requires behavioral simulation

Imperva/Incapsula

• utmvc: Script via /_Incapsula_Resource
• reese84: Cookie or x-d-token header

Kasada

• Headers: x-kpsdk-ct, x-kpsdk-cd
• Flow 1 (IPS): 429 on initial page load, must solve ips.js first
• Flow 2 (FP): Background /fp fingerprint requests