using-webctl

name	using-webctl
description	Browser automation via webctl CLI in Claude.ai containers with authenticated proxy support. Use when users mention webctl, browser automation, Playwright browsing, web scraping, or headless Chrome in container environments.
metadata	{"version":"0.0.2"}

Using webctl in Claude.ai Containers

Browser automation using webctl CLI with automatic proxy authentication handling for Claude.ai's egress-controlled environment.

When This Skill Applies

• User requests browser automation, web scraping, or page interaction
• webctl commands fail with ERR_TUNNEL_CONNECTION_FAILED
• Playwright/Chromium needs to work through authenticated HTTP proxy

Core Problem

Claude.ai containers route traffic through an authenticated egress proxy (HTTP_PROXY env var with JWT credentials). Chromium doesn't properly handle proxy authentication for HTTPS CONNECT tunnels, causing all HTTPS navigation to fail even though curl works.

Solution

A local forwarding proxy (port 18080) intercepts Chromium connections and injects Proxy-Authorization headers before forwarding to the real egress proxy.

Setup Procedure

1. Install webctl

pip install webctl --break-system-packages
webctl setup  # Downloads Chromium if needed

2. Deploy Auth Proxy Module

Copy scripts/auth_proxy.py to webctl's daemon directory:

cp /mnt/skills/user/using-webctl/scripts/auth_proxy.py \
   /usr/local/lib/python3.12/dist-packages/webctl/daemon/

3. Patch Session Manager

Apply this patch to /usr/local/lib/python3.12/dist-packages/webctl/daemon/session_manager.py.

Recent webctl versions split context creation into two branches (mobile emulation vs desktop). Both must receive the proxy config. Find the block (search for cfg = WebctlConfig.load() followed by the if mode == "unattended" and cfg.mobile_emulation: branch — around line 190):

cfg = WebctlConfig.load()
if mode == "unattended" and cfg.mobile_emulation:
    # Mobile emulation: cleaner pages, fewer ads, simpler layouts
    device = self._playwright.devices["Pixel 7"]
    context = await browser.new_context(
        storage_state=storage_state,
        **device,
    )
else:
    context = await browser.new_context(
        storage_state=storage_state, viewport={"width": 1280, "height": 720}
    )

Replace with:

cfg = WebctlConfig.load()
from .auth_proxy import get_local_proxy_url
proxy_url = get_local_proxy_url()
proxy_config = {"server": proxy_url} if proxy_url else None
if mode == "unattended" and cfg.mobile_emulation:
    # Mobile emulation: cleaner pages, fewer ads, simpler layouts
    device = self._playwright.devices["Pixel 7"]
    context = await browser.new_context(
        storage_state=storage_state,
        proxy=proxy_config,
        **device,
    )
else:
    context = await browser.new_context(
        storage_state=storage_state, viewport={"width": 1280, "height": 720},
        proxy=proxy_config
    )

If the file only has a single new_context call (older webctl), inject proxy=proxy_config into that one and place the three proxy_url lines just above it.

4. Verify

webctl start --mode unattended
webctl --quiet navigate "https://github.com"
webctl snapshot --interactive-only --limit 10
webctl stop

Quick Reference

Session Management

webctl start --mode unattended    # Headless browser
webctl stop                       # Full shutdown (use --keep-daemon to leave daemon running)
webctl status                     # Current state + console error counts

Navigation

webctl navigate "https://..."
webctl back / webctl forward / webctl reload

Observation

webctl snapshot --interactive-only --limit 30   # Buttons, links, inputs
webctl snapshot --within "role=main"            # Scope to container
webctl query "role=button name~=Submit"         # Debug queries
webctl screenshot --path shot.png

Interaction

webctl click 'role=button name~="Submit"'
webctl type 'role=textbox name~="Email"' "[e-mail]"
webctl type 'role=textbox name~="Search"' "query" --submit
webctl select 'role=combobox name~="Country"' --label "Germany"

Query Syntax

• role=button — By ARIA role
• name~="partial" — Contains (preferred, more robust)
• name="exact" — Exact match
• nth=0 — Select first when multiple matches

Wait Conditions

webctl wait network-idle
webctl wait 'exists:role=button name~="Continue"'
webctl wait 'url-contains:"/dashboard"'

Troubleshooting

ERR_TUNNEL_CONNECTION_FAILED

Auth proxy not loaded. Verify:

1. auth_proxy.py exists in webctl daemon directory
2. Session manager is patched
3. Restart daemon: webctl stop && webctl start --mode unattended

Multiple matches error

Add specificity or use nth=0:

webctl click 'role=link name="Sign in" nth=0'

Verify proxy is running

netstat -tlnp | grep 18080

Limitations

• Patch is session-local (container resets clear it)
• Only allowed domains in network config are accessible
• No Firefox support (download blocked by egress policy)

Domain Restrictions

Check <network_configuration> in system prompt for allowed domains. Common allowed: *.github.com, *.bsky.app, allowed API endpoints.

Output Filtering

Reduce context consumption:

webctl snapshot --interactive-only --limit 30    # Cap elements
webctl snapshot | grep -i "submit"               # Unix filtering
webctl --quiet navigate "..."                    # Suppress events