authenticated-web-extraction

name	authenticated-web-extraction
compatibility	Requires Node.js with Playwright, an installed Microsoft Edge channel, and a writable persistent browser profile directory.
description	Extract data from sites that require login (LinkedIn, GitHub, Sessionize, Microsoft 365, X, Meetup) using a persistent Playwright + Microsoft Edge profile. Covers profile setup, interactive sign-in capture, session-cookie re-injection (Chromium drops session-only auth cookies on shutdown), cookie-based auth detection, OAuth callback flags, and a task harness pattern. USE FOR: scrape LinkedIn, scrape GitHub, scrape Sessionize, authenticated scraping, persistent browser profile, Playwright Edge profile, msedge channel, headless authenticated extraction, session cookie persistence, OAuth callback cookie, ASP.NET ApplicationCookie, li_at, user_session, Sessionize cookie, login status check, launchPersistentContext, Edge tracking prevention, FedCm SSO, profile lock orphan msedge, CV resume data extraction from web. DO NOT USE FOR: posting or mutating user accounts, CAPTCHA-heavy targets, scraping at scale (no rate limiting), Microsoft Graph API access (use microsoft-todo-tasks), Outlook COM (use outlook-* skills).

Authenticated Web Extraction (CareerAuthBrowser)

Persistent Playwright + Microsoft Edge profile for pulling data out of sites that require login. Tool lives at %LOCALAPPDATA%\CareerAuthBrowser\ on the user's workstation; this skill documents how to operate and extend it.

When to Use

• User asks to extract their LinkedIn profile, GitHub repos, Sessionize sessions, Meetup events, etc.
• Copy/paste from the browser would be slow, lossy, or error-prone.
• A site uses interactive SSO (Microsoft, Google, GitHub OAuth) that pure-API tooling can't handle without app registration.

Architecture

Path	Role
%LOCALAPPDATA%\CareerAuthBrowser\scripts\open.mjs	Visible Edge window for interactive logins. Auto-snapshots cookies for tracked domains every 5s and on close.
%LOCALAPPDATA%\CareerAuthBrowser\scripts\extract.mjs	Generic harness. Re-injects saved cookies, runs tasks/<name>.mjs, then closes. Supports --headless.
%LOCALAPPDATA%\CareerAuthBrowser\tasks\	One ESM file per extraction. Default-export async ({ context, page, args, outDir }) => {…}.
%LOCALAPPDATA%\CareerAuthBrowser\auth-state\	Per-domain cookie JSON. Survives close so session-only cookies keep working. Never commit.
%LOCALAPPDATA%\CareerAuthBrowser\profile\	Edge user-data dir (cookies, localStorage, IndexedDB). Never commit.
%LOCALAPPDATA%\CareerAuthBrowser\extracted\	Default output for tasks.

Canonical sources of package.json, scripts/*.mjs, and the generic tasks (check-logins, dump-cookies) are bundled under bootstrap/ next to this file. See Bootstrap from Scratch.

Bootstrap from Scratch

If %LOCALAPPDATA%\CareerAuthBrowser\ does not exist (new machine, wiped profile, etc.), recreate it from the bundled bootstrap/ folder:

$dst = "$env:LOCALAPPDATA\CareerAuthBrowser"
$src = '<path-to-this-skill>\bootstrap'
New-Item -ItemType Directory -Path $dst, "$dst\scripts", "$dst\tasks" -Force | Out-Null
Copy-Item "$src\package.json"            "$dst\package.json"            -Force
Copy-Item "$src\scripts\open.mjs"        "$dst\scripts\open.mjs"        -Force
Copy-Item "$src\scripts\extract.mjs"     "$dst\scripts\extract.mjs"     -Force
Copy-Item "$src\tasks\check-logins.mjs"  "$dst\tasks\check-logins.mjs"  -Force
Copy-Item "$src\tasks\dump-cookies.mjs"  "$dst\tasks\dump-cookies.mjs"  -Force
cd $dst; npm install

npm install pulls Playwright (msedge channel reuses the system Edge install — no Chromium download needed). The user-data-dir, auth-state, and extracted output folders are created lazily on first run.

Standard Workflow

1. Check auth. Run check-logins first — never assume the profile is still signed in.

   cd "$env:LOCALAPPDATA\CareerAuthBrowser"; node scripts/extract.mjs check-logins --headless
   Get-Content "$env:LOCALAPPDATA\CareerAuthBrowser\extracted\login-status.md"
   

2. If not signed in, open the visible window for the user to sign in:

   cd "$env:LOCALAPPDATA\CareerAuthBrowser"; node scripts/open.mjs '<login URL>'
   

   Tell the user: "Sign in until you see the dashboard, then close the window." open.mjs snapshots cookies every 5 seconds, so the user does not need to wait after the dashboard renders. Re-run check-logins to confirm before continuing.

3. Run the extraction task headlessly. Read its output from extracted/.

4. Copy/transform the output into the user's workspace (typically Input/).

Critical Gotchas

Cookie names per site

Auth detection must use cookies, not DOM selectors. LinkedIn's markup changes between A/B variants and breaks selector-based checks.

Site	Auth cookie	Domain
LinkedIn	li_at	https://www.linkedin.com
GitHub	user_session	https://github.com
Sessionize	.AspNet.ApplicationCookie	https://sessionize.com
X	auth_token	https://x.com
Meetup	MEETUP_MEMBER	https://www.meetup.com

Sessionize cookie name is .AspNet.ApplicationCookie, not .AspNet.Cookies or .AspNetCore.Cookies. Sessionize runs ASP.NET (Identity / OWIN), not ASP.NET Core. Verify with tasks/dump-cookies.mjs if a new site behaves unexpectedly.

Session cookies vanish on Chromium shutdown

Sessionize (and many ASP.NET Identity sites) sets its auth cookie with no expires attribute, making it session-scoped. Chromium discards session cookies when the persistent context closes — even though the user-data-dir is preserved. The fix is implemented in open.mjs and extract.mjs:

1. While the browser is open, periodically read context.cookies(domain) and write to auth-state/<host>.json.
2. Promote any cookie with expires === -1 (session) to now + 365 days.
3. On every subsequent extract.mjs run, call context.addCookies(...) from auth-state/*.json before navigating.

Do not remove this logic. Without it, Sessionize logs out on every restart even though LinkedIn and GitHub keep working.

Edge / Chromium flags required for OAuth callbacks

Edge's tracking prevention blocks the third-party cookie set by sessionize.com/signin-microsoft after a Microsoft OAuth round-trip. Always launch with:

ignoreDefaultArgs: ['--enable-automation'],
args: [
  '--disable-blink-features=AutomationControlled',
  '--disable-features=TrackingPrevention,ThirdPartyStoragePartitioning,PrivacySandboxAdsAPIs,FedCm',
  '--disable-site-isolation-trials',
  // visible only:
  '--start-maximized'
]

AutomationControlled removal also reduces extra captchas on LinkedIn.

Profile-lock orphan processes

If open.mjs hangs immediately after launch with no visible window, orphan msedge.exe processes are holding the user-data-dir lock. Detect:

Get-Process msedge -ErrorAction SilentlyContinue | Format-Table Id, StartTime, MainWindowTitle
Test-Path "$env:LOCALAPPDATA\CareerAuthBrowser\profile\SingletonLock"

Fix: Stop-Process -Name msedge -Force. open.mjs has built-in detection that reports this and exits cleanly.

Never mutate user profiles automatically

Default rule for any extraction agent: extract → propose changes → user pastes manually. Do not click "Save" on LinkedIn About edits, do not push commits to GitHub repos found via the profile, do not RSVP to Meetup events. Ask before any mutation.

Adding a New Task

// tasks/<name>.mjs
import fs from 'node:fs';
import path from 'node:path';

export default async function run({ context, page, args, outDir }) {
  await page.goto('https://example.com/protected', { waitUntil: 'domcontentloaded' });

  // Detect logged-out state up front — saves debugging time.
  if (page.url().includes('/login')) {
    throw new Error('Not signed in. Run: node scripts/open.mjs https://example.com/login');
  }

  const data = await page.evaluate(() => {
    // ... DOM scraping in page context ...
    return { /* ... */ };
  });

  const outFile = path.join(outDir, '<name>.json');
  fs.writeFileSync(outFile, JSON.stringify(data, null, 2), 'utf8');
  console.log(`Wrote ${outFile}`);
}

Run with: node scripts/extract.mjs <name> --headless.

Pass extra args after the task name; they arrive as args in the task signature.

Existing Tasks

• check-logins — auth status report → extracted/login-status.md. Run before every extraction.
• dump-cookies — JSON dump of all cookies grouped by domain. Use to discover unknown auth cookie names.
• linkedin-probe — confirms LinkedIn identity is signed in.
• sessionize-probe — confirms Sessionize speaker dashboard renders.

Failure Modes & Diagnosis

Symptom	Likely cause	Fix
check-logins shows site as "not signed in" right after sign-in	Cookie name in tasks/check-logins.mjs is wrong	Run dump-cookies, find the auth-like cookie, update TARGETS
Site logs out between runs but in-session works	Session-scoped cookie not captured	Confirm auth-state/<host>.json exists and contains the auth cookie with a future expires; verify the snapshot timer in open.mjs is running
OAuth completes but /app/... redirects to /login mid-session	Third-party cookie blocked	Confirm Edge launch flags include TrackingPrevention,ThirdPartyStoragePartitioning,FedCm
open.mjs exits or hangs with no window	Profile lock from orphan msedge.exe	Stop-Process -Name msedge -Force
Task throws "Target page, context or browser has been closed"	Task continued after context.close(); usually a missing await	Audit await chain in the task

Companion Prompt

VS Code prompt that wires this skill into a chat workflow: %APPDATA%\Code\User\prompts\auth-extract.prompt.md.