web-scraping

name

web-scraping

description

Use when scraping Indian government portals, property tax systems, land records, ArcGIS services, ASP.NET WebForms sites, or any large-scale web data collection. Covers requests, Playwright, aiohttp, CAPTCHA solving, proxy rotation, resumable pipelines, and anti-detection. Triggers on: scrape, crawl, download portal data, extract records, bulk download, government website, CAPTCHA, ViewState, ArcGIS REST.

Web Scraping: Large-Scale Government Portal Data Collection

Overview

Patterns and tools for scraping Indian government portals at scale (millions of records). Covers the full stack: HTTP clients, browser automation, ASP.NET WebForms state machines, ArcGIS REST APIs, CAPTCHA solving, anti-detection, proxy rotation, resumable progress tracking, and async concurrency. Derived from production scrapers across MCD property tax, Haryana land records (jamabandi.nic.in), GHMC/CDMA (Telangana), Mumbai MCGM, Delhi DDA/DORIS, 99acres, and JustDial.

When to Use

• Scraping ASP.NET WebForms portals (ViewState/EventValidation token chains)
• Downloading from ArcGIS REST FeatureServer/MapServer endpoints
• Solving image CAPTCHAs with OCR (Tesseract)
• Building resumable scrapers that survive crashes mid-run
• Scaling to millions of records with async or process-level parallelism
• Bypassing anti-bot protections on commercial sites (99acres, JustDial)
• Reverse-engineering mobile app APIs (APK credential extraction)
• Scraping cascading dropdown forms (district > tehsil > village > record)

How Web Scraping Works: Mental Model

A scraper is a program that pretends to be a web browser. When you visit a website, your browser sends HTTP requests (GET to load pages, POST to submit forms) and receives HTML, JSON, or files back. A scraper does the same thing programmatically.

The key challenge is that websites are designed for humans, not machines. They use sessions (cookies that track who you are across requests), anti-bot protections (checking if you're a real browser), rate limits (blocking you if you send too many requests), and CAPTCHAs (puzzles to prove you're human). Scraping at scale means solving all of these simultaneously while being robust to crashes, network failures, and changing website behavior.

The two fundamental approaches:

1. Direct HTTP (requests, aiohttp): You craft HTTP requests manually, sending exactly the headers and form data the server expects. Faster, lighter, works for APIs and simple HTML pages. Most government portals work this way.

2. Browser automation (playwright, selenium, undetected-chromedriver): You control a real browser (Chrome/Firefox) that executes JavaScript, renders pages, and handles all the browser-level complexity. Slower, heavier, but necessary when content is JavaScript-rendered or anti-bot detection is sophisticated.

Tool Selection

Scenario	Tool	Why
JSON/XML API, simple forms	requests.Session	Fast, lightweight, handles cookies automatically
ASP.NET WebForms postback	requests.Session	Token management is HTTP-level, no JS needed
ArcGIS REST API	requests or esridump	Paginated JSON responses, no JS rendering
JavaScript-rendered content	playwright (async or sync)	Full browser engine, executes JS
Anti-bot commercial sites	undetected-chromedriver or Playwright + stealth	Patches Chrome to hide automation fingerprints
High-concurrency (>10 req/s)	aiohttp + asyncio	Non-blocking I/O, one thread handles hundreds of connections
Bulk downloads (files/PDFs)	requests with streaming	stream=True avoids loading entire file into memory
ESRI spatial data	esridump library	Handles ESRI-specific pagination logic internally

Decision flow: Can you get the data with curl? Use requests. Does the page need JS to render? Use Playwright. Need 50+ concurrent connections? Use aiohttp. Anti-bot protection? Use undetected-chromedriver.

Why requests.Session over bare requests?

A Session object persists cookies and headers across requests, just like a browser does. When you visit a government portal, the server sets a session cookie on your first request. Subsequent requests must include that cookie, or the server treats you as a new visitor. A Session handles this automatically. It also lets you set default headers (like User-Agent) once rather than on every request, and reuses TCP connections for better performance.

When to use browser automation vs. direct HTTP

Browser automation launches a real Chrome instance, which uses ~200MB of RAM and is 10-100x slower than direct HTTP. Use it only when:

• The page content is rendered by JavaScript (the HTML source is empty, content appears after JS runs)
• The site uses sophisticated bot detection (Cloudflare, DataDome, PerimeterX)
• You need to interact with complex UI elements (clicking through modals, infinite scroll)

For government portals, direct HTTP works 90% of the time. Even forms with dropdown cascades and CAPTCHAs can be handled with requests, because the server-side logic accepts plain HTTP POST requests.

Core Patterns

1. Resumable Progress Tracking

What problem does this solve? Scraping millions of records takes hours or days. If your script crashes at record 500,000, you don't want to start over from record 1. Progress tracking lets you skip already-scraped items on restart.

Three proven approaches, from simplest to most robust:

JSON progress file (simplest, good for <100K items):

A JSON file on disk that records which items are done. On startup, load it and skip those items. The trade-off: loading a JSON list of 1M IDs into memory is slow and uses ~100MB, so this doesn't scale well.

PROGRESS_FILE = Path("progress.json")
SAVE_INTERVAL = 100  # save every 100 items, not every item (disk I/O is expensive)

def load_progress():
    if PROGRESS_FILE.exists():
        return json.loads(PROGRESS_FILE.read_text())
    return {"done": [], "failed": []}

def save_progress(progress):
    # Write to temp file first, then rename atomically (see "Atomic Writes" below)
    tmp = PROGRESS_FILE.with_suffix(".tmp")
    tmp.write_text(json.dumps(progress))
    tmp.replace(PROGRESS_FILE)  # atomic on POSIX

progress = load_progress()
for i, item in enumerate(items):
    if item["id"] in progress["done"]:
        continue
    # ... scrape ...
    progress["done"].append(item["id"])
    if i % SAVE_INTERVAL == 0:
        save_progress(progress)

SQLite progress DB (best for >100K items, concurrent access):

SQLite is a file-based database that handles millions of rows efficiently. PRAGMA journal_mode=WAL (Write-Ahead Logging) allows multiple readers while one writer is active, which matters if you're running a dashboard that reads progress while the scraper writes.

conn = sqlite3.connect("progress.db")
conn.execute("PRAGMA journal_mode=WAL")  # allows concurrent reads during writes
conn.execute("""CREATE TABLE IF NOT EXISTS progress (
    id TEXT PRIMARY KEY, status TEXT, updated_at TEXT)""")

def mark_done(item_id):
    conn.execute("INSERT OR REPLACE INTO progress VALUES (?, 'done', datetime('now'))",
                 (item_id,))
    conn.commit()

def is_done(item_id):
    # PRIMARY KEY lookup is O(log n), fast even for millions of rows
    return conn.execute("SELECT 1 FROM progress WHERE id=?", (item_id,)).fetchone()

File-existence check (for per-entity output files):

The simplest possible resume: if the output file already exists, skip that item. No separate progress tracking needed. Works well when each scraped entity produces one output file.

output_path = output_dir / f"{entity_id}.json"
if output_path.exists():
    continue  # already scraped

Combining approaches: Use file-existence for coarse-grained resume (skip entire districts or colonies) and a progress DB for fine-grained tracking within each district.

2. Atomic File Writes

What problem does this solve? If your script crashes while writing a JSON file, you get a half-written, corrupt file. On restart, the file exists (so you'd skip it) but contains garbage data. Atomic writes prevent this.

How it works: Write to a temporary file first, then rename it to the final path. On POSIX systems (Linux, macOS), os.replace() is an atomic operation: the file either has the old content or the new content, never a partial state. If you crash during write_text(), only the .tmp file is corrupt, and the final file is untouched.

def save_atomic(path, data):
    tmp = path.with_suffix(".tmp")
    tmp.write_text(json.dumps(data, ensure_ascii=False, indent=2))
    tmp.replace(path)  # atomic on POSIX: instant rename, no partial state possible

3. Retry with Exponential Backoff

What problem does this solve? Network requests fail. Servers return 500 errors, connections time out, TCP connections reset. Retrying immediately usually fails again (the server is still overloaded). Exponential backoff waits longer each time: 1s, 2s, 4s, 8s, 16s. This gives the server time to recover.

urllib3's built-in retry (handles it automatically inside requests):

urllib3.Retry sits between your code and the network. When a request fails with a retryable status code (429 = rate limited, 500 = server error, etc.), it automatically waits and retries. Your code sees only the final success or failure.

from urllib3.util.retry import Retry
from requests.adapters import HTTPAdapter

def make_session():
    s = requests.Session()
    retry = Retry(
        total=5,             # up to 5 retries
        backoff_factor=1.0,  # wait 1s, 2s, 4s, 8s, 16s between retries
        status_forcelist=[408, 429, 500, 502, 503, 504],  # retry on these HTTP status codes
    )
    # Mount the retry adapter for both HTTP and HTTPS
    s.mount("https://", HTTPAdapter(max_retries=retry))
    s.mount("http://", HTTPAdapter(max_retries=retry))
    s.headers.update({"User-Agent": UA_CHROME})
    return s

Manual retry loop (when you need custom logic, e.g., refreshing sessions on failure):

for attempt in range(max_retries):
    try:
        r = session.get(url, timeout=30)
        r.raise_for_status()
        return r
    except (Timeout, ConnectionError) as e:
        # Exponential backoff with jitter: 1-2s, 2-3s, 4-5s, etc.
        # The random component prevents all workers from retrying at the same instant
        wait = min(60, 2 ** attempt + random.uniform(0, 1))
        time.sleep(wait)
        if attempt == max_retries - 1:
            raise  # give up after final attempt

4. Session Refresh

What problem does this solve? Government portal web servers often have aggressive session timeouts (5-20 minutes of inactivity) or per-session request limits. After N requests, the server stops responding or returns errors. Creating a fresh Session object establishes a new TCP connection with new cookies, effectively starting a new "visit."

REFRESH_EVERY = 200  # create new session every 200 requests
request_count = 0

def maybe_refresh():
    global request_count, session
    request_count += 1
    if request_count % REFRESH_EVERY == 0:
        session = make_session()
        # Re-load the form page to get fresh cookies and CSRF/ViewState tokens
        init_page = session.get(FORM_URL)
        parse_tokens(init_page.text)

5. Rate Limiting

What problem does this solve? Sending requests too fast gets you blocked (IP ban, 429 responses) or overloads the server (causing 500 errors for everyone, including real users). Rate limiting is about being a good citizen while still scraping efficiently.

Why randomize? Fixed delays (e.g., exactly 2.0s between requests) create a detectable pattern. Real humans don't click at exact intervals. Randomized delays look more natural and are harder for anti-bot systems to fingerprint.

# Simple: random delay between requests
time.sleep(random.uniform(1.5, 3.0))

# Better: jittered delay
# "Jitter" means adding extra randomness to prevent synchronized bursts.
# If you have 10 workers all using random.uniform(1.5, 3.0), they might
# all land on similar values and send requests at the same time.
# Jitter adds/subtracts up to 30% of the base delay to spread them out.
base_delay = random.uniform(rate_min, rate_max)
jitter = base_delay * 0.3 * random.uniform(-1, 1)
time.sleep(base_delay + jitter)

# Async: token bucket rate limiter
# A "token bucket" is a classic rate-limiting algorithm. Imagine a bucket that
# holds N tokens and refills at a fixed rate. Each request consumes one token.
# If the bucket is empty, you wait until a token is available.
# This gives you a steady average rate while allowing short bursts.
class RateLimiter:
    def __init__(self, rate):  # rate = max requests per second
        self.rate = rate
        self.tokens = rate      # start with a full bucket
        self.last = time.monotonic()
        self.lock = asyncio.Lock()

    async def acquire(self):
        async with self.lock:
            now = time.monotonic()
            # Refill tokens based on elapsed time
            self.tokens = min(self.rate, self.tokens + (now - self.last) * self.rate)
            self.last = now
            if self.tokens < 1:
                # No tokens available; wait until one is generated
                await asyncio.sleep((1 - self.tokens) / self.rate)
                self.tokens = 0
            else:
                self.tokens -= 1

6. Circuit Breaker

What problem does this solve? Sometimes a server goes down completely, or a specific endpoint starts returning errors for every request. Without a circuit breaker, your scraper keeps hammering the dead endpoint, wasting time and possibly getting your IP blocked. A circuit breaker "opens the circuit" (stops sending requests) after too many consecutive failures, waits for a recovery period, then cautiously tries again.

The state machine: Think of it like an electrical circuit breaker in your house:

• Closed (normal): Requests flow through. Failures are counted.
• Open (tripped): Too many failures. All requests are rejected immediately. Wait for recovery timeout.
• Half-open (testing): Recovery timeout has passed. Allow one test request. If it succeeds, close the circuit. If it fails, re-open.

class CircuitBreaker:
    def __init__(self, failure_threshold=10, recovery_timeout=300):
        self.failures = 0
        self.threshold = failure_threshold    # open circuit after this many failures
        self.recovery_timeout = recovery_timeout  # seconds to wait before retrying
        self.last_failure = None
        self.state = "closed"  # closed -> open -> half_open -> closed

    def allow_request(self):
        if self.state == "closed":
            return True
        if self.state == "open":
            # Has enough time passed to try again?
            if time.time() - self.last_failure > self.recovery_timeout:
                self.state = "half_open"
                return True  # allow one test request
            return False  # still in cooldown
        return self.state == "half_open"  # allow one test request

    def record_success(self):
        self.failures = 0
        self.state = "closed"  # circuit is healthy again

    def record_failure(self):
        self.failures += 1
        self.last_failure = time.time()
        if self.failures >= self.threshold:
            self.state = "open"  # trip the circuit

ASP.NET WebForms

What is it? ASP.NET WebForms is a Microsoft web framework from the early 2000s, still used by many Indian government portals (jamabandi.nic.in, DDA, DORIS, MCD). It's designed to make web pages behave like desktop forms: you click a dropdown, the page "posts back" to the server, which processes your selection and returns an updated page.

Why is it hard to scrape? Unlike modern REST APIs where each request is independent, WebForms maintains server-side state. Every request must include a ViewState token (a large base64 blob representing the page's current state). If you send the wrong ViewState, the server rejects your request. Each interaction is a POST that sends the current state back and receives new state.

See references/asp-net-webforms.md for the full pattern including cascading dropdowns, postback navigation, and token extraction.

Quick pattern:

# 1. GET page -> server sends you the initial ViewState (page state) as a hidden form field
r = session.get(url)
soup = BeautifulSoup(r.text, "html.parser")
viewstate = soup.find("input", {"name": "__VIEWSTATE"})["value"]
validation = soup.find("input", {"name": "__EVENTVALIDATION"})["value"]

# 2. POST with tokens + your selection. The server processes your selection,
#    updates its state, and returns a new ViewState for the next interaction.
data = {
    "__VIEWSTATE": viewstate,
    "__EVENTVALIDATION": validation,
    "__EVENTTARGET": "ctl00$ContentPlaceHolder1$ddlDistrict",  # which control you "clicked"
    "ctl00$ContentPlaceHolder1$ddlDistrict": district_code,     # what you selected
}
r = session.post(url, data=data)
# Parse new tokens from response for the next POST

ArcGIS REST API

What is it? ArcGIS is Esri's GIS platform. Many government agencies (HSAC Haryana, DDA Delhi, MCGM Mumbai) host spatial data (building footprints, cadastral parcels, ward boundaries) on ArcGIS servers. These servers expose a REST API that returns data as JSON with a standard query interface.

Why is pagination needed? ArcGIS servers limit responses to 1000-2000 features per request. To download all 341,000 building footprints in Mumbai, you need ~170 paginated requests.

See references/arcgis-rest-api.md for complete patterns including geometry conversion and layer discovery.

Quick pattern:

# Offset-based pagination: "give me records 0-1999, then 2000-3999, ..."
# where=1=1 means "all records" (SQL WHERE clause)
# outFields=* means "return all columns"
# f=json means "respond in JSON format"
offset = 0
while True:
    params = {"where": "1=1", "outFields": "*", "f": "json",
              "resultOffset": offset, "resultRecordCount": 2000}
    data = session.get(f"{base_url}/query", params=params).json()
    features.extend(data["features"])
    if not data.get("exceededTransferLimit"):
        break  # server says "no more records"
    offset += len(data["features"])

Anti-Detection

What is it? Websites detect scrapers by checking for signs of automation: missing headers that real browsers send, the navigator.webdriver JavaScript property being true, requests arriving at inhuman speed, or traffic from known datacenter IP addresses.

See references/anti-detection.md for the full toolkit: user-agent rotation, Playwright stealth scripts, proxy configuration, human-like behavior simulation, and SSL certificate handling.

Minimum viable anti-detection (sufficient for most government portals):

# The User-Agent header tells the server what browser you're using.
# Python's requests library sends "python-requests/2.28" by default,
# which immediately identifies you as a script. Set a real browser UA.
UA_CHROME = ("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
             "AppleWebKit/537.36 (KHTML, like Gecko) "
             "Chrome/120.0.0.0 Safari/537.36")
session.headers.update({
    "User-Agent": UA_CHROME,
    "Accept-Language": "en-US,en;q=0.9,hi;q=0.8",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
})

CAPTCHA Solving

What is it? CAPTCHAs are visual puzzles designed to block automated access. Indian government portals typically use simple image CAPTCHAs: a distorted image of 4-6 alphanumeric characters with colored noise lines drawn over them. These are solvable with OCR (Optical Character Recognition) using Tesseract, an open-source OCR engine.

The pipeline: Download CAPTCHA image, remove noise (colored lines), convert to black-and-white, run OCR, validate the result. Success rate is typically 60-80%, so you retry up to 5 times with fresh CAPTCHAs.

See references/captcha-solving.md for the OCR pipeline, noise removal techniques, Tesseract configuration, and the important discovery that some CAPTCHAs are client-side-only (validated in JavaScript but not on the server, so you can submit any value).

Quick pattern:

from PIL import Image
import pytesseract, numpy as np
from io import BytesIO

def solve_captcha(img_bytes):
    img = Image.open(BytesIO(img_bytes)).convert("RGBA")
    arr = np.array(img)
    # Remove red noise lines: find pixels where Red is high, Green and Blue are low
    red = (arr[:,:,0] > 150) & (arr[:,:,1] < 100) & (arr[:,:,2] < 100)
    arr[red] = [255, 255, 255, 255]  # replace red pixels with white
    # Convert to grayscale, then binary (black text on white background)
    gray = Image.fromarray(arr).convert("L")
    binary = gray.point(lambda x: 255 if x > 128 else 0)
    # Run Tesseract OCR with restricted character set
    text = pytesseract.image_to_string(
        binary, config="--psm 7 -c tessedit_char_whitelist=0123456789abcdefghijklmnopqrstuvwxyz"
    ).strip()
    return text if len(text) >= 3 else None  # reject gibberish

Async Concurrency

What is it? Synchronous scraping sends one request at a time: send request, wait for response, process it, send next request. Most of that time is spent waiting for the network. Async (asynchronous) scraping sends many requests simultaneously and processes responses as they arrive, using a single thread. This can achieve 30-100x throughput improvement.

Key concepts:

• asyncio: Python's built-in async framework. Uses an "event loop" that manages many concurrent tasks.
• aiohttp: An async HTTP client (like requests but non-blocking).
• Semaphore: A counter that limits how many tasks run simultaneously. Semaphore(30) means at most 30 requests in flight at once.
• gather: Runs multiple async tasks concurrently and collects their results.

See references/async-patterns.md for multi-session architectures, CSRF token management, chunked JSONL output, batch processing, and adaptive query splitting.

Quick pattern:

import aiohttp, asyncio

async def fetch(session, sem, url):
    async with sem:  # wait if 30 requests are already in flight
        async with session.get(url) as r:
            return await r.json()

async def main(urls, concurrency=30):
    sem = asyncio.Semaphore(concurrency)  # max 30 concurrent requests
    connector = aiohttp.TCPConnector(limit=concurrency + 2)  # TCP connection pool
    async with aiohttp.ClientSession(connector=connector) as session:
        tasks = [fetch(session, sem, url) for url in urls]
        return await asyncio.gather(*tasks, return_exceptions=True)

Process-Level Parallelism

When to use instead of async: When you need to scrape multiple independent targets (e.g., 22 districts), each with its own session, proxy, and progress state. Process-level parallelism means running separate Python processes, each completely independent. No shared state, no race conditions, no async complexity.

tmux-based orchestration: tmux is a terminal multiplexer that lets you run multiple terminal sessions in one window. Each pane runs an independent scraper process.

# Launch 22 workers in tmux, each scraping a different district with a different proxy
for i in $(seq 1 22); do
    tmux send-keys -t scrape:0.$((i-1)) \
        "python scrape.py --district $i --proxy-port $((8000+i))" Enter
done

Each worker is a separate Python process with its own session, proxy, and progress file. No shared state means no synchronization bugs.

Data Storage Patterns

The key insight: Never store everything in one giant file. If it corrupts, you lose everything. Store per-entity files as the source of truth, and rebuild combined files from parts.

Scale	Format	Pattern	Why
<10K records	Single CSV/JSON	Write at end	Simple, fast
10K-100K	Per-entity JSON + combined CSV	Atomic writes, periodic combine	Crash-safe, inspectable
100K-1M	Chunked JSONL (5K records/chunk)	canonical_00000.jsonl, etc.	Streaming-friendly, append-only
>1M	Per-entity JSON sharded by prefix	ptin_json/{first_4_chars}/{id}.json	Avoids filesystem limits (>100K files per dir is slow)
Spatial data	GeoPackage (.gpkg) or GeoJSON	geopandas for write	Spatial indexing, no 2GB limit like shapefile

JSONL (JSON Lines): One JSON object per line, newline-separated. Unlike regular JSON arrays, you can append records without loading the whole file. Each line is independently parseable, so a corrupt line doesn't destroy the whole file.

Incremental combined file: Keep per-entity files as source of truth. Rebuild the combined file by scanning all entity files. This means the combined file is always regenerable and never the authoritative copy.

def build_combined(entity_dir, output_csv):
    rows = []
    for f in sorted(entity_dir.glob("*.json")):
        rows.extend(json.loads(f.read_text()))
    pd.DataFrame(rows).to_csv(output_csv, index=False)

Common Mistakes

Mistake	Why It's Bad	Fix
Writing directly to final output path	Crash mid-write = corrupt file that looks "done"	Use atomic writes (tmpfile + os.replace)
No resume support	Crash at record 500K = start over from record 1	Add progress tracking before writing any scraping logic
Fixed delays between requests	Detectable pattern; bot detectors flag exact intervals	Use randomized delays with jitter
Catching bare Exception and continuing	Hides real bugs (KeyError, AttributeError) behind silent failures	Catch specific exceptions; let unexpected errors surface
Storing all data in memory	1M records * 1KB each = 1GB RAM; crash = everything lost	Write incrementally (every N records or every M seconds)
Single giant output file	Corrupt file = all data lost; slow to read/write	Shard by entity or chunk; rebuild combined file from parts
Hardcoding URLs and selectors	Site changes break your scraper silently	Use constants at top of file; extract tokens from page config dynamically
Treating HTTP 500 as always-error	Some APIs return 500 for "not found" (MCD UPIC details)	Add bail_on_500 flag for endpoints that misuse status codes
No logging	Can't diagnose why scraping stopped or slowed	Log: items processed, errors, rate, ETA. Use tqdm for progress
Using verify=False for SSL errors	Disables all certificate validation, vulnerable to MITM	Embed the missing intermediate CA cert into a custom bundle

Scraper Architecture Template

A minimal but production-ready scraper skeleton with all the patterns above:

"""Scraper template for government portals."""
import argparse, json, logging, random, time
from pathlib import Path
import requests
from bs4 import BeautifulSoup
from tqdm import tqdm

# --- Config (all tunable parameters at the top) ---
UA = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 ..."
RATE_MIN, RATE_MAX = 1.5, 3.0   # seconds between requests
MAX_RETRIES = 3
SAVE_INTERVAL = 50              # save progress every N items

# --- Session factory (creates a fresh, properly configured session) ---
def make_session():
    s = requests.Session()
    s.headers.update({"User-Agent": UA})
    # Add: retry adapter, proxy, custom cert bundle as needed
    return s

# --- Progress tracking (atomic reads and writes) ---
def load_progress(path):
    return json.loads(path.read_text()) if path.exists() else {"done": []}

def save_progress(path, progress):
    tmp = path.with_suffix(".tmp")
    tmp.write_text(json.dumps(progress))
    tmp.replace(path)  # atomic

# --- Main scraping loop ---
def scrape(items, output_dir, progress_path):
    session = make_session()
    progress = load_progress(progress_path)
    done_set = set(progress["done"])  # set for O(1) lookup

    for i, item in enumerate(tqdm(items)):
        if item["id"] in done_set:
            continue  # already scraped in a previous run

        # ... scrape item ...
        # ... save per-item output atomically ...

        done_set.add(item["id"])
        progress["done"].append(item["id"])
        if i % SAVE_INTERVAL == 0:
            save_progress(progress_path, progress)
        time.sleep(random.uniform(RATE_MIN, RATE_MAX))

    save_progress(progress_path, progress)  # final save

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--resume", action="store_true")
    parser.add_argument("--output-dir", type=Path, default=Path("output"))
    args = parser.parse_args()
    # ... load items, call scrape() ...