captcha-detection-metrics

# Analyzing the CAPTCHA detection metrics ## Introduction Context and guidance for analyzing the `captcha_detection.*` metrics defined in `metrics.yaml` and sent in the `captcha-detection` custom ping (`pings.yaml`). Everything lives in `toolkit/components/captchadetection/` in mozilla-central: `CaptchaDetectionChild.sys.mjs` (detects captchas in-frame), `CaptchaDetectionParent.sys.mjs` (records metrics), `CaptchaDetectionPingUtils.sys.mjs` (ping submission). File paths below are relative to that directory unless stated otherwise. Purpose (per `pings.yaml`): analyze how often CAPTCHAs appear in the wild, broken down by users' anti-tracking / anti-fingerprinting settings. This document is about *interpreting and querying* the data — it records the non-obvious things that are easy to get wrong. The ping is **captcha-triggered**: it is submitted when a client encounters a detected captcha (an outcome event flags unsubmitted data; `pages_visited` alone does not trigger submission). It is therefore NOT sent by every client — the reporting population is "clients who hit a detected captcha", which is the right denominator to keep in mind when sizing any analysis. Detection is DOM/heuristic based: an in-frame actor identifies a vendor's captcha and reports load and outcome events, which the parent turns into counters. The exact triggers are vendor-specific (see Metrics Overview). Because detection depends on the vendor's page structure, it can miss cases the heuristic doesn't match, and a trend change can reflect a vendor markup change rather than user behavior. ## Metrics Overview The ping carries, per client: - **Per-vendor counters** for each supported captcha provider (below). Every counter has a normal and a `_pbm` (Private Browsing Mode) variant, with one historical exception: `datadome_oc_pbm` was only added in Firefox 154 (bug 2054037). It is NULL, not 0, on submissions from older builds — including current ESR — so any window reaching back before 154 must treat it as missing rather than as a zero load count (see Datadome). - **Context metrics** recorded at collection time: `privacy_*` (tracking protection incl. cryptomining/known-fingerprinter toggles, fingerprinting protection, resistFingerprinting) and `network_cookie_*` (cookie behavior, opt-in partitioning). These describe the user's settings, not captcha events. - **`pages_visited`** (+ `_pbm`) — a browsing denominator. Counter-suffix legend: `oc` loaded, `ps` shown, `pc`/`cc` passed/completed, `pf`/`cf` failed, `ac` auto-completed, `bl` blocked. **Not all vendors record all suffixes** — see each subsection. Pair non-`_pbm` counters with `pages_visited` for normal browsing and `_pbm` counters with `pages_visited_pbm` for private browsing; never mix the two. **Counters are batched across pings, so a single ping can look inconsistent.** The ping accumulates counters and submits periodically (`CaptchaDetectionPingUtils`, ~once/24h), not once per captcha. A captcha's lifecycle can straddle a submission boundary: the widget loads (incrementing `oc`) before a ping is sent, and the user interacts / it resolves (`ps`/`ac`/`cc`/`cf`/`pc`/`pf`) only after — so the load lands in one ping and its outcome in the next (or an outcome lands in a ping whose load already shipped). Within a single ping you can therefore see an outcome with no matching load, or a load whose outcome never arrives, and a per-ping ratio can exceed 100%. These straddles are small and mostly cancel in aggregate; a *large* imbalance (an outcome far exceeding its load count) instead signals corrupt / non-organic data — which is why the invalid-ratio checks (Data Analysis Techniques) use a gross tolerance rather than a strict inequality. ### ArkoseLabs Counters: - `arkoselabs_oc` — an Arkose puzzle loaded (the game-like "FunCaptcha"). There's no checkbox to click first, so a load basically means the puzzle appeared. - `arkoselabs_pc` — the puzzle was solved (the Arkose server confirmed "correct"). - `arkoselabs_pf` — the puzzle was gotten wrong (the server said "wrong"). - `arkoselabs_solutions_required` — how many rounds ("waves") that one puzzle made the person do. Stored as a spread of values, not a single number. `pc`/`pf` come from a network-response observer on the Arkose API (`/fc/ca/`): the terminal `solved` boolean -> `pc` (true) or `pf` (false). **Arkose has a real `pf` fail counter**, so `pc/(pc+pf)` is a genuine solve rate (not a lower bound like the widget vendors). NOTE: `arkoselabs_ps` exists in the schema but is **never recorded (always 0)** — nothing in the code increments it. Ignore it. Also `solutions_required.count` is the deprecated internal field (bug 1799509, unpopulated) — the sample count is `pc + pf`, so mean rounds = `solutions_required.sum / (pc + pf)`. **Ratios (normal browsing):** - prevalence = `oc / pages_visited` (~0.0043%) — how often an Arkose puzzle shows up, out of all the pages people load. - **solve rate = `pc / (pc + pf)`** (~67%) — of the puzzles that got a clear win-or-lose answer, the share that were solved. This is the number to trust, because wins and losses are counted the same way. - completion = `(pc + pf) / oc` (~34%) — ArkoseLabs's version of the **interacted** (resolved) rate the other providers have; the dashboard groups it in the *Interacted* section. Of the puzzles that loaded, how many we actually saw end in a win or a loss. Only rough: people leave or close the tab, and we don't always catch the ending. - mean solutions required = `solutions_required.sum / (pc + pf)` (~2.26) — how many rounds a typical puzzle asks for. - Sanity check: wins + losses can't be more than loads (`pc + pf <= oc`), give or take the rare edge case. NOTE: `solutions_required` counts **waves within a single challenge, not retries**. FunCaptcha shows several image "waves" in sequence; the response observer counts each `/fc/ca/` wave and, when a terminal `solved` boolean arrives, records one `pc`/`pf` plus that accumulated wave-count, then resets. So the mean-rounds figure (~2.26) and the solve rate are **independent**: a 2-wave challenge is still ONE session ending in ONE outcome, and the waves are not themselves pass/fail events — so "avg ~2 waves" does NOT imply a ~50% solve rate. The ~67% solve rate means ~33% of *decided* challenges came back `solved=false` (genuine, server-confirmed fails); abandoned challenges never get a terminal boolean, so they are excluded from both `pc` and `pf` (this is why the solve rate is a true rate, not a lower bound like the widget vendors). ### AWS WAF — defined but DEAD (do not use) Counters (all scaffolded, never wired up): - `awswaf_ps` - `awswaf_pc` - `awswaf_pf` - `awswaf_solutions_required` They exist in the schema / Glean dictionary, but there is no AWS WAF handler or recording code (not in this tree, not in searchfox/mozilla-central), and the production table carries **zero data** for all of them over 90 days (verified: `max(awswaf_ps)=0`, no rows > 0). No ratios are computable. Treat as absent until an implementation lands; a cleanup bug to either implement or remove these metrics would be reasonable. ### Cloudflare Turnstile Counters: - `cloudflare_turnstile_oc` — a Turnstile challenge loaded on the page. Turnstile usually checks you in the background, so a load does NOT mean the person did anything. - `cloudflare_turnstile_cc` — the challenge ended in a pass (the green check). This includes passes that happen automatically with no clicking. - `cloudflare_turnstile_cf` — the challenge ended in a fail. What they mean — the child actor (`CaptchaDetectionChild.sys.mjs`): 1. Identifies a Turnstile challenge iframe by its URL and constructs a handler -> increments `..._oc`. 2. Watches the widget's shadow DOM for a `#success` / `#fail` div to become visible -> increments `..._cc` or `..._cf`. The observer disconnects after the first outcome, so **at most one `cc`/`cf` per `oc`**. `oc` fires when the challenge iframe loads (the handler is constructed on the iframe-URL match), independent of visibility: a Turnstile that never surfaces a `#success`/`#fail` div (invisible/managed mode, or torn down before it shows) yields an `oc` with no `cc`/`cf` outcome. Crucially, the handler only checks *which of* `#success`/`#fail` is displayed — there is **no interaction detection at all**. A managed / non-interactive Turnstile that passes on its own (the success checkmark appears with zero user action) increments `cc` identically to a pass the user actually worked through. So **`cc` is not evidence of user interaction**, and Turnstile records only `oc`/`cc`/`cf` (no `ps`/`ac`) — automatic and interactive passes cannot be distinguished. This gap is tracked in **bug 2054267** (under meta 2054266): add a Turnstile auto-vs-interactive signal so it can be segmented like reCAPTCHA/hCaptcha. Appropriate ratios (`oc`/`cc`/`cf` are separate counters at different lifecycle points — do NOT use raw `cc/oc` as a "solve rate"): - **Pass rate = `cc / (cc + cf)`** — of the challenges we actually saw finish (pass or fail), the share that passed. This is the cleanest "did it succeed" number, and it's always between 0 and 100%. Show `cc+cf` next to it, and don't trust it when only a handful of challenges finished. - **Interacted = `(cc + cf) / oc`** — of the Turnstiles that loaded, how many we saw finish at all. The rest never showed a pass or fail (they ran in the background, got closed, or our detector missed them) — that is NOT the same as the person "giving up." Despite the name, this is really "how many resolved", NOT "a human interacted": Turnstile often passes on its own, and those silent passes still count as `cc`. - **Prevalence = `oc / pages_visited`** — how often a Turnstile loads per page. Very small (around 0.005%). Good for "how common is Turnstile". - Don't use **`cc / oc`** as a "solve rate" — it mostly reflects how often Turnstile runs in the background, not whether people succeed. Caveat: pass rate is bounded 0-100% by construction; `Interacted`/`Prevalence` are not bounded at daily granularity (see the invariant under Data Analysis). ### Datadome Counters: - `datadome_oc` — Datadome stepped in with its full-page gate screen. This only fires when Datadome actually interrupts the person, not just because the site uses Datadome. - `datadome_ps` — that screen showed a puzzle to solve. - `datadome_pc` — the puzzle was passed. - `datadome_bl` — the person was hard-blocked (a "no entry" page, no puzzle to even try). Recording (`recordDatadomeEvent`): a load reports `ps` (puzzle) or `bl` (hard block) — mutually exclusive; a pass reports `pc`. So `ps + bl <= oc`, `pc <= ps`. There is **no auto-success** signal (a silent pass loads no interstitial). `datadome_oc_pbm` was missing until bug 2054037 added it in **Firefox 154**, so PBM *load* data only exists from 154 onward (older builds, current ESR included, report PBM `ps`/`bl`/`pc` but no `oc`). On those older submissions the column is NULL rather than 0, so a PBM prevalence denominator built from it silently drops them instead of dividing by zero — scope any `oc_pbm` ratio to 154+ explicitly. **Ratios (normal browsing):** - intervention prevalence = `oc / pages_visited` — how often Datadome interrupts someone per page (it actually stepped in, not just "the site uses it"). - block share = `bl / (ps + bl)` — when Datadome steps in, how often it's a flat-out block versus a puzzle you can actually solve (~27%). The main Datadome number. - solve rate given puzzle = `pc / ps` — of the puzzles shown, how many got passed. This is a floor, not the true rate: there is no "failed" counter, the pass can go uncounted if the page moves on, and lots of people just leave the site — all of which look like "not solved." - block exposure = `bl / pages_visited` — how often someone gets hard-blocked per page. A direct "how much does this hurt people" measure. Dropped as redundant (since `ps + bl ≈ oc`): puzzle share (= 1 − block share) and puzzle non-completion (= 1 − solve rate). ### Google reCAPTCHA v2 Counters: - `google_recaptcha_v2_oc` — the "I'm not a robot" checkbox loaded on the page. (Only the visible checkbox kind; the fully-invisible kind isn't counted.) It fires when the box loads, not when someone clicks it — so most of these are just sitting on the page, unused. - `google_recaptcha_v2_ps` — the person was shown an image puzzle ("pick all the buses"). - `google_recaptcha_v2_pc` — that image puzzle was solved. - `google_recaptcha_v2_ac` — the person was waved through with no puzzle at all. Recording (`updateGRecaptchaV2State`): `ImagesShown` -> `ps`; a checkmark -> `ac` if images were never shown, else `pc`. Per widget `ac` and `ps` are meant to be mutually exclusive with `pc <= ps` and `ac + ps <= oc` — but in release data **`ps` grossly over-counts** (see the "shown" note below), so `ac + ps <= oc` does NOT hold in practice. **No fail counter.** Interactions = `ac + ps`. **Ratios (normal browsing):** - loaded per page = `oc / pages_visited` — how often the checkbox loads per page. - interacted per page = `(ac + ps) / pages_visited` (or `(ac + ps) / oc` per load) — how often something actually happened with it (a wave-through or a puzzle). Most loaded boxes never get this far. - auto-success % of interactions = `ac / (ac + ps)` — of the boxes that did something, how many waved the person through with no puzzle. - success rate (challenged) = `pc / ps` — of the puzzles shown, how many got solved. - non-completion rate (challenged) = `(ps - pc) / ps` — the flip side: fails and give-ups lumped together (there is no separate "failed" counter). **Cross-provider caveat (validated against 90d release data, desktop + Android):** `pc/ps` is a within-provider **lower bound**, not a clean solve-success rate, and is **not comparable across providers**. The observed ~43% (reCAPTCHA) vs ~94% (hCaptcha) is NOT relative solve skill; two things drive it: - **`pc` loss + abandonment (the main driver):** the post-solve checkmark (`pc`) is often lost when solving submits the form / navigates away before it is recorded, and reCAPTCHA's multi-round image puzzles are genuinely abandoned more than hCaptcha's. Both push reCAPTCHA's `pc/ps` down (measurement + behaviour). - **Different definitions of "shown":** reCAPTCHA `ps` fires on challenge-element *existence* (`rc-imageselect` present in the bframe); hCaptcha `ps` fires on *visibility* (`aria-hidden` flips). reCAPTCHA's existence check **does over-count**, confirmed in 90d release data: `ps` re-fires on the persistent element, reaching **76,156 in a single ping** and pushing `ac + ps > oc` for ~165k clients (~0.4%). Being fixed / aligned in **bug 2054272** (under meta 2054266). `ps` is also interaction-gated — only ~1.6% of `oc` — so it does NOT mean "the widget was present"; it means a challenge was *instantiated for an engaged user*. Separately, `oc` counts widget *loads*: reCAPTCHA sits on ~6% of pageviews with only ~2.4% ever interacted (passive embeds / periodic token-refresh reloads), so oc-denominated ratios mean "widget present", not "user challenged". (That passive majority shows up in the *interacted* rate, not in `pc/ps` — ignored widgets never reach `ps`.) ### hCaptcha Counters: - `hcaptcha_oc` — the hCaptcha checkbox loaded on the page. Like reCAPTCHA, this fires at load, not when someone actually uses it. - `hcaptcha_ps` — the person was shown the puzzle. - `hcaptcha_pc` — the puzzle was passed. - `hcaptcha_ac` — the person was waved through with no puzzle. Same recording model and mutual exclusivity as reCAPTCHA v2; no fail counter. Unlike reCAPTCHA, hCaptcha `ps` fires when the challenge frame becomes **visible** (`aria-hidden` -> false) — a tighter "genuinely shown" denominator that does not have reCAPTCHA's existence over-count (max `ps` per ping ~400, not ~76k), so `ac + ps <= oc` holds far more tightly. That, plus hCaptcha being deployed as hard gates (~43% of loads are interacted vs reCAPTCHA's ~2.4%), is why its `pc/ps` runs ~94%: a measurement + deployment difference, not that hCaptcha is "easier" to solve. **Ratios (normal browsing):** identical shape to reCAPTCHA v2 with the `hcaptcha_` prefix — loaded/page `oc/pages`, interacted/page `(ac+ps)/pages`, auto-success% `ac/(ac+ps)`, success `pc/ps`, non-completion `(ps-pc)/ps`. ## Known metric gaps (tracked) Improvements to these metrics are tracked under meta **bug 2054266**: - **bug 2054267** — Cloudflare Turnstile cannot distinguish automatic from interactive passes (`cc` counts silent auto-passes; no `ac`/`ps`). - **bug 2054272** — reCAPTCHA `ps` uses element *existence* while hCaptcha uses *visibility*; investigate and align so "shown" means the same across providers. - AWS WAF metrics are scaffolded but dead (see above) — implement or remove. Keep these in mind when comparing providers: the counters were not all designed to the same definition. ## Data Analysis Techniques Ratios are provider-specific (see Metrics Overview). Independent of provider, the techniques below control *which rows* enter the numerator/denominator. Always pair a counter with the matching-mode `pages_visited` and report the denominator count so small samples are visible. ### Cohorts by privacy settings (composite key) Build a composite key from the `privacy_*` / `network_cookie_*` context metrics and treat each distinct combination as a cohort. Split into a **non-PBM key** (the non-`_pbm` settings, paired with non-PBM counters) and a **PBM key** (the