원클릭으로 Manus에서 모든 스킬 실행

snapshot-churns-out-one-outcome-class-biases-the-rate

스타1

포크0

업데이트2026년 7월 8일 12:14

A RATE (accept rate, win rate, conversion rate, churn rate, approval rate) computed over a current-state SNAPSHOT table is biased when the snapshot differentially retains/prunes one outcome class — losers/denials/cancellations often churn out faster (status overwritten to Withdrawn/Closed/Deleted) than winners. Use when: (1) computing a per-segment accept/admit/ conversion rate FROM a snapshot/upserted table, (2) a rate looks suspiciously high/clean, (3) a denominator is "everyone with a decision" but read from current state, (4) ranking segments by a rate where the ranking itself is the deliverable. Fix: count numerator AND denominator from the immutable event/history log (the full outcome universe), use the snapshot only for attribution. NOT PIT feature leakage (that's a future value back-stamped onto a past row); this is present-tense selection bias in the rate's denominator.

설치

Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.

Manus에서 실행

출처

wan-huiyan

wan-huiyan/overnight-workflows

GitHub 저장소 열기 Creator 저장소 보기

다운로드

Manus에서 실행

A snapshot that churns out one outcome class biases any rate computed over it

Problem

You compute a rate — accept rate, conversion rate, win rate, approval rate — as favorable_outcomes / all_decided, reading both from a current-state snapshot (an upserted table that keeps only the latest row per entity). If the snapshot differentially prunes one outcome class, the rate is biased and the per-segment ranking is distorted — even though every individual number "looks right."

The mechanism: unfavorable outcomes often churn out of the snapshot faster than favorable ones. A rejected application gets withdrawn/closed/re-coded (WD/WA/Closed/Cancelled) and the original "Denied" status is overwritten or the row drops; the accepted ones persist. So the snapshot over-retains the numerator class relative to the denominator class → the rate is inflated, and because the pruning rate differs by segment, the ranking is unevenly biased (not a constant offset).

Trigger conditions

Computing a per-segment rate (accept/admit/conversion/win/approval/pass) where the ranking is the deliverable (e.g. "which programs/reps/cohorts are most selective/effective").
The denominator is "everyone who reached a decision/outcome," read from a snapshot / upserted / latest-state table rather than an append-only event log.
A rate looks suspiciously high or clean (e.g. 95% accept) or a segment looks less selective/ effective than domain experts expect.
The table has churn states (Withdrawn, Cancelled, Closed, Deleted, Deferred) that can overwrite the original outcome.

Solution

Measure the pruning before trusting the rate. Join the snapshot to the immutable event/history log and compute retention by outcome class: what fraction of each outcome's events still appear in the snapshot. If retention differs across classes, the snapshot rate is biased.
Count the rate from the full outcome universe (the append-only event/history log), keyed on the entity id — numerator AND denominator. Use the snapshot only to attach attributes (segment, level, owner) via an inner join, and report the share of history-outcome rows that have no snapshot attribute row (those are the pruned ones you can't attribute — a visible coverage gap).
If attribution is impossible for the pruned class, label the metric honestly ("rate among currently-retained records — NOT the authoritative rate") and put "authoritative rate per segment" on the stakeholder question list.
Note the bias direction: over-retaining the favorable class inflates the rate, so a corrected read usually makes selective/low-rate segments look even more extreme — i.e. the bias generally understates, not overstates, a "this segment is different" headline (a useful robustness point).

Verification

Retention-by-outcome probe returns materially different fractions per class (the smoking gun).
The history-universe rate differs from the snapshot rate by more than rounding, and the per-segment ordering changes for at least some segments.
Independent re-derivation: an adversarial agent re-computing the rate from the raw event log reproduces the corrected number, not the snapshot one.

Example (illustrative — from a verified engagement)

Per-segment accept rate for a competitiveness ranking, first cut from a current-state snapshot (analytics.application_snapshot, an upserted latest-row-per-entity table):

Retention probe over one year of decisions: the snapshot retains ~95% of accepted applications but only ~78% of rejected ones (denials churn out under WD/WA re-codes). Pooled accept rate ~87% (snapshot) vs ~84% (true) — and the per-segment ranking was unevenly biased.
Fix: count ever_accept / ever_reject / ever_waitlist from the append-only status-history table (application_status_events, keyed on application_id), the full decision universe; use the snapshot only to attach the segment attribute (program / level). The most selective segment came out at a verified ~11% accept (a highly selective program) — the corrected (more selective) number, confirming the bias understated the headline.

Notes

Distinct from PIT feature leakage ([[snapshot-feature-pit-leak-rate-measure-history-change-after-target]], [[null-bucket-hides-progressors-in-snapshot-training]]): those are about a feature/label on a training row being back-stamped from current state. This is present-tense selection bias in a rate's denominator on an analytics read — no target date involved.
Related: [[pit-history-reconstruction-needs-canonical-code-source-of-truth]] (the history log needs a canonical code set), [[probe-prior-status-before-bucketing-a-pit-code]] (churn codes like WD/WA can overwrite a prior decision — handle a fallback when the history has no decision event but the snapshot shows a withdrawal-after-decision).
The same trap hits any churning snapshot: CRM win-rate (lost deals deleted), support resolution-rate (spam tickets purged), loan approval-rate (declined apps archived). Always probe retention-by-outcome before ranking segments by a rate read from current state.
See also [[near-absorbing-state-rate-deflated-by-snapshot-residue]] — the sibling observation-time trap: same "don't trust a rate off current state," but the bias is when you observe a fast-draining pool (the residue's conversion rate), not which outcome class the snapshot retains.

이 저장소의 다른 Skills

같은 저장소

ship-the-correction-to-every-rendered-surface

wan-huiyan/overnight-workflows

The LAST MILE of a de-stale (protocol step 9): a corrected observational finding is not shipped until every RENDERED surface matches the ledger. Use AFTER the honest ledger (step 8) whenever a figure or a marker-vs-lever / causal verdict changes and that number already appears on a live surface — a served/baked HTML page, an embedded chart image, a client+dashboard "twin" pair, a dashboard payload or cache table. The trap: you correct the TEXT/captions and believe you shipped, but the baked chart image still renders the retired number (and its old causal title), a twin doc still asserts the retired claim, or a cached payload still serves the old value — so a corrected caption sits next to a chart that contradicts it. Also covers HOW to draw a retired-causal / marker-not-lever finding honestly. Trigger when you edit a figure that is displayed anywhere, or review a de-stale PR for completeness.

2026-07-171

overnight-multi-issue-implementation

wan-huiyan/overnight-workflows

Run an overnight autonomous workflow that takes a cluster of related GitHub issues (typically a P1 review-panel finding set) and ships them to merged PRs by morning. Use when: (1) the user wants 6-15 related issues closed in one autonomous run, (2) the issues split naturally into two PRs (e.g., hardening + features, or refactor + new-functionality), (3) the user is going to sleep and won't be available to merge PR1 between phases, (4) each issue has clear acceptance criteria so each task can be implemented + tested + reviewed independently. Specializes `superpowers:subagent-driven- development` for the "issues -> stacked PRs by morning" problem shape: stacks PR2 on PR1's branch (so PR2 doesn't wait for human PR1-merge mid- night), audits tracker IDs against main before claiming (concurrent sessions steal ids), runs final code-review subagent before proposing merge, and surfaces important findings as PR comments before squashing (so review trail survives). Sister plugin to `overnight-review-client- delivery` (

2026-07-171

funnel-stage-lift-needs-downstream-capacity-check

wan-huiyan/overnight-workflows

Guard the step where a VALID early-funnel lift becomes a recommendation. A segment genuinely starts/converts more at stage N — and the write-up is about to say "prioritize this segment" as if that buys the END outcome (enrollment, purchase, activation). Use when: (1) a finding, headline, card, or play recommends tilting effort toward a segment based on a stage-N rate (lead→start, start→submit, signup→trial); (2) the favored segment funnels into a capacity-capped or selective downstream stage (limited seats/inventory/approval slots, an admissions committee, a review queue); (3) you are REWORDING or promoting an existing funnel-stage finding — the reword inherits the original's stage scope, so re-check it; (4) a stakeholder asks "but doesn't that program/product only take a few?". The lift can be real AND the recommendation still over-promise: extra stage-N entrants feed a bottleneck, not the end outcome. Distinct from cohort-milestone-lift-is-funnel-position-not-effect (which attacks the lift's VALIDITY); here

2026-07-101

observational-analysis-rigor

wan-huiyan/overnight-workflows

The validity gate for any finding from OBSERVATIONAL data (no randomization). Use this BEFORE trusting or shipping a data finding, and when reviewing/verifying one. Runs a 9-step protocol — leak-free point-in-time cohort · probe outcome−anchor before an event-anchored design · decompose pooled rates by the structural axis (composition / Simpson) · de-confound with a multiple-testing-corrected regression · marker-vs-lever discipline · coverage-limited-join unbiasedness · triple-probe headlines · honest de-stale ledger · ship the correction to every rendered surface — that catches the finding which is genuinely surprising but WRONG (a composition artifact, a leak, an anchor-timing inversion, or an intent marker sold as a lever). Invoke whenever you analyze funnels, cohorts, lift, conversion rates, feature signals, retention, or any "do X → they convert / advance" claim from observational data; whenever a gradient or effect "looks too good"; and whenever you verify a finding before it reaches a stakeholder. The

2026-07-101

amplifying-an-existing-number-is-a-provenance-recheck-trigger

wan-huiyan/overnight-workflows

A request to ELEVATE / amplify / feature an existing number that's already in a deliverable — turn a prose figure into a chart, make it a headline, restore "the breakdown we had" — is itself the trigger to re-verify that number's provenance BEFORE amplifying it. Use when: (1) asked to add a visual/chart for a number that already exists as text, (2) asked to make an existing stat more prominent, (3) a stakeholder says "we had this insight, put it back / show it bigger", (4) a number was seeded earlier as "client feedback" / "stakeholder feedback" / "per the reviewer" and is being built on. Pre-existing and especially stakeholder-contributed numbers carry FALSE AUTHORITY — they look already-vetted, so they skip the provenance scrutiny a brand-new number gets, and may trace to nothing. Verify (grep the source, re-derive with triple-probe) before making them louder. Prevents amplifying a fabrication.

2026-07-081

blind-rederive-pass-when-orchestrator-already-read-the-answer

wan-huiyan/overnight-workflows

When you orchestrate verification subagents AFTER reading the claims/numbers yourself, do NOT hand the agents the published number to "check" — that anchors them into replicate-and-confirm, not independent verification, and their agreement proves nothing because you already knew the answer. Instead BLIND the re-derivation pass: give each agent the QUESTION, not the number ("what % of X are Y?", never "verify it's 43%"). Put the published number only in front of a SECOND, adversary pass whose job is to break it via a DIFFERENT join/source. Use when: (1) you've read a finding doc / analysis / PR claim in full and are about to dispatch agents to verify it; (2) you're writing a re-derivation/triple-check/fact-check subagent prompt and tempted to include the target number "so they know what to confirm"; (3) a handoff asks you to "independently re-derive" something whose answer is already written in the doc you were handed; (4) you want subagent convergence to mean real agreement, not anchoring on a number you leak

2026-07-081

name	snapshot-churns-out-one-outcome-class-biases-the-rate
description	A RATE (accept rate, win rate, conversion rate, churn rate, approval rate) computed over a current-state SNAPSHOT table is biased when the snapshot differentially retains/prunes one outcome class — losers/denials/cancellations often churn out faster (status overwritten to Withdrawn/Closed/Deleted) than winners. Use when: (1) computing a per-segment accept/admit/ conversion rate FROM a snapshot/upserted table, (2) a rate looks suspiciously high/clean, (3) a denominator is "everyone with a decision" but read from current state, (4) ranking segments by a rate where the ranking itself is the deliverable. Fix: count numerator AND denominator from the immutable event/history log (the full outcome universe), use the snapshot only for attribution. NOT PIT feature leakage (that's a future value back-stamped onto a past row); this is present-tense selection bias in the rate's denominator.
author	Claude Code
version	1.0.0
date	"2026-06-24T00:00:00.000Z"
disable-model-invocation	true

A snapshot that churns out one outcome class biases any rate computed over it

Problem

Trigger conditions

Computing a per-segment rate (accept/admit/conversion/win/approval/pass) where the ranking is the deliverable (e.g. "which programs/reps/cohorts are most selective/effective").
The denominator is "everyone who reached a decision/outcome," read from a snapshot / upserted / latest-state table rather than an append-only event log.
A rate looks suspiciously high or clean (e.g. 95% accept) or a segment looks less selective/ effective than domain experts expect.
The table has churn states (Withdrawn, Cancelled, Closed, Deleted, Deferred) that can overwrite the original outcome.

Solution

Measure the pruning before trusting the rate. Join the snapshot to the immutable event/history log and compute retention by outcome class: what fraction of each outcome's events still appear in the snapshot. If retention differs across classes, the snapshot rate is biased.
Count the rate from the full outcome universe (the append-only event/history log), keyed on the entity id — numerator AND denominator. Use the snapshot only to attach attributes (segment, level, owner) via an inner join, and report the share of history-outcome rows that have no snapshot attribute row (those are the pruned ones you can't attribute — a visible coverage gap).
If attribution is impossible for the pruned class, label the metric honestly ("rate among currently-retained records — NOT the authoritative rate") and put "authoritative rate per segment" on the stakeholder question list.
Note the bias direction: over-retaining the favorable class inflates the rate, so a corrected read usually makes selective/low-rate segments look even more extreme — i.e. the bias generally understates, not overstates, a "this segment is different" headline (a useful robustness point).

Verification

Retention-by-outcome probe returns materially different fractions per class (the smoking gun).
The history-universe rate differs from the snapshot rate by more than rounding, and the per-segment ordering changes for at least some segments.
Independent re-derivation: an adversarial agent re-computing the rate from the raw event log reproduces the corrected number, not the snapshot one.

Example (illustrative — from a verified engagement)

Per-segment accept rate for a competitiveness ranking, first cut from a current-state snapshot (analytics.application_snapshot, an upserted latest-row-per-entity table):

Retention probe over one year of decisions: the snapshot retains ~95% of accepted applications but only ~78% of rejected ones (denials churn out under WD/WA re-codes). Pooled accept rate ~87% (snapshot) vs ~84% (true) — and the per-segment ranking was unevenly biased.
Fix: count ever_accept / ever_reject / ever_waitlist from the append-only status-history table (application_status_events, keyed on application_id), the full decision universe; use the snapshot only to attach the segment attribute (program / level). The most selective segment came out at a verified ~11% accept (a highly selective program) — the corrected (more selective) number, confirming the bias understated the headline.

Notes

Distinct from PIT feature leakage ([[snapshot-feature-pit-leak-rate-measure-history-change-after-target]], [[null-bucket-hides-progressors-in-snapshot-training]]): those are about a feature/label on a training row being back-stamped from current state. This is present-tense selection bias in a rate's denominator on an analytics read — no target date involved.
Related: [[pit-history-reconstruction-needs-canonical-code-source-of-truth]] (the history log needs a canonical code set), [[probe-prior-status-before-bucketing-a-pit-code]] (churn codes like WD/WA can overwrite a prior decision — handle a fallback when the history has no decision event but the snapshot shows a withdrawal-after-decision).
The same trap hits any churning snapshot: CRM win-rate (lost deals deleted), support resolution-rate (spam tickets purged), loan approval-rate (declined apps archived). Always probe retention-by-outcome before ranking segments by a rate read from current state.
See also [[near-absorbing-state-rate-deflated-by-snapshot-residue]] — the sibling observation-time trap: same "don't trust a rate off current state," but the bias is when you observe a fast-draining pool (the residue's conversion rate), not which outcome class the snapshot retains.