원클릭으로 Manus에서 모든 스킬 실행

cohort-milestone-lift-is-funnel-position-not-effect

스타1

포크0

업데이트2026년 7월 8일 12:14

Diagnose the trap where a cohort defined by a LATE-funnel milestone (aid application filed, application submitted, deposit paid, event attended, document received) shows a strong raw outcome "lift" — and you're about to frame a client/dashboard action ("these convert higher, target/nudge them"). Use when: (1) a cohort's raw conversion/deposit/enrollment rate is much higher than a comparison group (+5–25pp) and the cohort membership is itself reached only AFTER getting partway down the funnel; (2) the comparison ("non-X") group is contaminated by a population that structurally CAN'T have the milestone (e.g. non-domestic applicants can't file a domestic aid form); (3) someone proposes a new feature or a "high-yield" nudge card off that gap. The raw gap is composition + funnel position, not a causal effect of the milestone. Decisive test = matched-grain stratification holding the confounders fixed in BOTH arms; sign-instability across strata is the fingerprint of residual composition; beware the seductive positi

설치

Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.

Manus에서 실행

출처

wan-huiyan

wan-huiyan/overnight-workflows

GitHub 저장소 열기 Creator 저장소 보기

다운로드

Manus에서 실행

A late-funnel-milestone cohort's raw "lift" is funnel position + composition, not effect

Problem

You're asked to surface a cohort defined by a milestone that members reach only after progressing down the funnel — "aid application completed", "application submitted", "deposited", "attended an event". The cohort's raw outcome rate dwarfs the comparison group (e.g. aid- complete enroll ~11% vs ~5%, deposit ~70% vs ~46%). It's tempting to frame a client action around it: "these convert higher — prioritize/nudge them." That framing is almost always wrong, because membership in the cohort is downstream of the very outcome you're crediting it for. This error class has produced retracted findings and published lifts that failed to reproduce on re-analysis, for exactly this reason.

Context / Trigger conditions

A cohort filter where the defining event happens late (post-application, post-decision).
A raw outcome gap that "looks great" and someone wants to ship/feature/nudge on it.
The "non-cohort" comparison group silently includes a population that cannot have the milestone (non-domestic applicants can't file a domestic aid form; no-application members can't have a decision date).
Trigger phrases: "aid-form completers convert way higher, let's target them", "deposited members enroll at 95% so deposit is our best signal", "event attendees are 5× — build the card", "add a feature, the lift is huge".

Solution — the decisive matched-grain test (do this BEFORE framing any action)

Name both confounds. (a) Composition: does the comparison group include a sub-pop that structurally can't have the milestone? (b) Funnel position: is the milestone reached only after a funnel step the cohort therefore over-represents?
Hold them fixed in BOTH arms and recompute. Restrict both the cohort and the comparison to the same composition slice (e.g. residency_status IN ('C','R') in both) AND the same funnel stage (e.g. application_stage = 'accepted' in both), then recompute the gap. If it collapses toward zero or flips sign, the raw gap was composition/position, not effect.
Check sign stability across strata. Compute the within-stage gap in several stages. If the sign is unstable (e.g. −2 / −12 / +9 pp), that instability is the fingerprint of residual composition — not a real effect in any direction.
Distrust the seductive positive cut. If the only cut that stays strongly positive does so by conditioning on a variable that (a) only exists for the cohort arm and (b) is a near- proxy for the outcome (e.g. aid_record_year = '<current-cycle>' ≈ "live currently-enrolling record"), that's asymmetric outcome-conditioning — the retracted-finding error class. Reject it.
Use realized labels, not predicted scores, and check whether the model already prices the milestone: if the matched-grain score gap vanishes/reverses, no new feature is warranted (see predicted-score-cohort-comparison-flips-on-realized-backtest).
Reframe the product action. If there's no defensible lift, frame the cohort as a process-eligibility fact ("Accepted, aid application on file: ready for aid-award packaging") — true by construction, needs no yield claim — NOT as a high-converter target. Add a build guardrail: copy must not imply "prioritize because they convert", or it collapses back into the rejected narrative. Tighten the cohort to the operational queue (exclude already-past-decision rows; current cycle only).

Verification

Raw gap large; matched-grain (both-arms) gap ≈ 0 or sign-unstable → confirmed composition.
The strong positive survives ONLY under asymmetric/outcome-adjacent conditioning → reject.
Realized-backtest and predicted-score orderings disagree → cite neither as a yield effect.

Example (aid-application-completed cohort)

Raw: aid-complete (has_aid_doc=1 within an aid-eligible slice) converts at a visibly higher rate than incomplete (say ~11% vs ~5%, a +6pp gap). But holding the aid-eligible slice fixed in both arms collapses the gap to ≈0 (even slightly negative), and within a fixed application stage the sign is unstable (it flips positive/negative across strata). The apparent lift was further inflated by a downstream-conditioned proxy: the has_aid_doc=1 arm was implicitly conditioned on a current-cycle aid record (a liveness proxy the incomplete arm never gets — members tied to a stale record converted at nearly zero). Verdict: no defensible lift; the model already prices aid-completion (matched-grain score gap ≈ 0). Correct framing = aid-packaging readiness (process-eligibility), with the cohort tightened to current-cycle still-open records.

Notes

The two confounds compound: strip composition first (it often does most of the work), then funnel position.
"Deposited → 95% enrollment" is the purest instance: deposit is almost the outcome.
This is the inverse framing discipline to null-bucket-hides-progressors-in-snapshot-training (which is about the cohort DEFINITION hiding signal); here the cohort definition manufactures a spurious one.

References

Internal analysis brief (RESULTS section).
Companion skills: predicted-score-cohort-comparison-flips-on-realized-backtest; marginal-lift-collapses-on-pre-event-temporal-restriction; ml-bug-blast-radius-natural-baseline-conflation.

이 저장소의 다른 Skills

같은 저장소

ship-the-correction-to-every-rendered-surface

wan-huiyan/overnight-workflows

The LAST MILE of a de-stale (protocol step 9): a corrected observational finding is not shipped until every RENDERED surface matches the ledger. Use AFTER the honest ledger (step 8) whenever a figure or a marker-vs-lever / causal verdict changes and that number already appears on a live surface — a served/baked HTML page, an embedded chart image, a client+dashboard "twin" pair, a dashboard payload or cache table. The trap: you correct the TEXT/captions and believe you shipped, but the baked chart image still renders the retired number (and its old causal title), a twin doc still asserts the retired claim, or a cached payload still serves the old value — so a corrected caption sits next to a chart that contradicts it. Also covers HOW to draw a retired-causal / marker-not-lever finding honestly. Trigger when you edit a figure that is displayed anywhere, or review a de-stale PR for completeness.

2026-07-171

overnight-multi-issue-implementation

wan-huiyan/overnight-workflows

Run an overnight autonomous workflow that takes a cluster of related GitHub issues (typically a P1 review-panel finding set) and ships them to merged PRs by morning. Use when: (1) the user wants 6-15 related issues closed in one autonomous run, (2) the issues split naturally into two PRs (e.g., hardening + features, or refactor + new-functionality), (3) the user is going to sleep and won't be available to merge PR1 between phases, (4) each issue has clear acceptance criteria so each task can be implemented + tested + reviewed independently. Specializes `superpowers:subagent-driven- development` for the "issues -> stacked PRs by morning" problem shape: stacks PR2 on PR1's branch (so PR2 doesn't wait for human PR1-merge mid- night), audits tracker IDs against main before claiming (concurrent sessions steal ids), runs final code-review subagent before proposing merge, and surfaces important findings as PR comments before squashing (so review trail survives). Sister plugin to `overnight-review-client- delivery` (

2026-07-171

funnel-stage-lift-needs-downstream-capacity-check

wan-huiyan/overnight-workflows

Guard the step where a VALID early-funnel lift becomes a recommendation. A segment genuinely starts/converts more at stage N — and the write-up is about to say "prioritize this segment" as if that buys the END outcome (enrollment, purchase, activation). Use when: (1) a finding, headline, card, or play recommends tilting effort toward a segment based on a stage-N rate (lead→start, start→submit, signup→trial); (2) the favored segment funnels into a capacity-capped or selective downstream stage (limited seats/inventory/approval slots, an admissions committee, a review queue); (3) you are REWORDING or promoting an existing funnel-stage finding — the reword inherits the original's stage scope, so re-check it; (4) a stakeholder asks "but doesn't that program/product only take a few?". The lift can be real AND the recommendation still over-promise: extra stage-N entrants feed a bottleneck, not the end outcome. Distinct from cohort-milestone-lift-is-funnel-position-not-effect (which attacks the lift's VALIDITY); here

2026-07-101

observational-analysis-rigor

wan-huiyan/overnight-workflows

The validity gate for any finding from OBSERVATIONAL data (no randomization). Use this BEFORE trusting or shipping a data finding, and when reviewing/verifying one. Runs a 9-step protocol — leak-free point-in-time cohort · probe outcome−anchor before an event-anchored design · decompose pooled rates by the structural axis (composition / Simpson) · de-confound with a multiple-testing-corrected regression · marker-vs-lever discipline · coverage-limited-join unbiasedness · triple-probe headlines · honest de-stale ledger · ship the correction to every rendered surface — that catches the finding which is genuinely surprising but WRONG (a composition artifact, a leak, an anchor-timing inversion, or an intent marker sold as a lever). Invoke whenever you analyze funnels, cohorts, lift, conversion rates, feature signals, retention, or any "do X → they convert / advance" claim from observational data; whenever a gradient or effect "looks too good"; and whenever you verify a finding before it reaches a stakeholder. The

2026-07-101

amplifying-an-existing-number-is-a-provenance-recheck-trigger

wan-huiyan/overnight-workflows

A request to ELEVATE / amplify / feature an existing number that's already in a deliverable — turn a prose figure into a chart, make it a headline, restore "the breakdown we had" — is itself the trigger to re-verify that number's provenance BEFORE amplifying it. Use when: (1) asked to add a visual/chart for a number that already exists as text, (2) asked to make an existing stat more prominent, (3) a stakeholder says "we had this insight, put it back / show it bigger", (4) a number was seeded earlier as "client feedback" / "stakeholder feedback" / "per the reviewer" and is being built on. Pre-existing and especially stakeholder-contributed numbers carry FALSE AUTHORITY — they look already-vetted, so they skip the provenance scrutiny a brand-new number gets, and may trace to nothing. Verify (grep the source, re-derive with triple-probe) before making them louder. Prevents amplifying a fabrication.

2026-07-081

blind-rederive-pass-when-orchestrator-already-read-the-answer

wan-huiyan/overnight-workflows

When you orchestrate verification subagents AFTER reading the claims/numbers yourself, do NOT hand the agents the published number to "check" — that anchors them into replicate-and-confirm, not independent verification, and their agreement proves nothing because you already knew the answer. Instead BLIND the re-derivation pass: give each agent the QUESTION, not the number ("what % of X are Y?", never "verify it's 43%"). Put the published number only in front of a SECOND, adversary pass whose job is to break it via a DIFFERENT join/source. Use when: (1) you've read a finding doc / analysis / PR claim in full and are about to dispatch agents to verify it; (2) you're writing a re-derivation/triple-check/fact-check subagent prompt and tempted to include the target number "so they know what to confirm"; (3) a handoff asks you to "independently re-derive" something whose answer is already written in the doc you were handed; (4) you want subagent convergence to mean real agreement, not anchoring on a number you leak

2026-07-081

name	cohort-milestone-lift-is-funnel-position-not-effect
description	Diagnose the trap where a cohort defined by a LATE-funnel milestone (aid application filed, application submitted, deposit paid, event attended, document received) shows a strong raw outcome "lift" — and you're about to frame a client/dashboard action ("these convert higher, target/nudge them"). Use when: (1) a cohort's raw conversion/deposit/enrollment rate is much higher than a comparison group (+5–25pp) and the cohort membership is itself reached only AFTER getting partway down the funnel; (2) the comparison ("non-X") group is contaminated by a population that structurally CAN'T have the milestone (e.g. non-domestic applicants can't file a domestic aid form); (3) someone proposes a new feature or a "high-yield" nudge card off that gap. The raw gap is composition + funnel position, not a causal effect of the milestone. Decisive test = matched-grain stratification holding the confounders fixed in BOTH arms; sign-instability across strata is the fingerprint of residual composition; beware the seductive positive cut that conditions ONE arm on a near-outcome proxy. Product fix: frame as process-eligibility ("ready for X"), not yield. See also: predicted-score-cohort-comparison-flips-on-realized-backtest, marginal-lift-collapses-on-pre-event-temporal-restriction, ml-bug-blast-radius-natural-baseline-conflation.
author	Claude Code
version	1.0.0
date	"2026-06-01T00:00:00.000Z"
disable-model-invocation	true

A late-funnel-milestone cohort's raw "lift" is funnel position + composition, not effect

Problem

Context / Trigger conditions

A cohort filter where the defining event happens late (post-application, post-decision).
A raw outcome gap that "looks great" and someone wants to ship/feature/nudge on it.
The "non-cohort" comparison group silently includes a population that cannot have the milestone (non-domestic applicants can't file a domestic aid form; no-application members can't have a decision date).
Trigger phrases: "aid-form completers convert way higher, let's target them", "deposited members enroll at 95% so deposit is our best signal", "event attendees are 5× — build the card", "add a feature, the lift is huge".

Solution — the decisive matched-grain test (do this BEFORE framing any action)

Name both confounds. (a) Composition: does the comparison group include a sub-pop that structurally can't have the milestone? (b) Funnel position: is the milestone reached only after a funnel step the cohort therefore over-represents?
Hold them fixed in BOTH arms and recompute. Restrict both the cohort and the comparison to the same composition slice (e.g. residency_status IN ('C','R') in both) AND the same funnel stage (e.g. application_stage = 'accepted' in both), then recompute the gap. If it collapses toward zero or flips sign, the raw gap was composition/position, not effect.
Check sign stability across strata. Compute the within-stage gap in several stages. If the sign is unstable (e.g. −2 / −12 / +9 pp), that instability is the fingerprint of residual composition — not a real effect in any direction.
Distrust the seductive positive cut. If the only cut that stays strongly positive does so by conditioning on a variable that (a) only exists for the cohort arm and (b) is a near- proxy for the outcome (e.g. aid_record_year = '<current-cycle>' ≈ "live currently-enrolling record"), that's asymmetric outcome-conditioning — the retracted-finding error class. Reject it.
Use realized labels, not predicted scores, and check whether the model already prices the milestone: if the matched-grain score gap vanishes/reverses, no new feature is warranted (see predicted-score-cohort-comparison-flips-on-realized-backtest).
Reframe the product action. If there's no defensible lift, frame the cohort as a process-eligibility fact ("Accepted, aid application on file: ready for aid-award packaging") — true by construction, needs no yield claim — NOT as a high-converter target. Add a build guardrail: copy must not imply "prioritize because they convert", or it collapses back into the rejected narrative. Tighten the cohort to the operational queue (exclude already-past-decision rows; current cycle only).

Verification

Raw gap large; matched-grain (both-arms) gap ≈ 0 or sign-unstable → confirmed composition.
The strong positive survives ONLY under asymmetric/outcome-adjacent conditioning → reject.
Realized-backtest and predicted-score orderings disagree → cite neither as a yield effect.

Example (aid-application-completed cohort)

Notes

The two confounds compound: strip composition first (it often does most of the work), then funnel position.
"Deposited → 95% enrollment" is the purest instance: deposit is almost the outcome.
This is the inverse framing discipline to null-bucket-hides-progressors-in-snapshot-training (which is about the cohort DEFINITION hiding signal); here the cohort definition manufactures a spurious one.

References

Internal analysis brief (RESULTS section).
Companion skills: predicted-score-cohort-comparison-flips-on-realized-backtest; marginal-lift-collapses-on-pre-event-temporal-restriction; ml-bug-blast-radius-natural-baseline-conflation.