build-abuse-detection-pipeline

name	build-abuse-detection-pipeline
description	Design an abuse/fraud/spam detection pipeline — inventory signals, decide rules vs. ML (or hybrid), wire signal → score → threshold → action/reviewer-queue, and set the operating point from a precision/recall tradeoff. Reach for this when the user asks to catch an abuse type, choose signals, or route to a reviewer queue. Used by abuse-detection-engineer (primary).

Skill: build-abuse-detection-pipeline

Invoked by: abuse-detection-engineer (primary). Co-driven with trust-safety-policy-lead so each score band maps to a proportional enforcement action.

When to invoke: "catch "; "rules or ML?"; "build a detection pipeline"; "where do I set the threshold?"; "how do I route to a reviewer queue?".

Output: a signal inventory + a rules-vs-ML decision + a signal → score → threshold → action/queue pipeline + the operating point (threshold + precision/recall + FP/FN rationale) + a reviewer-label feedback loop.

Procedure

1. Inventory the signals first (signals before models). Sweep five families: content (text/image/URL features), behavioral (action rate, session shape), graph/relationship (shared device/IP, follow rings), velocity (burst/new-account), and reputation (account age, prior strikes). Many abuses fall to a cheap rule on a strong signal.
2. Decide rules vs. ML vs. hybrid. Traverse ../../knowledge/enforcement-decision-tree.md: rules for the obvious, explainable, cold-start case; ML for the gray zone, high-volume, and adapting adversary. Mature stacks are hybrid — rules gate the obvious, the model scores the rest.
3. Wire the pipeline: signal → score → threshold → action vs. reviewer-queue. Auto-action only in the high-precision band; route the gray zone to a human-review queue (whose volume must be survivable).
4. Set the operating point from cost, not 0.5. Choose the threshold from the cost of a false positive (a real user wrongly punished) vs. a false negative (abuse that gets through) for that policy. State the precision/recall at that point on the PR curve.
5. Close the loop. Reviewer decisions are labels — feed them back to recalibrate the threshold and retrain. Monitor for evasion and drift; a static threshold decays as the adversary adapts.
6. Confirm the eval is sound, then mark seams. Send the eval to applied-statistics for metric CIs / sample size / class-imbalance handling; map the score band to an action via trust-safety-policy-lead; an LLM classifier → claude-app-engineering.

Worked example

User: "Catch fake-review spam on our product pages."

• Signals: velocity (N reviews / account / hour), graph (reviews from accounts sharing a device fingerprint), content (near-duplicate text), reputation (account age < 24h).
• Approach: hybrid — a rule auto-removes the device-ring burst (precision ~0.99, explainable); an ML scorer handles the gray zone of single suspicious reviews.
• Pipeline: score → ≥0.9 auto-remove (high-precision band) → 0.6–0.9 reviewer queue → <0.6 allow.
• Operating point: 0.9 chosen because a false positive (removing a genuine review) costs seller trust; precision 0.96 / recall 0.71 at that cutoff. Eval CI confirmed via applied-statistics.
• Loop: reviewer overturns recalibrate the 0.6–0.9 band weekly.

Guardrails

• Never report a threshold without the precision/recall it buys on a named eval set at a stated date.
• Never auto-action outside the high-precision band — the gray zone is a human-review case.
• A detector with no feedback loop rots; wire reviewer labels back in. See ../../knowledge/trust-safety-metrics.md.