data-product-validation

name	data-product-validation
version	0.1.0
description	Score whether a data product idea is worth building before committing resources. Validation scorecard, experiment design, and go/kill decisions. Use when evaluating feasibility, making go/no-go decisions, validating demand, sizing bets, or when someone asks "is this worth building?" or "should we invest in this?"
user-invocable	false

Validation Scorecard

Score each dimension 1-5. Multiply all five for a composite score (max 3,125).

1. Demand Frequency

How often do consumers need this answer?

Score	Frequency	Example
5	Daily or real-time	ICU readmission risk scores
4	Weekly	Regional performance reports
3	Monthly	Quarterly business reviews
2	Quarterly	Annual planning data
1	Rarely or once	Ad-hoc executive request

2. Decision Impact

What happens when consumers don't have this data?

Score	Impact	Example
5	Critical decision blocked	Can't discharge patients without risk assessment
4	Significant delay or cost	Team spends 20+ hours/week on manual workaround
3	Moderate inconvenience	Report takes 3 hours instead of 5 minutes
2	Minor friction	Slightly slower process, acceptable workaround exists
1	Mild inconvenience	Nice to have, nobody changes behavior without it

3. Workaround Effort

What are consumers doing today instead?

Score	Effort	Example
5	Custom tooling built	Analyst maintains a 47-tab Excel model updated daily
4	Significant manual process	3 people spend 2 days/week compiling reports
3	Partial automation	Script exists but breaks frequently, needs babysitting
2	Simple workaround	Quick Excel export, takes 30 minutes
1	Haven't tried	Nobody has attempted to solve this yet

4. Data Feasibility

Can we actually build this with available data?

Score	Feasibility	Example
5	Data exists, quality verified	Source audited, passes data-quality-assessment checks
4	Data exists, quality unknown	Source available but no quality baseline established
3	Data exists, known issues	Source has gaps or accuracy problems that need fixing first
2	Data partially exists	Need to combine 3+ sources, some missing
1	Data doesn't exist	Would need new data collection or acquisition

Cross-reference data-quality-assessment for the 5-dimension scoring. A Data Feasibility score of 1 kills the idea regardless of other scores.

5. Schema Risk

How locked-in are consumers once we ship?

Score	Risk	Example
5	Easily re-modeled	Internal API, few consumers, versioning supported
4	Moderate coupling	Dashboard consumed by 5-10 users, change is disruptive but manageable
3	Significant coupling	20+ downstream queries depend on current schema
2	High coupling	External consumers or contractual SLAs on schema shape
1	Breaking change impossible	Regulated output, schema changes require compliance review

Thresholds

Composite Score	Decision	Action
250+	Build	Shape a pitch, allocate a cycle
100-249	Investigate	Run a 1-week experiment to de-risk the lowest-scoring dimension
<100	Kill	Document why and move on. Revisit only if new evidence surfaces

CRITICAL: A Data Feasibility score of 1 kills the idea at any composite score. You can't build a data product without data.

Experiment Types

When the scorecard says "Investigate," pick the cheapest experiment that addresses the weakest dimension:

Sample Query (1 day): Write the SQL. Run it against production data. Does the result answer the consumer's question? Tests Data Feasibility and reveals quality issues fast. If the query returns garbage, kill early.

Manual Pipeline (1 week): Build the transformation by hand for one consumer. Deliver a spreadsheet or flat file. Measure: did they use it? Did it change a decision? Tests Demand Frequency and Decision Impact with real behavior.

Schema Prototype (2-3 weeks): Build a minimal star schema with one fact table and 2-3 dimensions. Load a month of data. Let 2-3 consumers query it. Tests Schema Risk and reveals modeling assumptions before they calcify.

Type 1 vs Type 2 Decisions

Not all data product decisions are equal:

Type 1 (irreversible): Schema design decisions. Data source commitments. Terminology mappings published to external consumers. SCD strategies once historical data accumulates. These deserve full validation and the scorecard process.

Type 2 (reversible): Adding a metric to an existing dashboard. New filters on an existing report. Feature additions to established products. These don't need a scorecard. Just build and measure.

Discovery budget should match decision type. Spend 2 weeks validating a Type 1. Spend 2 hours on a Type 2.