design-backtesting

name	design-backtesting
description	Design the historical backtesting library for the target OS — what historical artifacts to collect during onboarding, how to organize them, and how skills use them for calibration and live reference.

/design-backtesting - Backtesting Library Design

When to Use

• After /design-skills (you need to know which skills produce comparable output)
• When the domain expert confirms users have historical examples of past work
• When designing an OS where output quality is judged against personal standards, not just domain best practices

Inputs

• Required: domain-input/domain-workflow.md — to understand what the user produces
• Required: output/designs/skill-designs.md — to map historical artifacts to skills
• Auto-loaded: domain-input/audiences.md — to understand who evaluates output quality
• Auto-loaded: domain-input/quality-contract.md — to define comparison dimensions
• Auto-loaded: domain-input/identity-model.md — to understand the user's experience level
• Reference: reference/backtesting-anatomy.md — structural template

Process

Step 1: Identify Backtestable Skills

Read the skill designs and identify which skills produce output that the user has historically created manually.

For each skill, ask:

1. Does this skill generate something the user used to do by hand?
2. Would the user have saved examples of their past work?
3. Can the input and output be provided as separate files?

Classify each skill:

• Backtestable: User has historical examples with separable input/output
• Partially backtestable: User has examples but input/output are interleaved (defer full support to v3.0; user can manually separate for now)
• Not backtestable: Skill is a new capability without historical precedent

Step 2: Define Artifact Types

For each backtestable skill, define:

1. What is the input? The blank — the prompt, brief, JD, or request that initiates the work
2. What is the ground truth? The completed result the user actually delivered
3. What is the quality signal? How do you know a historical example represents good work? (outcome-based: it got results; expert-judged: the user considers it their best work)
4. How many cases are realistic? Don't ask for 50 if the user realistically has 5

Step 3: Design the Ordering Scheme

Choose the primary ordering based on the domain:

• Date-ordered when the user's standards evolve over time (recent > old). Common in creative domains where style shifts.
• Importance-ordered when some cases are clearly higher stakes than others. Common in sales, consulting, job search.
• Outcome-ordered when historical cases have measurable success/failure signals. Common in any domain with feedback loops.

Ask the domain expert: "If the OS could only look at 3 of the user's past examples, which 3 should it pick and why?"

Step 4: Define Comparison Dimensions

Design the scoring rubric for backtesting comparisons. Each dimension should be:

• Independent (don't conflate structure with content)
• Observable (the system can actually measure it, not just claim it)
• Weighted (some dimensions matter more for calibration than others)

Standard dimensions (customize per domain):

1. Structural alignment — does the OS output follow the same organization?
2. Content coverage — does it hit the same key points?
3. Tone and style — does it sound like the user?
4. Domain vocabulary — does it use the same technical language?
5. Judgment calls — does it emphasize the same things the user chose to emphasize?

Step 5: Design Live Reference Behavior

Determine whether skills should reference the backtesting library during live generation:

• Which skills benefit from live reference? (usually the same backtestable skills)
• How many cases to read per generation? (recommend 2-3; more adds latency without proportional quality gain)
• What to extract from ground truth? (patterns, not content — the OS should learn style, not copy text)
• How to acknowledge reference? (e.g., "Calibrated against N historical examples")

Step 6: Design Onboarding Flow

Draft the onboarding section for the generated OS's START-HERE.md:

1. When in the setup flow does this appear? (after context-library, before first skill use)
2. What's the minimum viable library? (how many cases before backtesting is useful)
3. What's the time estimate per case? (be honest — if separation takes 10 min per case, say so)
4. What does the user actually need to do? (step by step: create folder, paste input, paste output, update index)

Step 7: Design the Metrics Framework

Define the quantitative metrics the OS will track during backtesting. For each metric type, determine applicability:

1. Speed metric:

   • What is the bottleneck the OS replaces? (triage, drafting, classification, analysis, etc.)
   • Can the domain expert provide manual time estimates per historical case?
   • What's the expected order-of-magnitude improvement? (minutes vs hours, hours vs days)

2. Coverage metric:

   • What are the "units of work" within a case? (questions in a questionnaire, sections in a document, requirements in a spec, etc.)
   • How should units be classified? (green/yellow/red or domain-specific categories)
   • Is coverage growth realistic? (Does the OS's knowledge base actually grow as cases are processed?)

3. Quality metric:

   • Which approach fits: head-to-head scoring (requires domain expert time), consistency checking (automated), or both?
   • What does the 1-5 scale mean in this domain? Write calibrated descriptions for each level.
   • How much domain expert time is needed for scoring? (be honest — even 15 minutes of expert time scoring a sample is valuable)

4. Outcome prediction metric:

   • Does the domain have clear binary outcomes? (won/lost, hired/rejected, published/rejected)
   • Are there enough cases with known outcomes? (minimum 5 for directional signal)
   • If not applicable, explicitly note "skip — no binary outcomes" or "skip — insufficient sample"

Step 8: Design Sequential Backtesting Mode

Determine whether the OS benefits from sequential (compounding) backtesting:

1. Does the OS have a growing knowledge base? If the OS learns from each case processed (importing answers, building a response library, accumulating patterns), sequential mode is valuable. If each case is independent, standard mode is sufficient.

2. Define the import step: After processing each case, what gets imported into the OS's knowledge base? (the ground truth answers, extracted patterns, new vocabulary, etc.)

3. Define the chronological ordering: Cases must be processed in the order they were originally created. Confirm the domain expert can provide dates.

4. Define expected growth curve shape:

   • Cold start (case #1): what coverage is realistic with empty knowledge?
   • Working maturity (case #5-10): what coverage is the target?
   • Diminishing returns: at what point does adding more cases stop improving coverage meaningfully?

5. Define the compounding proof: What specific claim does the growth curve support? (e.g., "response library reaches X% coverage by case #5" or "drafting time drops by Y% after 3 cases")

Step 9: Design the Composite Report Template

Draft the composite report template for this domain:

1. Which metrics are included? (all 4, or a subset)
2. What's the composite summary sentence structure? (this becomes the defensible claim the user can put in a case study, resume, or stakeholder presentation)
3. What distinguishes live cases from backtested cases in reporting?

Output

Write the complete backtesting library design to domain-input/backtesting-library.md.

Tell the designer:

Backtesting library designed.

Backtestable skills: [N] of [total] skills
  - /[skill-1] — [artifact type] ([estimated N] cases available)
  - /[skill-2] — [artifact type] ([estimated N] cases available)

Partially backtestable (v3.0 — auto-separation needed):
  - /[skill-3] — [why separation is hard]

Not backtestable:
  - /[skill-4] — [why: new capability, no historical precedent]

Ordering: [date | importance | outcome]
Comparison dimensions: [N] dimensions defined
Live reference: [enabled for N skills | disabled]
Minimum viable library: [N] cases

Metrics framework:
  - Speed: [applicable — bottleneck is X | not applicable — why]
  - Coverage: [applicable — unit of work is X | not applicable — why]
  - Quality: [head-to-head | consistency | both] — expert time needed: ~N min
  - Outcome prediction: [applicable — N cases with outcomes | skip — why]

Sequential backtesting: [enabled — knowledge compounds via X | disabled — cases are independent]
  - Expected growth curve: case #1 ~X% → case #5 ~Y% → maturity ~Z%

Composite summary template: [drafted | not applicable]

Next steps:
1. Run /generate-os to include the backtesting infrastructure
2. Or run /design-batch-architecture if backtesting should support batch runs

Quality Checks

Good backtesting design:

• Every backtestable skill is identified with a clear artifact type mapping
• Input/output separation is realistic (not asking users to do impossible extraction)
• Ordering scheme matches the domain's actual quality signal (don't use date-ordering in a domain where recency doesn't correlate with quality)
• Comparison dimensions are specific to the domain, not generic
• Minimum viable library is realistic (3-5 cases, not 50)
• Onboarding time estimate is honest
• Live reference behavior is designed, not just formal backtesting
• Metrics framework includes at least speed + one other metric type
• Sequential backtesting is designed when the OS has a growing knowledge base
• Expected growth curve shape is stated with realistic numbers
• Composite summary template is drafted with traceable claims
• Domain expert time commitment for quality scoring is stated honestly

Bad backtesting design:

• Tries to make every skill backtestable (some skills are genuinely new capabilities)
• Asks users to provide 20+ cases before the system is useful
• Comparison dimensions are generic ("quality", "accuracy") without domain-specific definitions
• Ignores the input/output separation challenge (assumes users have clean separate files)
• No live reference — backtesting is isolated from actual generation
• No ordering scheme — treats all historical examples as equally important
• No quantitative metrics — only qualitative "it works" claims
• Claims outcome prediction on a sample of 2-3 cases (too small for any signal)
• Sequential backtesting designed for a domain where cases are genuinely independent (no compounding)
• Composite summary includes numbers that can't be traced back to specific metrics