design-batch-architecture

name	design-batch-architecture
description	Design the parallel/headless worker execution model for processing items at scale — worker prompts, state tracking, and merge pipelines.

/design-batch-architecture - Batch Architecture Design

When to Use

• When the OS needs to process 10+ items in a single operation
• After /design-skills (need to know which skills support batch mode)
• When the designer mentions "scan," "batch," "bulk," "parallel," or "process all"
• Skip if all operations are one-at-a-time

Inputs

• Auto-loaded: output/designs/skill-designs.md (which skills need batch variants)
• Auto-loaded: domain-input/domain-workflow.md (scale operations in the workflow)
• Auto-loaded: domain-input/data-contract.md (ownership rules for batch output)

Process

Step 1: Identify Batch-Eligible Skills

From the skill designs and workflow, list operations that could process multiple items:

Operation	Skill	Typical Volume	Time per Item	Parallelizable?
[e.g., Evaluate offers]	/[skill]	10-50 per batch	3-5 min	Yes
[e.g., Scan portals]	/[skill]	50-200 per run	30s-2min	Partially (Playwright: 1)

Step 2: Design Worker Prompt

The worker prompt (batch/batch-prompt.md) must be completely self-contained:

1. System instructions: Equivalent to reading CLAUDE.md + relevant skill. Everything needed to process one item.
2. Context file references: Worker reads context files independently (identity file, config, etc.)
3. Output format: Exact structure for worker output (report format, tracker line format)
4. Tool constraints: "You are running in batch mode. [Tool X] is unavailable. Use [fallback] instead."
5. Verification rules: How to check output without interactive tools

Ask the designer: "If you had to explain to a colleague how to evaluate one [item] without any prior conversation, what would you tell them?" That's the worker prompt.

Step 3: Design State Tracking

State file schema (batch/batch-state.tsv):

Column	Type	Values	Purpose
id	integer	auto-increment	Unique item identifier
input	string	URL, text, or reference	What to process
status	enum	pending, in-progress, completed, failed, skipped	Current state
output_path	string	path to output file	Where result lives
error	string	error message if failed	Debugging
retries	integer	0-N	How many times retried
timestamp	datetime	ISO 8601	When last updated

Step 4: Design Output Collection & Merge

Each worker produces individual output files. Design the merge:

1. Worker output location: Where does each worker write? (batch/tracker-additions/[id].tsv, [output-folder]/[report].md)
2. Merge script: What consolidates worker outputs into canonical state? (merge-[domain].mjs)
3. Deduplication: How to prevent duplicate entries on merge? (check by [key field])
4. Idempotency: Running merge twice produces same result?

Step 5: Design Safety Controls

• Lock file: batch/batch.lock prevents concurrent batch runs
• Dry-run: --dry-run shows what would be processed without executing
• Concurrency limit: max [N] parallel workers (constrain exclusive-access tools to 1)
• Resume: --start-from N resumes from a specific item
• Retry: --retry-failed retries only failed items

Step 6: Design Batch Input Mechanism

How items enter the batch:

• Manual: user creates batch/batch-input.tsv
• Scanner: /[scan-skill] discovers items and populates batch input
• Pipeline: items flow from data/[pipeline].md to batch input

Output

Write batch architecture to output/designs/batch-designs.md.

Tell the designer:

Batch architecture designed:
- [N] batch-eligible operations
- Worker prompt: self-contained, [N] context file references
- State tracking: [N]-column TSV with [N] status values
- Merge: [script name], dedup by [key], idempotent: [yes/no]
- Safety: lock file, dry-run, max [N] parallel, resume, retry

Key constraint: [e.g., "Playwright limited to 1 concurrent worker"]

Next: Run /design-data-sources if the OS ingests external data.
Or skip to /generate-os.

Quality Checks

Good batch architecture:

• Worker prompt is self-contained (no conversation dependency)
• State file enables resume without re-processing
• Failures are isolated (one bad item doesn't poison the batch)
• Merge is idempotent
• Safety controls are explicit (lock, dry-run, concurrency limit)

Bad batch architecture:

• Workers depend on conversation context
• No state tracking — crash = restart from beginning
• One failure stops the entire batch
• Merge creates duplicates on re-run
• No concurrency limits for exclusive-access tools