| name | design-batch-architecture |
| description | Design the parallel/headless worker execution model for processing items at scale — worker prompts, state tracking, and merge pipelines. |
/design-batch-architecture - Batch Architecture Design
When to Use
- When the OS needs to process 10+ items in a single operation
- After
/design-skills (need to know which skills support batch mode)
- When the designer mentions "scan," "batch," "bulk," "parallel," or "process all"
- Skip if all operations are one-at-a-time
Inputs
- Auto-loaded:
output/designs/skill-designs.md (which skills need batch variants)
- Auto-loaded:
domain-input/domain-workflow.md (scale operations in the workflow)
- Auto-loaded:
domain-input/data-contract.md (ownership rules for batch output)
Process
Step 1: Identify Batch-Eligible Skills
From the skill designs and workflow, list operations that could process multiple items:
| Operation | Skill | Typical Volume | Time per Item | Parallelizable? |
|---|
| [e.g., Evaluate offers] | /[skill] | 10-50 per batch | 3-5 min | Yes |
| [e.g., Scan portals] | /[skill] | 50-200 per run | 30s-2min | Partially (Playwright: 1) |
Step 2: Design Worker Prompt
The worker prompt (batch/batch-prompt.md) must be completely self-contained:
- System instructions: Equivalent to reading CLAUDE.md + relevant skill. Everything needed to process one item.
- Context file references: Worker reads context files independently (identity file, config, etc.)
- Output format: Exact structure for worker output (report format, tracker line format)
- Tool constraints: "You are running in batch mode. [Tool X] is unavailable. Use [fallback] instead."
- Verification rules: How to check output without interactive tools
Ask the designer: "If you had to explain to a colleague how to evaluate one [item] without any prior conversation, what would you tell them?" That's the worker prompt.
Step 3: Design State Tracking
State file schema (batch/batch-state.tsv):
| Column | Type | Values | Purpose |
|---|
| id | integer | auto-increment | Unique item identifier |
| input | string | URL, text, or reference | What to process |
| status | enum | pending, in-progress, completed, failed, skipped | Current state |
| output_path | string | path to output file | Where result lives |
| error | string | error message if failed | Debugging |
| retries | integer | 0-N | How many times retried |
| timestamp | datetime | ISO 8601 | When last updated |
Step 4: Design Output Collection & Merge
Each worker produces individual output files. Design the merge:
- Worker output location: Where does each worker write? (batch/tracker-additions/[id].tsv, [output-folder]/[report].md)
- Merge script: What consolidates worker outputs into canonical state? (merge-[domain].mjs)
- Deduplication: How to prevent duplicate entries on merge? (check by [key field])
- Idempotency: Running merge twice produces same result?
Step 5: Design Safety Controls
- Lock file: batch/batch.lock prevents concurrent batch runs
- Dry-run: --dry-run shows what would be processed without executing
- Concurrency limit: max [N] parallel workers (constrain exclusive-access tools to 1)
- Resume: --start-from N resumes from a specific item
- Retry: --retry-failed retries only failed items
Step 6: Design Batch Input Mechanism
How items enter the batch:
- Manual: user creates batch/batch-input.tsv
- Scanner: /[scan-skill] discovers items and populates batch input
- Pipeline: items flow from data/[pipeline].md to batch input
Output
Write batch architecture to output/designs/batch-designs.md.
Tell the designer:
Batch architecture designed:
- [N] batch-eligible operations
- Worker prompt: self-contained, [N] context file references
- State tracking: [N]-column TSV with [N] status values
- Merge: [script name], dedup by [key], idempotent: [yes/no]
- Safety: lock file, dry-run, max [N] parallel, resume, retry
Key constraint: [e.g., "Playwright limited to 1 concurrent worker"]
Next: Run /design-data-sources if the OS ingests external data.
Or skip to /generate-os.
Quality Checks
Good batch architecture:
- Worker prompt is self-contained (no conversation dependency)
- State file enables resume without re-processing
- Failures are isolated (one bad item doesn't poison the batch)
- Merge is idempotent
- Safety controls are explicit (lock, dry-run, concurrency limit)
Bad batch architecture:
- Workers depend on conversation context
- No state tracking — crash = restart from beginning
- One failure stops the entire batch
- Merge creates duplicates on re-run
- No concurrency limits for exclusive-access tools
