automation-distill

name	automation-distill
description	End-to-end methodology for distilling repetitive workflows into reusable automation. Mines conversation history and codebase for repeated patterns, implements project-specific automation, then extracts cross-project universal templates with sanitization and parameterization. Triggers when user mentions identifying repetitive operations, building reusable automation, extracting universal templates, finding what's worth automating, distilling workflows, asks "what can be automated" across a project, or wants to review/retire existing automation.
user-invocable	true
allowed-tools	Read, Write, Edit, Bash, Glob, Grep

Automation Distill

Three-stage methodology: Mine → Implement → Universalize.

Each stage produces a deliverable that feeds the next. Stop at any stage if scope doesn't justify deeper investment.

Key term: workflow cycle

A workflow cycle = one complete end-to-end iteration of the project's main workflow (e.g., train one model + evaluate + report = one cycle; process one batch + ship = one cycle). Each project defines its own cycle boundary.

This methodology uses cycle counts (not calendar time) as the time anchor. Rationale: project tempos vary by orders of magnitude — cycles normalize value automatically.

When to invoke

• Project has gone through ≥ 3 cycles with similar operations recurring
• User notices "I keep doing X"
• New project starts and wants to inherit prior automation
• Periodic review (every few cycles): audit existing automation for decay / removal candidates

When NOT to automate (strategic veto)

Signal	Why skip
Each instance differs substantially	Automation constrains creativity, premature lock-in
Repetition IS the thinking time	Automating removes reflection that informs decisions
Cumulative < 5 instances AND single-instance cost < 2min	Build cost exceeds lifetime savings
One-off project / no reuse window	No amortization possible
Output requires irreplaceable human judgment	Automation degrades quality
Repetition is symptom of bad architecture	Fix root cause instead

Quantitative thresholds

Frequency expressed as repetition signals, not calendar rates:

Tier	Repetition signal	Action
P0	≥ 3 instances per workflow cycle, OR cumulative ≥ 20 in transcripts, OR appears in ≥ 5 distinct sessions	Build now
P1	≥ 1 instance per workflow cycle, OR cumulative ≥ 10, OR appears in ≥ 3 distinct sessions	Next cycle
P2	Cross-cycle sporadic, cumulative ≥ 5	If time permits
Skip	Cumulative < 3, OR one-off, OR cost < 2min/instance	Stay manual

Counting rule: actual invocations in transcripts, never estimates. Cost rule: median wall-clock from start to validated result.

Decision tree

Observed: repeated operation
  ├─ Strategic veto applies?  → STOP, document why
  ├─ Tier = Skip?              → STOP, leave manual
  ├─ Synchronous + < 1s + event-triggered    → Hook
  ├─ Multi-step + explicit invocation        → Subagent
  ├─ Cross-project methodology + reusable    → Skill
  ├─ User correction or persistent rule      → Memory file
  └─ Project state needed at session start   → SessionStart hook

Stage 1: Mine — Pattern identification

Input

• Conversation transcripts / session history (all available)
• Project artifacts (scripts, docs, configs)
• User pain points

Process

1. Scan transcripts for operations repeated ≥ 3 times
2. Abstract action → pattern (the what, not the how)
3. Group into universal categories (see catalog)
4. Count cumulative occurrences, distinct sessions, per-cycle frequency → assign tier

Universal pattern catalog

Pattern	Essence	Mechanism
State inspection	"Where are we now" — long-running tasks, cross-session	SessionStart hook + monitor subagent
Template derivation	Parameterized clone of prior work	Skill + sed/template
Consistency sync	One change ripples to N locations	PostToolUse hook
Invariant check	Structures default to wrong; verify explicitly	Pre/PostToolUse hook + assertion lib
Decision support	Data + presentation, decision stays with user	Subagent only
Root cause analysis	Anomaly → hypothesis → fix → invariant	Subagent + assertion library
Cross-version comparability	Single-variable design front-loaded	Skill + design checklist
Data lineage	Multi-version derivation trace	Skill + provenance graph
Embedded knowledge	User corrections become persistent rules	Memory + reflexive hook

Validation

• Each pattern entry cites ≥ 3 concrete transcript instances
• Tier assignment justified by counted frequency, not estimate
• Patterns with no clear mechanism mapping → flag, don't force-fit

Deliverable

analysis/repetitive_patterns.md: pattern catalog + frequencies + ROI tier per entry.

Stage 2: Implement — Project-specific automation

Input

Stage 1 catalog, sorted by tier.

Process

1. Map each P0/P1 pattern to mechanism per decision tree
2. Implement P0 first
3. Hardcode project-specific paths/conventions for now
4. Test each artifact in isolation before integrating

Deliverables

• .claude/hooks/ — event-driven scripts
• .claude/agents/ — subagent definitions
• .claude/skills/ — project-specific skills
• .claude/settings.json — hook registration (edit minimally, validate with jq -e)
• Memory files for user corrections

Validation (per artifact)

Artifact	Smoke test
Hook	Add sentinel echo "$(date) fired" >> .claude/logs/hook-trace.log line; trigger matching tool; tail log
Subagent	Invoke with synthetic input matching frontmatter; verify output schema
Skill	Open fresh session, mention trigger keyword; observe skill listing surfaces it
Memory	Reference in subsequent prompt; verify behavior reflects rule

Failure modes:

• Hook silent fail: jq -e '.hooks' .claude/settings.json must exit 0; check stderr in .claude/logs/
• Skill not triggering: description likely missing user's phrasing → re-read transcript, expand trigger keywords
• Subagent timeout: prompt too open-ended → constrain output schema + word limit

Verification before Stage 3

• Survived ≥ 3 full workflow cycles in real use
• Sentinel logs show actual invocation count within ±2× of estimate
• No silent regressions on existing workflow

Stage 3: Universalize — Cross-project extraction

Input

Stage 2 artifacts that survived ≥ 3 workflow cycles or ≥ 2 distinct subtask types.

Process

1. Universality score per artifact:

   • Count project-specific terms (paths, names, jargon)
   • density > 60% → drop; 20-60% → parameterize; < 20% → keep verbatim

2. Sanitize:

   • Strip hardcoded paths
   • Replace project/user/dataset names with placeholders
   • Scan for accidental PII: usernames, internal hostnames, secrets
   • Safety net: grep -rE '(/home/|/Users/|@.*\.com|192\.168|api[_-]?key)' <artifact>

3. Parameterize:

   • Hooks: env vars for paths/regexes
   • Skills: prompt args for context
   • Subagents: input fields documented in frontmatter

4. Abstract:

   • Replace concrete steps with workflow phases
   • Describe what the phase achieves, not how

5. Rename:

   • Verb-noun universal naming
   • Avoid domain coupling

6. Boundary clarification:

   • Each artifact's input/output explicitly defined
   • Reference other artifacts by name, not path

Validation

• Install templates in a second project (real or scratch)
• Run smoke tests from Stage 2 in new context
• Any failure → parameterization incomplete, iterate

Deliverable

automation_universal/ with sanitized templates + README documenting install/smoke-test paths.

Skip Stage 3 if

• Artifacts haven't survived ≥ 3 cycles
• Domain value too high to abstract cleanly
• No plan to apply to other projects

Conflict handling

Conflict	Resolution
Multiple hooks on same event	Merge into one orchestrator script; or order explicitly in settings.json array
Multiple skills with overlapping triggers	Tighten descriptions to disambiguate scenarios; or merge into one with branching
Project-level vs user-level skill collision	Project wins by default; user override always permitted
Hook fires too often	Add matcher narrowing or input guard; never disable silently

Observability

• Each hook writes one-line sentinel: <unix-ts>,<event>,<matched_file> → .claude/logs/hook-trace.log
• Each subagent invocation logged via the harness (no extra work needed)
• Skill triggering tracked by reviewing transcripts every few cycles
• Compare actual invocation count to Stage 1 estimate; > 2× off → re-tier

Lifecycle management

Review after every N (≥ 3) workflow cycles or when project rhythm shifts:

Signal	Action
Hook trigger count = 0 across last 3 cycles	Candidate for retirement
Subagent output never referenced in subsequent turns	Output format wrong or skill obsolete
Skill triggers but Claude bypasses guidance	Description/body misaligned, rewrite or remove
Project structure changed, hook regex no longer matches	Update regex or retire
Actual count > 2× estimate over multiple cycles	Promotion candidate to universal pack
Actual count < 0.5× estimate	Tier overstated, downgrade or retire
Multiple new patterns observed	Re-run Stage 1, treat as incremental update

Retirement procedure: mv to .claude/_archive/<cycle-tag>_<name>/, remove from settings.json, update README.

Anti-patterns

Strategic

• Automating thinking time — If repetition is where you reason, automating it removes the reasoning
• Building before validating ROI — Skipping tier assignment leads to low-value automation hoarding
• Treating one-off as recurring — Wait for ≥ 3 real instances before mechanism design
• Solving symptom not cause — Repeated cleanup of bad output? Fix the generator, not the cleanup

Implementation

• Hook for long task — Blocks all subsequent operations; use subagent instead
• Subagent replacing decision — Removes user agency; subagent prepares data only
• Settings file overhaul — One syntax error silently disables all hooks; edit incrementally, validate with jq -e
• Skill description too narrow — Misses synonyms; include user's actual phrasings from transcript
• Project paths in universal template — Defeats reuse; parameterize or drop

Cross-project

• Direct copy-paste install — Overwrites target's settings.json; use jq merge
• No smoke test after install — Silent failure surfaces only at point of need
• Maintaining EN/ZH versions independently — Drift inevitable; pick one as canonical, translate on change

Team / collaboration

File	Strategy
.claude/hooks/, .claude/agents/, .claude/skills/	Commit to repo if team-shared
.claude/settings.json	Commit team-wide hooks; personal hooks → settings.local.json
.claude/settings.local.json	gitignore
Memory files	Personal, never commit

For multi-developer teams: factor skills into shared repo, install via submodule or symlink.

Reuse checklist (from 0 to 1)

Bringing this methodology to a new project:

• Identify ≥ 3 instances of any repeated operation
• Assign tier; skip if Skip-tier
• Pick mechanism via decision tree
• Implement P0 only (one artifact); skip everything else first round
• Smoke test; commit
• Use for ≥ 1 workflow cycle before adding P1
• After ≥ 3 cycles: retire any unused, promote any over-triggered to universal

End-to-end example

User: "I keep manually checking training progress and updating reports"

Stage 1 (Mine):
- Pattern: state inspection, ~3 instances per training cycle, cumulative 60+ → P0
- Pattern: consistency sync, ~1 per cycle, cumulative 15+ → P1
- Strategic veto check: not thinking time, not creative; OK to automate

Stage 2 (Implement):
- Hook: SessionStart loader injects training state
- Subagent: task-monitor reports detailed progress on demand
- Hook: PostToolUse warns when derived report lacks index entry
- Smoke test: sentinel logs confirm all 3 fire correctly

Stage 3 (Universalize, after surviving 3+ training cycles):
- state-snapshot.sh with env-var configurability
- task-monitor.md agent with prompt-arg interface
- index-sync.sh with regex configurability
- Smoke test: install in second project, verify

Companion artifacts

data-lineage and derive-experiment are concrete instances produced by this methodology; reference them as templates when implementing Stage 2 deliverables.

Examples

Example 1 — Mining triggered by user complaint Input: "Every time I derive a new version I forget to update the lineage file" Output:

• Stage 1: Pattern H (data lineage), tier P1 (~1 per version-derivation cycle, ~10min/instance)
• Stage 2: install derive-experiment skill that auto-writes lineage in step 5
• Stage 3 (later): extract data-lineage schema as universal template

Example 2 — Cross-project transfer Input: "Starting a new ML project. What automation should I bring?" Output:

• Skip Stage 1 (analysis already done)
• Stage 2: install universal templates, parameterize via env vars
• Run smoke test before relying on them

Example 3 — Premature distillation refused Input: "I just wrote a script today. Make it a universal skill." Output:

• Refuse Stage 3 (no real-use evidence, < 3 cycles)
• Suggest Stage 2 (project skill); re-evaluate after ≥ 3 cycles

Example 4 — Doc-heavy project (non-ML) Input: "Docs team keeps forgetting to update the changelog when shipping features" Output:

• Stage 1: consistency sync, tier P0 (≥ 3 per release cycle)
• Stage 2: PostToolUse hook on CHANGELOG-adjacent files, warn if version section missing
• Stage 3: parameterize as index-sync.sh, already in universal pack

Example 5 — Strategic veto wins Input: "Each PR description is repetitive, let's templatize it" Output:

• Strategic veto: PR description IS the reflection on what changed; templatizing loses signal
• Recommend manual; offer subagent that drafts a starting point user can edit

Example 6 — Cycle-based review triggers retirement Input: "Audit our automation, what's still earning its keep?" Output:

• Pull sentinel logs from .claude/logs/hook-trace.log
• 3 hooks fired 0 times across last 3 cycles → retire to _archive/
• 1 skill triggered but Claude bypassed → rewrite description or remove
• 1 hook fired 5× expected → consider promoting to universal pack