djx-apply

name	djx_apply
description	Data-Juicer execution reference: recipe execution, timeout configuration, verification, retry strategy, error handling. Trigger keywords: apply_recipe, execute, run, timeout, dry_run, dry-run, recipe failure, execution error, dj-process. Use when apply_recipe errors out, execution times out, timeout adjustment is needed, or dry_run verification is required. Related skills: data-juicer (main flow), djx_plan (plan building).
allowed-tools	Bash, Read
argument-hint	<plan_path>
user-invocable	true

Data-Juicer Skills: Apply (Execution Reference)

Main flow: See the data-juicer skill. This skill provides detailed reference for apply_recipe execution.

---

Prerequisites

Condition	Requirement	Verification Command
Plan file	YAML format, exists and is valid	cat plan.yaml
Environment	py-data-juicer>=1.4.0 installed	dj-process --version
Input data	Dataset path is correct and accessible	head -n 1 <dataset_path>

---

Core Concepts

Concept	Description
plan_path	YAML plan file path, generated by plan_save
dry_run	Validation mode — does not actually execute; checks plan validity
timeout	Execution timeout in seconds; terminates after timeout
confirm	Execution confirmation — must be true for actual execution

---

apply_recipe Schema

Parameter	Type	Required	Default	Description
plan_path	str	Yes	—	YAML plan file path
dry_run	bool	No	false	Validate only, do not execute
timeout	int	No	300	Execution timeout (seconds)
confirm	bool	No	false	Must be true to execute

Key: --yes (CLI flag) and "confirm": true (JSON field) are both required. --yes skips the interactive prompt; confirm provides logical confirmation.

---

Command Format

# Dry-run validation (recommended to run first)
djx tool run apply_recipe --yes --input-json '{"plan_path": "<plan.yaml>", "dry_run": true}'

# Actual execution
djx tool run apply_recipe --yes --input-json '{"plan_path": "<plan.yaml>", "confirm": true, "timeout": 600}'

---

Timeout Configuration Guide

Scenario	Recommended Timeout	Notes
Small dataset, simple operators	300s	Default value
Medium dataset (10K-100K)	600s	Common scenario
Large dataset (100K+)	1800s	Requires longer processing time
LLM-based operators	3600s+	Language detection, quality scoring, etc.
Image/video processing	3600s+	Multimodal processing is slower

---

Retry Strategy

Attempt	Action
1st failure	Read the error message carefully; check exit code and stderr; fix input and retry
2nd failure	Stop. Report the error. Do not continue retrying.

Maximum 2 apply_recipe attempts. Infinite retries waste budget and are futile.

---

Error Handling

Scenario	Error Symptom	Solution
confirmation_required (exit 3)	Command not executed	Add --yes flag
confirm=false no response	No output	Set "confirm": true in JSON
Timeout	timeout exceeded	Increase timeout value
dj-process not found	Command not found	uv pip install py-data-juicer
OOM	Process killed	Reduce parallelism in system_spec
Plan file doesn't exist	file not found	Check plan_path
Plan format error	invalid YAML	Check plan file syntax
Operator doesn't exist	operator not found	Verify operator name comes from retrieve output

---

Must-Read Pitfalls

1. Both --yes and confirm:true Are Required

Wrong: Passing only one

# Missing --yes
djx tool run apply_recipe --input-json '{"plan_path": "plan.yaml", "confirm": true}'
# → confirmation_required (exit 3)

# Missing confirm
djx tool run apply_recipe --yes --input-json '{"plan_path": "plan.yaml"}'
# → No response (confirm defaults to false)

Correct: Pass both

djx tool run apply_recipe --yes --input-json '{"plan_path": "plan.yaml", "confirm": true, "timeout": 600}'

2. Increase Timeout for Large Datasets

Default 300s is insufficient for large datasets:

# 100K+ records
djx tool run apply_recipe --yes --input-json '{"plan_path": "plan.yaml", "confirm": true, "timeout": 1800}'

3. Dry-run Before Execution

# Step 1: Validate
djx tool run apply_recipe --yes --input-json '{"plan_path": "plan.yaml", "dry_run": true}'

# Step 2: Execute after confirmation
djx tool run apply_recipe --yes --input-json '{"plan_path": "plan.yaml", "confirm": true, "timeout": 600}'

---

Typical Workflow

# 1. Validate plan
djx tool run apply_recipe --yes --input-json '{"plan_path": ".djx/plans/my_plan.yaml", "dry_run": true}'

# 2. Execute
djx tool run apply_recipe --yes --input-json '{
  "plan_path": ".djx/plans/my_plan.yaml",
  "confirm": true,
  "timeout": 600
}'

# 3. Verify output
djx tool run inspect_dataset --input-json '{"dataset_source": {"path": "<output_path>"}, "sample_size": 10}'

---

Skill Responsibilities

Scenario	Skill to Use
Main flow (Steps 1-9)	data-juicer
Build plan/spec	djx_plan
Execution and error handling	djx_apply (this skill)
Search for operators	djx_retrieve