data-juicer-agents

Data-Juicer execution and file management: shell command execution, Python code execution, file read/write/insert operations. Trigger keywords: execute_shell, execute_python, view_file, write_file, insert_file, shell command, run Python, read file, write file. Use when you need to execute system commands, run Python code snippets, or manipulate text files. Note: Data processing should use Data-Juicer operators, not Python scripts. Related skills: data-juicer (main flow), djx_apply (execution).

Full data processing harness based on Data-Juicer for cleaning, filtering, deduplicating, and transforming JSONL/JSON datasets. Trigger keywords: data cleaning, data filtering, deduplication, dedup, dataset, dataset processing, JSONL processing, text cleaning, HTML cleaning, low-quality filtering, data preprocessing, ETL, data pipeline. Error triggers: dj-process failure, operator not found, plan validation failure, recipe execution timeout, OOM, permission denied, apply_recipe error. Related skills: djx_install (installation), djx_auth (authentication), djx_local_model (private data), djx_context (dataset inspection), djx_retrieve (operator search), djx_plan (plan building), djx_apply (execution), djx_dev (custom operators). Note: Simple statistics (wc -l, head) do not require this skill — use shell commands directly.

Data-Juicer Agents installation and setup guide: environment setup, dependency installation, version verification, common issues. Trigger keywords: install, setup, environment configuration, pip install, uv install, djx command not found, ModuleNotFoundError, version issues, Python version. Use for first-time installation, environment problems, installation failures, or version incompatibilities. Related skills: djx_auth (authentication), data-juicer (main flow).

Data-Juicer local model processing: Ollama configuration, private data handling, offline workflows. Trigger keywords: local model, Ollama, private data, sensitive data, offline, do not send to cloud, local LLM. Use when processing sensitive/private data, running offline, or configuring local models. Related skills: djx_auth (authentication), data-juicer (main flow), djx_retrieve (local retrieval).

Data-Juicer operator retrieval reference: retrieve_operators (local), retrieve_operators_api (API), get_operator_info, list_operator_catalog. Trigger keywords: retrieve_operators, retrieve_operators_api, search operators, find operator, which operator, intent, retrieval, bm25, regex, LLM mode, operator info, operator catalog. Use when searching for suitable operators, unsure which operator to use, or retrieve errors occur. Related skills: data-juicer (main flow), djx_auth (authentication), djx_local_model (local mode).

Data-Juicer execution reference: recipe execution, timeout configuration, verification, retry strategy, error handling. Trigger keywords: apply_recipe, execute, run, timeout, dry_run, dry-run, recipe failure, execution error, dj-process. Use when apply_recipe errors out, execution times out, timeout adjustment is needed, or dry_run verification is required. Related skills: data-juicer (main flow), djx_plan (plan building).

Data-Juicer dataset inspection: inspect_dataset usage, field analysis, modality detection, statistics. Trigger keywords: inspect_dataset, inspect dataset, dataset structure, fields, modality, statistics, sample, profile, data overview. Use when you need to understand dataset structure, view fields, detect modality, or get statistics. Related skills: data-juicer (main flow), djx_plan (plan building).

Data-Juicer plan building reference: detailed schema and parameter documentation for build_dataset_spec, build_process_spec, build_system_spec, assemble_plan, and plan_save. Trigger keywords: build_spec, assemble_plan, plan_save, build plan, YAML, specification, dataset spec, process spec, system spec. Use when building plans and needing to check detailed parameters, or when spec building fails. Related skills: data-juicer (main flow), djx_retrieve (operator retrieval), djx_apply (execution).

Use the Data-Juicer Agents harness for dataset inspection, cleaning, filtering, deduplication, transformation, and recipe-oriented data processing through the `djx tool` interface. Trigger when the user explicitly mentions Data-Juicer, data-juicer-agents, data_juicer_agents, dj-agents, or djx, or when the user asks for data processing work that should be handled through the Data-Juicer Agents tool-based CLI harness.

Data-Juicer authentication and configuration: API Key setup, model selection, environment variables, cloud/local backend switching. Trigger keywords: authentication, auth, API Key, DASHSCOPE, MODELSCOPE, environment variables, 401 Unauthorized, token, API key configuration, model configuration. Use when configuring API Keys, switching models, configuring backends, or encountering authentication errors. Related skills: djx_install (installation), djx_local_model (local models), data-juicer (main flow).

Data-Juicer custom operator development: scaffold generation, testing, pipeline integration. Trigger keywords: develop_operator, custom operator, create operator, new operator, scaffold, extension. Use when existing operators don't meet requirements and custom processing logic is needed. Related skills: data-juicer (main flow), djx_plan (integrating custom operators).