data-juicer

name	data-juicer
description	多模态数据清洗与去重 - 提供200+算子，支持云原生分布式处理
category	pipeline
emoji	🔄
github_url	https://github.com/datajuicer/data-juicer
triggers	["数据清洗","数据去重","多模态处理","data juicer","分布式"]
inputs	["json","jsonl","parquet"]
outputs	["json","jsonl","parquet"]
use_cases	["数据清洗","数据去重","多模态数据处理"]

Data-Juicer

多模态数据清洗与去重大规模流水线，提供200+算子。

安装

pip install data-juicer

使用

dj-process --config config.yaml