nemo-curator

name	nemo-curator
description	GPU加速数据清洗 - RAPIDS加速的PB级文本图像视频音频处理
category	pipeline
emoji	⚡
github_url	https://github.com/NVIDIA/NeMo-Curator
triggers	["GPU加速","模糊去重","质量过滤","nemo curator","RAPIDS"]
inputs	["json","parquet"]
outputs	["json","parquet"]
use_cases	["GPU加速去重","大规模数据过滤"]

NeMo Curator

NVIDIA开发的大规模数据清洗工具，集成GPU加速技术。

安装

pip install nemo-curator