nemo-curator

name	nemo-curator
description	GPU-accelerated data curation for LLM training. Supports text/image/video/audio. Features fuzzy deduplication (16× faster), quality filtering (30+ heuristics), semantic deduplication, PII redaction, NSFW detection. Scales across GPUs with RAPIDS. Use for preparing high-quality training datasets, cleaning web data, or deduplicating large corpora.
version	1.0.0
author	Orchestra Research
license	MIT
dependencies	["nemo-curator","cudf","dask","rapids"]
platforms	["linux","macos"]
metadata	{"hermes":{"tags":["Data Processing","NeMo Curator","Data Curation","GPU Acceleration","Deduplication","Quality Filtering","NVIDIA","RAPIDS","PII Redaction","Multimodal","LLM Training Data"]}}

NeMo Curator - GPU-Accelerated Data Curation

NVIDIA's toolkit for preparing high-quality training data for LLMs.

When to use NeMo Curator

Use NeMo Curator when:

• Preparing LLM training data from web scrapes (Common Crawl)
• Need fast deduplication (16× faster than CPU)
• Curating multi-modal datasets (text, images, video, audio)
• Filtering low-quality or toxic content
• Scaling data processing across GPU cluster

Performance:

• 16× faster fuzzy deduplication (8TB RedPajama v2)
• 40% lower TCO vs CPU alternatives
• Near-linear scaling across GPU nodes

Use alternatives instead:

• datatrove: CPU-based, open-source data processing
• dolma: Allen AI's data toolkit
• Ray Data: General ML data processing (no curation focus)

Quick start

Installation

# Text curation (CUDA 12)
uv pip install "nemo-curator[text_cuda12]"

# All modalities
uv pip install "nemo-curator[all_cuda12]"

# CPU-only (slower)
uv pip install "nemo-curator[cpu]"

Basic text curation pipeline

from nemo_curator import ScoreFilter, Modify
from nemo_curator.datasets import DocumentDataset
import pandas as pd

# Load data
df = pd.DataFrame({"text": ["Good document", "Bad doc", "Excellent text"]})
dataset = DocumentDataset(df)

# Quality filtering
def quality_score(doc):
    return len(doc["text"].split()) >   

filtered = ScoreFilter(quality_score)(dataset)


 nemo_curator.modules  ExactDuplicates
deduped = ExactDuplicates()(filtered)


deduped.to_parquet()