Skip to main content
nemo-curator GPU-accelerated data curation for LLM training. Supports text/image/video/audio. Features fuzzy deduplication (16× faster), quality filtering (30+ heuristics), semantic deduplication, PII redaction, NSFW detection. Scales across GPUs with RAPIDS. Use for preparing high-quality training datasets, cleaning web data, or deduplicating large corpora.
Aller à l'installation Skills Marketplace Découvrez et explorez les compétences IA créées par la communauté.
Installer avec Codex ou Claude Copiez ce prompt, collez-le dans Codex, Claude ou un autre assistant, puis laissez-le vérifier la page du skill et l'installer pour vous.
Afficher les détails du prompt Une commande directe contourne le prompt de vérification. Examinez la source avant de l'exécuter.
npx skills add https://github.com/davila7/claude-code-templates --skill nemo-curatorLa commande reste sur une seule ligne. Faites défiler horizontalement pour la vérifier avant de la copier.
Vous préférez une copie locale ? Téléchargez les fichiers actuellement disponibles dans SkillsMP.
Télécharger Zip Téléchargement... Plus depuis ce dépôt Build AI agents that interact with computers like humans do - viewing screens, moving cursors, clicking buttons, and typing text. Covers Anthropic's Computer Use, OpenAI's Operator/CUA, and open-source alternatives. Critical focus on sandboxing, security, and handling the unique challenges of vision-based control. Use when: computer use, desktop automation agent, screen control AI, vision-based agent, GUI automation.
Expert in building products that wrap AI APIs (OpenAI, Anthropic, etc.) into focused tools people will pay for. Not just 'ChatGPT but different' - products that solve specific problems with AI. Covers prompt engineering for products, cost management, rate limiting, and building defensible AI businesses. Use when: AI wrapper, GPT product, AI tool, wrap AI, AI SaaS.
github-workflow-automation Automate GitHub workflows with AI assistance. Includes PR reviews, issue triage, CI/CD integration, and Git operations. Use when automating GitHub workflows, setting up PR review automation, creating GitHub Actions, or triaging issues.
Explorateur de fichiers
3 fichiers Métiers associés SOC
Basé sur la classification professionnelle SOC
name nemo-curator description GPU-accelerated data curation for LLM training. Supports text/image/video/audio. Features fuzzy deduplication (16× faster), quality filtering (30+ heuristics), semantic deduplication, PII redaction, NSFW detection. Scales across GPUs with RAPIDS. Use for preparing high-quality training datasets, cleaning web data, or deduplicating large corpora. version 1.0.0 author Orchestra Research license MIT tags ["Data Processing","NeMo Curator","Data Curation","GPU Acceleration","Deduplication","Quality Filtering","NVIDIA","RAPIDS","PII Redaction","Multimodal","LLM Training Data"] dependencies ["nemo-curator","cudf","dask","rapids"]
NeMo Curator - GPU-Accelerated Data Curation
NVIDIA's toolkit for preparing high-quality training data for LLMs.
When to use NeMo Curator
Preparing LLM training data from web scrapes (Common Crawl)
Need fast deduplication (16× faster than CPU)
Curating multi-modal datasets (text, images, video, audio)
Filtering low-quality or toxic content
Scaling data processing across GPU cluster
16× faster fuzzy deduplication (8TB RedPajama v2)40% lower TCO vs CPU alternativesNear-linear scaling across GPU nodes Use alternatives instead :
datatrove : CPU-based, open-source data processingdolma : Allen AI's data toolkitRay Data : General ML data processing (no curation focus)
Quick start
Installation
uv pip install "nemo-curator[text_cuda12]"
uv pip install "nemo-curator[all_cuda12]"
uv pip install "nemo-curator[cpu]"
Basic text curation pipeline from nemo_curator import ScoreFilter, Modify
from nemo_curator.datasets import DocumentDataset
import pandas as pd
df = pd.DataFrame({"text" : ["Good document" , "Bad doc" , "Excellent text" ]})
dataset = DocumentDataset(df)
def quality_score (doc ):
return len (doc["text" ].split()) > 5
filtered = ScoreFilter(quality_score)(dataset)
from nemo_curator.modules import ExactDuplicates
deduped = ExactDuplicates()(filtered)
deduped.to_parquet("curated_data/" )
Data curation pipeline
Stage 1: Quality filtering from nemo_curator.filters import (
WordCountFilter,
RepeatedLinesFilter,
UrlRatioFilter,
NonAlphaNumericFilter
)
from nemo_curator import ScoreFilter
dataset = dataset.filter (WordCountFilter(min_words=50 , max_words=100000 ))
dataset = dataset.filter (RepeatedLinesFilter(max_repeated_line_fraction=0.3 ))
dataset = dataset.filter (UrlRatioFilter(max_url_ratio=0.2 ))
Stage 2: Deduplication from nemo_curator.modules import ExactDuplicates
deduped = ExactDuplicates(id_field="id" , text_field="text" )(dataset)
Fuzzy deduplication (16× faster on GPU):
from nemo_curator.modules import FuzzyDuplicates
fuzzy_dedup = FuzzyDuplicates(
id_field="id" ,
text_field="text" ,
num_hashes=260 ,
num_buckets=20 ,
hash_method="md5"
)
deduped = fuzzy_dedup(dataset)
from nemo_curator.modules import SemanticDuplicates
semantic_dedup = SemanticDuplicates(
id_field="id" ,
text_field="text" ,
embedding_model="sentence-transformers/all-MiniLM-L6-v2" ,
threshold=0.8
)
deduped = semantic_dedup(dataset)
Stage 3: PII redaction from nemo_curator.modules import Modify
from nemo_curator.modifiers import PIIRedactor
pii_redactor = PIIRedactor(
supported_entities=["EMAIL_ADDRESS" , "PHONE_NUMBER" , "PERSON" , "LOCATION" ],
anonymize_action="replace"
)
redacted = Modify(pii_redactor)(dataset)
Stage 4: Classifier filtering from nemo_curator.classifiers import QualityClassifier
quality_clf = QualityClassifier(
model_path="nvidia/quality-classifier-deberta" ,
batch_size=256 ,
device="cuda"
)
high_quality = dataset.filter (lambda doc: quality_clf(doc["text" ]) > 0.5 )
GPU acceleration
GPU vs CPU performance Operation CPU (16 cores) GPU (A100) Speedup Fuzzy dedup (8TB) 120 hours 7.5 hours 16× Exact dedup (1TB) 8 hours 0.5 hours 16× Quality filtering 2 hours 0.2 hours 10×
Multi-GPU scaling from nemo_curator import get_client
import dask_cuda
client = get_client(cluster_type="gpu" , n_workers=8 )
deduped = FuzzyDuplicates(...)(dataset)
Multi-modal curation
Image curation from nemo_curator.image import (
AestheticFilter,
NSFWFilter,
CLIPEmbedder
)
aesthetic_filter = AestheticFilter(threshold=5.0 )
filtered_images = aesthetic_filter(image_dataset)
nsfw_filter = NSFWFilter(threshold=0.9 )
safe_images = nsfw_filter(filtered_images)
clip_embedder = CLIPEmbedder(model="openai/clip-vit-base-patch32" )
image_embeddings = clip_embedder(safe_images)
Video curation from nemo_curator.video import (
SceneDetector,
ClipExtractor,
InternVideo2Embedder
)
scene_detector = SceneDetector(threshold=27.0 )
scenes = scene_detector(video_dataset)
clip_extractor = ClipExtractor(min_duration=2.0 , max_duration=10.0 )
clips = clip_extractor(scenes)
video_embedder = InternVideo2Embedder()
video_embeddings = video_embedder(clips)
Audio curation from nemo_curator.audio import (
ASRInference,
WERFilter,
DurationFilter
)
asr = ASRInference(model="nvidia/stt_en_fastconformer_hybrid_large_pc" )
transcribed = asr(audio_dataset)
wer_filter = WERFilter(max_wer=0.3 )
high_quality_audio = wer_filter(transcribed)
duration_filter = DurationFilter(min_duration=1.0 , max_duration=30.0 )
filtered_audio = duration_filter(high_quality_audio)
Common patterns
Web scrape curation (Common Crawl) from nemo_curator import ScoreFilter, Modify
from nemo_curator.filters import *
from nemo_curator.modules import *
from nemo_curator.datasets import DocumentDataset
dataset = DocumentDataset.read_parquet("common_crawl/*.parquet" )
pipeline = [
WordCountFilter(min_words=100 , max_words=50000 ),
RepeatedLinesFilter(max_repeated_line_fraction=0.2 ),
SymbolToWordRatioFilter(max_symbol_to_word_ratio=0.3 ),
UrlRatioFilter(max_url_ratio=0.3 ),
LanguageIdentificationFilter(target_languages=["en" ]),
ExactDuplicates(id_field="id" , text_field="text" ),
FuzzyDuplicates(id_field="id" , text_field="text" , num_hashes=260 ),
PIIRedactor(),
NSFWClassifier(threshold=0.8 )
]
for stage in pipeline:
dataset = stage(dataset)
dataset.to_parquet("curated_common_crawl/" )
Distributed processing from nemo_curator import get_client
from dask_cuda import LocalCUDACluster
cluster = LocalCUDACluster(n_workers=8 )
client = get_client(cluster=cluster)
dataset = DocumentDataset.read_parquet("s3://large_dataset/*.parquet" )
deduped = FuzzyDuplicates(...)(dataset)
client.close()
cluster.close()
Performance benchmarks
Fuzzy deduplication (8TB RedPajama v2)
CPU (256 cores) : 120 hoursGPU (8× A100) : 7.5 hoursSpeedup : 16×
Exact deduplication (1TB)
CPU (64 cores) : 8 hoursGPU (4× A100) : 0.5 hoursSpeedup : 16×
Quality filtering (100GB)
CPU (32 cores) : 2 hoursGPU (2× A100) : 0.2 hoursSpeedup : 10×
Cost comparison CPU-based curation (AWS c5.18xlarge × 10):
Cost: $3.60/hour × 10 = $36/hour
Time for 8TB: 120 hours
Total : $4,320 GPU-based curation (AWS p4d.24xlarge × 2):
Cost: $32.77/hour × 2 = $65.54/hour
Time for 8TB: 7.5 hours
Total : $491.55 Savings : 89% reduction ($3,828 saved)
Supported data formats
Input : Parquet, JSONL, CSVOutput : Parquet (recommended), JSONLWebDataset : TAR archives for multi-modal
Use cases
NVIDIA used NeMo Curator to prepare Nemotron-4 training data
Open-source datasets curated: RedPajama v2, The Pile
References
Resources 