Skip to main content
nemo-curator GPU-accelerated data curation for LLM training. Supports text/image/video/audio. Features fuzzy deduplication (16× faster), quality filtering (30+ heuristics), semantic deduplication, PII redaction, NSFW detection. Scales across GPUs with RAPIDS. Use for preparing high-quality training datasets, cleaning web data, or deduplicating large corpora.
Zur Installation springen Skills Marktplatz Entdecken und erkunden Sie KI-Skills, die von der Community erstellt wurden.
Mit Codex oder Claude installieren Kopieren Sie diesen Prompt, fügen Sie ihn in Codex, Claude oder einen anderen Assistant ein und lassen Sie die Skill-Seite prüfen und installieren.
Prompt-Details anzeigen Ein direkter Befehl überspringt den Prüf-Prompt. Prüfen Sie die Quelle, bevor Sie ihn ausführen.
npx skills add https://github.com/davila7/claude-code-templates --skill nemo-curatorDer Befehl bleibt in einer Zeile. Scrollen Sie horizontal, um ihn vor dem Kopieren vollständig zu prüfen.
Sie bevorzugen eine lokale Kopie? Laden Sie die Dateien herunter, die SkillsMP derzeit vorliegen.
ZIP herunterladen Herunterladen... Verwandte Berufe SOC
Basierend auf der SOC-Berufsklassifikation
name nemo-curator description GPU-accelerated data curation for LLM training. Supports text/image/video/audio. Features fuzzy deduplication (16× faster), quality filtering (30+ heuristics), semantic deduplication, PII redaction, NSFW detection. Scales across GPUs with RAPIDS. Use for preparing high-quality training datasets, cleaning web data, or deduplicating large corpora. version 1.0.0 author Orchestra Research license MIT tags ["Data Processing","NeMo Curator","Data Curation","GPU Acceleration","Deduplication","Quality Filtering","NVIDIA","RAPIDS","PII Redaction","Multimodal","LLM Training Data"] dependencies ["nemo-curator","cudf","dask","rapids"]
NeMo Curator - GPU-Accelerated Data Curation
NVIDIA's toolkit for preparing high-quality training data for LLMs.
When to use NeMo Curator
Preparing LLM training data from web scrapes (Common Crawl)
Need fast deduplication (16× faster than CPU)
Curating multi-modal datasets (text, images, video, audio)
Filtering low-quality or toxic content
Scaling data processing across GPU cluster
16× faster fuzzy deduplication (8TB RedPajama v2)40% lower TCO vs CPU alternativesNear-linear scaling across GPU nodes Use alternatives instead :
datatrove : CPU-based, open-source data processingdolma : Allen AI's data toolkitRay Data : General ML data processing (no curation focus)
Quick start
Installation
uv pip install "nemo-curator[text_cuda12]"
uv pip install "nemo-curator[all_cuda12]"
uv pip install "nemo-curator[cpu]"
Basic text curation pipeline from nemo_curator import ScoreFilter, Modify
from nemo_curator.datasets import DocumentDataset
import pandas as pd
df = pd.DataFrame({"text" : ["Good document" , "Bad doc" , "Excellent text" ]})
dataset = DocumentDataset(df)
def quality_score (doc ):
return len (doc["text" ].split()) > 5
filtered = ScoreFilter(quality_score)(dataset)
from nemo_curator.modules import ExactDuplicates
deduped = ExactDuplicates()(filtered)
deduped.to_parquet("curated_data/" )
Data curation pipeline
Stage 1: Quality filtering from nemo_curator.filters import (
WordCountFilter,
RepeatedLinesFilter,
UrlRatioFilter,
NonAlphaNumericFilter
)
from nemo_curator import ScoreFilter
dataset = dataset.filter (WordCountFilter(min_words=50 , max_words=100000 ))
dataset = dataset.filter (RepeatedLinesFilter(max_repeated_line_fraction=0.3 ))
dataset = dataset.filter (UrlRatioFilter(max_url_ratio=0.2 ))
Stage 2: Deduplication from nemo_curator.modules import ExactDuplicates
deduped = ExactDuplicates(id_field="id" , text_field="text" )(dataset)
Fuzzy deduplication (16× faster on GPU):
from nemo_curator.modules import FuzzyDuplicates
fuzzy_dedup = FuzzyDuplicates(
id_field="id" ,
text_field="text" ,
num_hashes=260 ,
num_buckets=20 ,
hash_method="md5"
)
deduped = fuzzy_dedup(dataset)
from nemo_curator.modules import SemanticDuplicates
semantic_dedup = SemanticDuplicates(
id_field="id" ,
text_field="text" ,
embedding_model="sentence-transformers/all-MiniLM-L6-v2" ,
threshold=0.8
)
deduped = semantic_dedup(dataset)
Stage 3: PII redaction from nemo_curator.modules import Modify
from nemo_curator.modifiers import PIIRedactor
pii_redactor = PIIRedactor(
supported_entities=["EMAIL_ADDRESS" , "PHONE_NUMBER" , "PERSON" , "LOCATION" ],
anonymize_action="replace"
)
redacted = Modify(pii_redactor)(dataset)
Stage 4: Classifier filtering from nemo_curator.classifiers import QualityClassifier
quality_clf = QualityClassifier(
model_path="nvidia/quality-classifier-deberta" ,
batch_size=256 ,
device="cuda"
)
high_quality = dataset.filter (lambda doc: quality_clf(doc["text" ]) > 0.5 )
GPU acceleration
GPU vs CPU performance Operation CPU (16 cores) GPU (A100) Speedup Fuzzy dedup (8TB) 120 hours 7.5 hours 16× Exact dedup (1TB) 8 hours 0.5 hours 16× Quality filtering 2 hours 0.2 hours 10×
Multi-GPU scaling from nemo_curator import get_client
import dask_cuda
client = get_client(cluster_type="gpu" , n_workers=8 )
deduped = FuzzyDuplicates(...)(dataset)
Multi-modal curation
Image curation from nemo_curator.image import (
AestheticFilter,
NSFWFilter,
CLIPEmbedder
)
aesthetic_filter = AestheticFilter(threshold=5.0 )
filtered_images = aesthetic_filter(image_dataset)
nsfw_filter = NSFWFilter(threshold=0.9 )
safe_images = nsfw_filter(filtered_images)
clip_embedder = CLIPEmbedder(model="openai/clip-vit-base-patch32" )
image_embeddings = clip_embedder(safe_images)
Video curation from nemo_curator.video import (
SceneDetector,
ClipExtractor,
InternVideo2Embedder
)
scene_detector = SceneDetector(threshold=27.0 )
scenes = scene_detector(video_dataset)
clip_extractor = ClipExtractor(min_duration=2.0 , max_duration=10.0 )
clips = clip_extractor(scenes)
video_embedder = InternVideo2Embedder()
video_embeddings = video_embedder(clips)
Audio curation from nemo_curator.audio import (
ASRInference,
WERFilter,
DurationFilter
)
asr = ASRInference(model="nvidia/stt_en_fastconformer_hybrid_large_pc" )
transcribed = asr(audio_dataset)
wer_filter = WERFilter(max_wer=0.3 )
high_quality_audio = wer_filter(transcribed)
duration_filter = DurationFilter(min_duration=1.0 , max_duration=30.0 )
filtered_audio = duration_filter(high_quality_audio)
Common patterns
Web scrape curation (Common Crawl) from nemo_curator import ScoreFilter, Modify
from nemo_curator.filters import *
from nemo_curator.modules import *
from nemo_curator.datasets import DocumentDataset
dataset = DocumentDataset.read_parquet("common_crawl/*.parquet" )
pipeline = [
WordCountFilter(min_words=100 , max_words=50000 ),
RepeatedLinesFilter(max_repeated_line_fraction=0.2 ),
SymbolToWordRatioFilter(max_symbol_to_word_ratio=0.3 ),
UrlRatioFilter(max_url_ratio=0.3 ),
LanguageIdentificationFilter(target_languages=["en" ]),
ExactDuplicates(id_field="id" , text_field="text" ),
FuzzyDuplicates(id_field="id" , text_field="text" , num_hashes=260 ),
PIIRedactor(),
NSFWClassifier(threshold=0.8 )
]
for stage in pipeline:
dataset = stage(dataset)
dataset.to_parquet("curated_common_crawl/" )
Distributed processing from nemo_curator import get_client
from dask_cuda import LocalCUDACluster
cluster = LocalCUDACluster(n_workers=8 )
client = get_client(cluster=cluster)
dataset = DocumentDataset.read_parquet("s3://large_dataset/*.parquet" )
deduped = FuzzyDuplicates(...)(dataset)
client.close()
cluster.close()
Performance benchmarks
Fuzzy deduplication (8TB RedPajama v2)
CPU (256 cores) : 120 hoursGPU (8× A100) : 7.5 hoursSpeedup : 16×
Exact deduplication (1TB)
CPU (64 cores) : 8 hoursGPU (4× A100) : 0.5 hoursSpeedup : 16×
Quality filtering (100GB)
CPU (32 cores) : 2 hoursGPU (2× A100) : 0.2 hoursSpeedup : 10×
Cost comparison CPU-based curation (AWS c5.18xlarge × 10):
Cost: $3.60/hour × 10 = $36/hour
Time for 8TB: 120 hours
Total : $4,320 GPU-based curation (AWS p4d.24xlarge × 2):
Cost: $32.77/hour × 2 = $65.54/hour
Time for 8TB: 7.5 hours
Total : $491.55 Savings : 89% reduction ($3,828 saved)
Supported data formats
Input : Parquet, JSONL, CSVOutput : Parquet (recommended), JSONLWebDataset : TAR archives for multi-modal
Use cases
NVIDIA used NeMo Curator to prepare Nemotron-4 training data
Open-source datasets curated: RedPajama v2, The Pile
References
Resources Mehr aus diesem Repository Build AI agents that interact with computers like humans do - viewing screens, moving cursors, clicking buttons, and typing text. Covers Anthropic's Computer Use, OpenAI's Operator/CUA, and open-source alternatives. Critical focus on sandboxing, security, and handling the unique challenges of vision-based control. Use when: computer use, desktop automation agent, screen control AI, vision-based agent, GUI automation.
Expert in building products that wrap AI APIs (OpenAI, Anthropic, etc.) into focused tools people will pay for. Not just 'ChatGPT but different' - products that solve specific problems with AI. Covers prompt engineering for products, cost management, rate limiting, and building defensible AI businesses. Use when: AI wrapper, GPT product, AI tool, wrap AI, AI SaaS.
github-workflow-automation Automate GitHub workflows with AI assistance. Includes PR reviews, issue triage, CI/CD integration, and Git operations. Use when automating GitHub workflows, setting up PR review automation, creating GitHub Actions, or triaging issues.
