Cleans raw text data (Unicode normalization, deduplication via MinHash/SimHash, perplexity filtering, language ID) and manages annotation pipelines (inter-annotator agreement, gold sets, calibration). Use when preparing training or evaluation data that needs…
Quellsprache: Englisch