semdedup

name	semdedup
description	语义去重 - 基于高维特征嵌入进行相似度聚类去除语义冗余
category	quality
emoji	🔀
github_url	https://github.com/facebookresearch/SemDeDup
triggers	["语义去重","相似度聚类","冗余去除","semdedup"]
inputs	["json","parquet"]
outputs	["json","parquet"]
use_cases	["语义去重","冗余样本去除"]

SemDeDup

Facebook开发的语义去重工具。

安装

pip install semdedup

使用

semdedup --input data.jsonl --output deduped.jsonl