duplicate-detection

name	duplicate-detection
description	Use when user needs to find duplicate or similar content. Triggers on: duplicate, deduplication, plagiarism detection, similar content, near-duplicate, similarity detection, content dedup, find copies.

Duplicate Detection

Batch detection of duplicate or highly similar content using vector similarity.

When to Activate

Activate this skill when:

• User needs to find duplicates in a dataset
• User mentions "deduplication", "plagiarism", "similar content"
• User wants to clean data by removing near-duplicates
• User needs to merge similar items (FAQ, tickets, products)

Do NOT activate when:

• User needs general semantic search → use semantic-search
• User needs to group by topic → use clustering
• User needs recommendation → use rec-system

Interactive Flow

Step 1: Understand Duplicate Type

"What type of duplicates are you looking for?"

A) Exact duplicates (100% identical)

• Copy-paste content
• File deduplication
• Use hash-based detection (fast)

B) Near-duplicates (semantically similar)

• Paraphrased content
• Rewritten articles
• Use vector similarity (accurate)

C) Both

• Hash for exact, vector for near
• Most comprehensive

Which do you need? (A/B/C)

Step 2: Determine Threshold

"How similar should content be to be considered duplicate?"

Threshold	Interpretation	Use Case
0.95+	Near identical	Strict deduplication
0.85-0.95	Very similar	Plagiarism detection
0.75-0.85	Related	FAQ merging
0.65-0.75	Loosely related	Topic clustering

What threshold fits your needs?

Step 3: Confirm Configuration

"Based on your requirements:

• Method: Hash + Vector (comprehensive)
• Similarity threshold: 0.90
• Embedding model: BGE-large

Proceed? (yes / adjust [what])"

Core Concepts

Mental Model: Finding Twins

Think of duplicate detection as finding twins in a crowd:

• Identical twins (exact duplicates) → Same fingerprint (hash)
• Fraternal twins (near-duplicates) → Similar appearance (vectors)

┌─────────────────────────────────────────────────────────┐
│                  Duplicate Detection                     │
│                                                          │
│  New Content: "Machine learning requires lots of data"   │
│                         │                                │
│          ┌──────────────┴──────────────┐                │
│          │                             │                │
│          ▼                             ▼                │
│  ┌───────────────┐           ┌───────────────┐         │
│  │   Hash Check  │           │ Vector Search │         │
│  │   (Exact)     │           │  (Semantic)   │         │
│  └───────┬───────┘           └───────┬───────┘         │
│          │                           │                  │
│          ▼                           ▼                  │
│  "Content hash matches      "Similar to: 'ML needs     │
│   doc_123" → EXACT DUP       big datasets' (0.92)"     │
│                              → NEAR DUPLICATE           │
│                                                          │
│  Result: {"is_duplicate": true, "type": "near",         │
│           "similarity": 0.92, "match": "doc_123"}       │
└─────────────────────────────────────────────────────────┘

Hash vs Vector Detection

Method	Speed	Finds	Misses
Hash	★★★★★	Exact copies	Paraphrases
Vector	★★★	Semantic similarity	Different meanings same words
Both	★★★	Comprehensive	Best coverage

Implementation

from pymilvus import MilvusClient, DataType
from sentence_transformers import SentenceTransformer
import hashlib

class DuplicateDetector:
    def __init__(self, uri: str = "./milvus.db", threshold: float = 0.9):
        self.client = MilvusClient(uri=uri)
        self.model = SentenceTransformer('BAAI/bge-large-en-v1.5')
        self.threshold = threshold
        self.collection_name = "duplicate_detection"
        self._init_collection()

    def _init_collection(self):
        if self.client.has_collection(self.collection_name):
            return

        schema = self.client.create_schema()
        schema.add_field("id", DataType.VARCHAR, is_primary=True, max_length=64)
        schema.add_field("content_hash", DataType.VARCHAR, max_length=64)
        schema.add_field("content", DataType.VARCHAR, max_length=65535)
        schema.add_field("source", DataType.VARCHAR, max_length=512)
        schema.add_field("embedding", DataType.FLOAT_VECTOR, dim=1024)

        index_params = self.client.prepare_index_params()
        index_params.add_index(field_name="embedding", index_type=, metric_type=)
        index_params.add_index(field_name=, index_type=)

        .client.create_collection(
            collection_name=.collection_name,
            schema=schema,
            index_params=index_params
        )

     () -> :
        
        normalized = .join(content.lower().split())
         hashlib.md5(normalized.encode()).hexdigest()

     () -> :
        
        content_hash = ._hash_content(content)

        
        exact_match = .client.query(
            collection_name=.collection_name,
            =,
            output_fields=[, ],
            limit=
        )
         exact_match:
             {
                : ,
                : ,
                : exact_match[][],
                : exact_match[][],
                : 
            }

        
        embedding = .model.encode(content).tolist()
        results = .client.search(
            collection_name=.collection_name,
            data=[embedding],
            limit=,
            output_fields=[, , ]
        )

         results[]  results[][][] >= .threshold:
             {
                : ,
                : ,
                : results[][][][],
                : results[][][][],
                : results[][][],
                : results[][][][][:] + 
            }

         {
            : ,
            : results[][][]  results[]  
        }

     ():
        
        content_hash = ._hash_content(content)
        embedding = .model.encode(content).tolist()

        .client.insert(
            collection_name=.collection_name,
            data=[{
                : content_id,
                : content_hash,
                : content,
                : source,
                : embedding
            }]
        )

     () -> :
        
        unique = []
        duplicates = []

         item  items:
            result = .check_duplicate(item[], item.get(, ))
            result[] = item[]

             result[]:
                duplicates.append(result)
            :
                unique.append(item)
                .add_content(item[], item[], item.get(, ))

         {
            : unique,
            : duplicates,
            : (unique),
            : (duplicates),
            : (duplicates) / (items)  items  
        }


detector = DuplicateDetector(threshold=)


result = detector.check_duplicate()
 result[]:
    ()
:
    detector.add_content(, , )


results = detector.batch_dedup([
    {: , : , : },
    {: , : , : },  
    {: , : , : },
])
()

Threshold Selection Guide

Use Case	Threshold	Rationale
Exact dedup (data cleaning)	0.95+	Only near-identical
Plagiarism detection	0.85-0.90	Allow rewording
FAQ merging	0.80-0.85	Same question, different words
Similar content grouping	0.70-0.80	Topically related

Common Pitfalls

❌ Pitfall 1: Threshold Too Low

Problem: Everything is marked as duplicate

Why: Low threshold catches even loosely related content

Fix: Start with 0.90 and adjust down if needed

❌ Pitfall 2: Hash Only Detection

Problem: Paraphrased duplicates not detected

Why: Hash only catches exact matches

Fix: Always use vector similarity for semantic duplicates

❌ Pitfall 3: Not Normalizing Before Hash

Problem: Same content with different whitespace not detected

Why: Hash is sensitive to exact characters

Fix: Normalize content before hashing

normalized = ''.join(content.lower().split())

❌ Pitfall 4: Processing Order Matters

Problem: Different "original" detected depending on order

Why: First item becomes the reference

Fix: Process by timestamp (oldest first) or have clear policy

When to Level Up

Need	Upgrade To
Group similar content	clustering
Plagiarism with source matching	verticals/plagiarism.md
Large-scale batch processing	Add core:ray

References

• Plagiarism detection: verticals/plagiarism.md
• Content deduplication: verticals/content-dedup.md
• FAQ merging: verticals/faq-merge.md