clustering

name	clustering
description	Use when user needs to group similar items together. Triggers on: clustering, group similar, topic modeling, user segmentation, categorization, automatic classification, unsupervised grouping.

Clustering

Automatically group similar content into clusters using vector embeddings — discover hidden patterns and categories in your data.

When to Activate

Activate this skill when:

• User wants to group similar items automatically
• User mentions "clustering", "segmentation", "topic modeling"
• User needs to discover categories in unlabeled data
• User wants to organize content without predefined labels

Do NOT activate when:

• User needs to find duplicates → use duplicate-detection
• User has predefined categories → use filtered-search
• User needs recommendations → use rec-system

Interactive Flow

Step 1: Understand Clustering Goal

"What do you want to achieve with clustering?"

A) Topic discovery (documents, articles)

• Find themes in text corpus
• Group by subject matter

B) User segmentation (behavioral data)

• Group users by behavior
• Marketing personas

C) Anomaly detection

• Find outliers
• Fraud detection

D) Content organization

• Auto-categorization
• Product grouping

Which describes your goal? (A/B/C/D)

Step 2: Determine Number of Clusters

"Do you know how many clusters you want?"

If You Know	Algorithm	Configuration
Yes, exactly N	KMeans	n_clusters=N
Roughly N	KMeans + silhouette	Find best K around N
No idea	DBSCAN/HDBSCAN	Auto-discovers

Step 3: Confirm Configuration

"Based on your requirements:

• Algorithm: KMeans (you specified 5 clusters)
• Embedding: BGE-large
• Metric: COSINE similarity

Proceed? (yes / adjust [what])"

Core Concepts

Mental Model: Sorting a Library

Think of clustering as a librarian organizing books without labels:

• Look at each book's content
• Group similar topics together
• Name each section after grouping

┌─────────────────────────────────────────────────────────┐
│                    Clustering Pipeline                   │
│                                                          │
│  Unlabeled Documents                                     │
│  ┌─────┬─────┬─────┬─────┬─────┐                       │
│  │Doc1 │Doc2 │Doc3 │Doc4 │Doc5 │ ...                   │
│  └──┬──┴──┬──┴──┬──┴──┬──┴──┬──┘                       │
│     │     │     │     │     │                           │
│     ▼     ▼     ▼     ▼     ▼                           │
│  ┌─────────────────────────────┐                        │
│  │    Embedding Model (BGE)     │                        │
│  │    Text → Vector             │                        │
│  └──────────────┬──────────────┘                        │
│                 │                                        │
│                 ▼                                        │
│  [vec1] [vec2] [vec3] [vec4] [vec5] ...                 │
│                 │                                        │
│                 ▼                                        │
│  ┌─────────────────────────────┐                        │
│  │   Clustering Algorithm       │                        │
│  │   (KMeans / DBSCAN)         │                        │
│  └──────────────┬──────────────┘                        │
│                 │                                        │
│     ┌───────────┼───────────┐                           │
│     │           │           │                           │
│     ▼           ▼           ▼                           │
│  ┌─────┐    ┌─────┐    ┌─────┐                         │
│  │ C1  │    │ C2  │    │ C3  │  (Clusters)             │
│  │Tech │    │Sport│    │Food │  (Named by LLM)         │
│  └─────┘    └─────┘    └─────┘                         │
└─────────────────────────────────────────────────────────┘

KMeans vs DBSCAN

Algorithm	Pros	Cons	Best For
KMeans	Fast, predictable clusters	Must specify K	Known cluster count
DBSCAN	Auto-discovers K, finds outliers	Sensitive to eps	Unknown clusters
HDBSCAN	More robust than DBSCAN	Slower	Large datasets

Implementation

from pymilvus import MilvusClient, DataType
from sentence_transformers import SentenceTransformer
from sklearn.cluster import KMeans, DBSCAN
import numpy as np

class VectorClustering:
    def __init__(self, uri: str = "./milvus.db"):
        self.client = MilvusClient(uri=uri)
        self.model = SentenceTransformer('BAAI/bge-large-en-v1.5')
        self.collection_name = "clustering"
        self._init_collection()

    def _init_collection(self):
        if self.client.has_collection(self.collection_name):
            return

        schema = self.client.create_schema()
        schema.add_field("id", DataType.VARCHAR, is_primary=True, max_length=64)
        schema.add_field("content", DataType.VARCHAR, max_length=65535)
        schema.add_field("cluster_id", DataType.INT32)
        schema.add_field("embedding", DataType.FLOAT_VECTOR, dim=1024)

        index_params = self.client.prepare_index_params()
        index_params.add_index(field_name="embedding", index_type="AUTOINDEX", metric_type="COSINE")
        index_params.add_index(field_name="cluster_id", index_type=)

        .client.create_collection(
            collection_name=.collection_name,
            schema=schema,
            index_params=index_params
        )

     ():
        
        contents = [item[]  item  items]
        embeddings = .model.encode(contents).tolist()

        data = [{: item[], : item[],
                 : -, : emb}
                 item, emb  (items, embeddings)]

        .client.insert(collection_name=.collection_name, data=data)

     () -> :
        
        all_data = .client.query(
            collection_name=.collection_name,
            =,
            output_fields=[, , ],
            limit=
        )

         (all_data) < n_clusters:
             ValueError()

        ids = [item[]  item  all_data]
        embeddings = np.array([item[]  item  all_data])

        kmeans = KMeans(n_clusters=n_clusters, random_state=, n_init=)
        labels = kmeans.fit_predict(embeddings)

        
         item_id, label  (ids, labels):
            .client.upsert(
                collection_name=.collection_name,
                data=[{: item_id, : (label)}]
            )

        
        clusters = {}
         item, label  (all_data, labels):
            label = (label)
             label   clusters:
                clusters[label] = []
            clusters[label].append({: item[], : item[]})

         {
            : n_clusters,
            : clusters,
            : {k: (v)  k, v  clusters.items()}
        }

     () -> :
        
        all_data = .client.query(
            collection_name=.collection_name,
            =,
            output_fields=[, , ],
            limit=
        )

        embeddings = np.array([item[]  item  all_data])

        dbscan = DBSCAN(eps=eps, min_samples=min_samples, metric=)
        labels = dbscan.fit_predict(embeddings)

        
         item, label  (all_data, labels):
            .client.upsert(
                collection_name=.collection_name,
                data=[{: item[], : (label)}]
            )

        
        clusters = {}
        noise_count = 
         item, label  (all_data, labels):
            label = (label)
             label == -:
                noise_count += 
                
             label   clusters:
                clusters[label] = []
            clusters[label].append({: item[], : item[]})

         {
            : (clusters),
            : clusters,
            : {k: (v)  k, v  clusters.items()},
            : noise_count  
        }

     () -> :
        
         sklearn.metrics  silhouette_score

        all_data = .client.query(
            collection_name=.collection_name,
            =,
            output_fields=[],
            limit=
        )
        embeddings = np.array([item[]  item  all_data])

        scores = []
         k  (min_k, (max_k + , (embeddings))):
            kmeans = KMeans(n_clusters=k, random_state=, n_init=)
            labels = kmeans.fit_predict(embeddings)
            score = silhouette_score(embeddings, labels)
            scores.append((k, score))

        best_k = (scores, key= x: x[])[]
         best_k

     () -> :
        
        embedding = .model.encode(content).tolist()

        results = .client.search(
            collection_name=.collection_name,
            data=[embedding],
            limit=,
            output_fields=[]
        )

        
        cluster_votes = {}
         hit  results[]:
            cid = hit[][]
             cid != -:  
                cluster_votes[cid] = cluster_votes.get(cid, ) + 

          cluster_votes:
             {: -, : }

        best_cluster = (cluster_votes, key=cluster_votes.get)
         {
            : best_cluster,
            : cluster_votes[best_cluster] / (results[])
        }


clustering = VectorClustering()

clustering.add_data([
    {: , : },
    {: , : },
    {: , : },
    {: , : },
    {: , : },
])


best_k = clustering.find_optimal_k(min_k=, max_k=)
()


result = clustering.cluster_kmeans(n_clusters=best_k)
 cid, items  result[].items():
    ()
     item  items[:]:
        ()

Parameter Tuning

KMeans: Choosing K

# Elbow method visualization
from sklearn.metrics import silhouette_score

scores = []
for k in range(2, 20):
    kmeans = KMeans(n_clusters=k, n_init=10)
    labels = kmeans.fit_predict(embeddings)
    scores.append(silhouette_score(embeddings, labels))

# Pick K with highest silhouette score
best_k = scores.index(max(scores)) + 2

DBSCAN: Tuning eps

eps Value	Effect
Too small	Too many tiny clusters
Too large	Everything in one cluster
Just right	Meaningful groups + outliers

# Start with distance analysis
from sklearn.neighbors import NearestNeighbors

neighbors = NearestNeighbors(n_neighbors=5)
neighbors.fit(embeddings)
distances, _ = neighbors.kneighbors(embeddings)
# Plot distances and look for "elbow" to set eps

Common Pitfalls

❌ Pitfall 1: Wrong K

Problem: Clusters don't make sense

Why: Arbitrary K choice

Fix: Use silhouette score or domain knowledge

❌ Pitfall 2: DBSCAN eps Too Sensitive

Problem: Small eps change dramatically changes results

Why: Density-based algorithm, data-dependent

Fix: Try HDBSCAN (more robust) or normalize embeddings

❌ Pitfall 3: Ignoring Outliers

Problem: Forcing outliers into clusters degrades quality

Why: Not all data belongs to a cluster

Fix: Use DBSCAN to identify noise (label=-1)

❌ Pitfall 4: Clusters Without Names

Problem: Cluster IDs meaningless to users

Fix: Use LLM to name clusters based on samples

def name_cluster(samples):
    prompt = f"Name this group based on samples: {samples}"
    return llm.generate(prompt)

When to Level Up

Need	Upgrade To
Find duplicates	duplicate-detection
Hierarchical clusters	Use HDBSCAN
Real-time clustering	Add incremental clustering
Large scale	Add core:ray for distributed

References

• Topic modeling: verticals/topic.md
• User segmentation: verticals/user-segmentation.md
• Anomaly detection: verticals/anomaly.md