deduplication

name	deduplication
description	Event deduplication with canonical selection, reputation scoring, and hash-based grouping for multi-source data aggregation. Handles both ID-based and content-based deduplication.
license	MIT
compatibility	TypeScript/JavaScript
metadata	{"category":"data-access","time":"4h","source":"drift-masterguide"}

Event Deduplication

Canonical selection with reputation scoring and hash-based grouping for multi-source data.

When to Use This Skill

• Aggregating data from multiple sources (news, events, products)
• Same content appears from different outlets/sources
• Need to pick the "best" version from duplicates
• Tracking deduplication metrics for optimization

Core Concepts

Simple URL deduplication isn't enough. Production needs:

• Grouping by semantic similarity (same story, different outlets)
• Canonical selection (pick the "best" version)
• Reputation scoring (prefer authoritative sources)
• Both ID-based and content-based deduplication

Two modes:

1. ID-based: When sources have unique IDs, keep the "best" version when IDs collide
2. Content-based: Group by semantic similarity, select canonical from each group

Implementation

TypeScript

import { createHash } from 'crypto';

interface DeduplicationResult<T> {
  items: T[];
  originalCount: number;
  dedupedCount: number;
  reductionPercent: number;
  duplicateGroups?: number;
}

// ============================================
// ID-Based Deduplication
// ============================================

function deduplicateById<T extends { id: string }>(
  items: T[],
  preferFn: (existing: T, candidate: T) => T
): DeduplicationResult<T> {
  const seen = new Map<string, T>();
  
  for (const item of items) {
    const existing = seen.get(item.id);
    if (existing) {
      seen.set(item.id, preferFn(existing, item));
    } else {
      seen.set(item.id, item);
    }
  }
  
  const dedupedItems = Array.from(seen.values());
  const reductionPercent = items. > 
    ? .(( - dedupedItems. / items.) * )
    : ;
  
   {
    : dedupedItems,
    : items.,
    : dedupedItems.,
    reductionPercent,
  };
}





  {
  : ;
  : ;
  : ;
  : ;
  ?: ;
}


 ():  {
   normalizedTitle = article.
    .()
    .(, )
    .()
    .(, );

   dateStr = article.?.(, ).(, ) || ;

   ;
}


 ():  {
   ().(url).().(, );
}


 ():  {
  
   tier1 = [, , , , 
                 , , ];
   (tier1.( domain.(r)))  ;
  
  
   tier2 = [, , , 
                 , , ];
   (tier2.( domain.(r)))  ;
  
  
   tier3 = [, , , ];
   (tier3.( domain.(r)))  ;
  
   ;
}


 selectCanonical<T  >(
  : { : T; :  }[]
): { : T; :  } {
   group.( {
     bestScore = (best..) + 
                      .(best.. || );
     currentScore = (current..) + 
                         .(current.. || );
    
     currentScore > bestScore ? current : best;
  });
}


 deduplicateArticles<T  >(
  : { : ; : T[] }[]
): <T & { :  }> {
   groups =  <, { : T; :  }[]>();
   totalArticles = ;

  
   ( { sourceName, articles }  sourceResults) {
     ( article  articles) {
      totalArticles++;
       key = (article);
      
       (!groups.(key)) {
        groups.(key, []);
      }
      groups.(key)!.({ : article, : sourceName });
    }
  }

  
   : (T & { :  })[] = [];
  
   ( group  groups.()) {
     canonical = (group);
    items.({ ...canonical., : canonical. });
  }

   reductionPercent = totalArticles >  
    ? .(( - items. / totalArticles) * )
    : ;

  .();

   {
    items,
    : totalArticles,
    : items.,
    reductionPercent,
    : groups.,
  };
}

Usage Examples

ID-Based Deduplication

const events = await fetchEvents();

const result = deduplicateById(events, (existing, candidate) => {
  // Prefer events with coordinates
  if (!existing.lat && candidate.lat) return candidate;
  // Prefer higher sentiment magnitude
  if (Math.abs(candidate.sentiment) > Math.abs(existing.sentiment)) {
    return candidate;
  }
  return existing;
});

console.log(`Reduced ${result.reductionPercent}% duplicates`);

Multi-Source Aggregation

const results = await Promise.all([
  fetchFromSourceA(),
  fetchFromSourceB(),
  fetchFromSourceC(),
]);

const { items, reductionPercent } = deduplicateArticles([
  { sourceName: 'source-a', articles: results[0] },
  { sourceName: 'source-b', articles: results[1] },
  { sourceName: 'source-c', articles: results[2] },
]);

// items now contains canonical articles with source attribution

Best Practices

1. Semantic grouping - Group by normalized content, not just URL
2. Reputation scoring - Prefer authoritative sources as canonical
3. Best version selection - When IDs collide, keep version with most data
4. Reduction tracking - Log how much deduplication helped
5. Source attribution - Track which source the canonical came from

Common Mistakes

• Simple URL deduplication (misses same story from different outlets)
• Random selection from duplicates (lose quality signal)
• No normalization (case/punctuation differences create false negatives)
• Not tracking reduction metrics (can't optimize)
• Hardcoded source lists (make configurable)

Related Patterns

• batch-processing - Process deduplicated items efficiently
• validation-quarantine - Validate before deduplication
• checkpoint-resume - Track which files have been deduplicated