Deduplicate HubSpot contacts at production scale — surviving import storms, wrong-winner
merges, fuzzy-match blind spots, association orphans, rate-limit exhaustion, and silent
merge failures on conflicting lifecycle or opt-out status. Use when cleaning a CRM after
a bulk import, running a nightly dedup pipeline on millions of records, recovering from
a merge that destroyed the wrong timeline, or building fuzzy matching beyond HubSpot's
native email-uniqueness. Trigger with "hubspot dedup", "hubspot merge contacts",
"hubspot duplicate contacts", "hubspot contact cleanup", "hubspot import duplicates",
"hubspot fuzzy match contacts".
Deduplicate HubSpot contacts at production scale — surviving import storms, wrong-winner
merges, fuzzy-match blind spots, association orphans, rate-limit exhaustion, and silent
merge failures on conflicting lifecycle or opt-out status. Use when cleaning a CRM after
a bulk import, running a nightly dedup pipeline on millions of records, recovering from
a merge that destroyed the wrong timeline, or building fuzzy matching beyond HubSpot's
native email-uniqueness. Trigger with "hubspot dedup", "hubspot merge contacts",
"hubspot duplicate contacts", "hubspot contact cleanup", "hubspot import duplicates",
"hubspot fuzzy match contacts".
allowed-tools
Read, Bash(curl:*), Bash(jq:*), Bash(python3:*)
version
2.9.0
license
MIT
author
Jeremy Longshore <jeremy@intentsolutions.io>
compatibility
Designed for Claude Code
tags
["hubspot","crm","deduplication","data-quality"]
HubSpot Contact Deduplication
Overview
Merge duplicate contacts in HubSpot and operate that process in production, at scale, without data loss. This is not a one-click cleanup guide — it is the logic your pipeline runs when a sales ops team imports 80,000 leads from a tradeshow CSV that already exist in the CRM, when a merge destroys the "winner" contact's email history, when a fuzzy match on "Jon" vs "John" leaves a six-figure deal associated to a ghost record, and when on-call discovers that 40,000 contacts were merged without checking opt-out flags.
The six production failures this skill prevents:
Import storms creating thousands of exact duplicates — HubSpot enforces email uniqueness only at the property level; the merge API has no dedup-all-at-once endpoint. A 100K-row CSV import where 60% of rows already exist creates 60,000 duplicates that must be found and merged one pair at a time within a 100 req/10s rate envelope.
Merge destroying the wrong timeline — POST /crm/v3/objects/contacts/merge requires a primaryObjectId. Picking the wrong one demotes the older contact's full activity timeline — calls, emails, form submissions — to the discarded record's history.
Property-based dedup missing fuzzy matches — Email-exact dedup leaves "john@gmail.com" and "jon.smith@googlemail.com" as separate records. Phone dedup leaves "+1 (512) 867-5309" and "5128675309" as separate records. Without normalization your CRM accumulates a shadow population of semantically identical but technically distinct contacts.
Post-merge association orphans — When a secondary contact has deals, tickets, or company associations, HubSpot re-parents most automatically — but not all. Custom object associations and some third-party-integration links may not follow.
Rate-limit exhaustion on large catalogs — A 1-million-contact dedup scan requires 10,000 batch reads (2.7 hours at full throughput, before merge calls). Naive single-threaded loops exhaust the 500K daily quota before the search phase finishes.
Silent merge failures on conflicting lifecycle or opt-out status — The merge API returns 200 even when the resulting contact has hs_email_optout=true overriding the primary's opted-in status. HubSpot's "most recently updated value wins" rule is wrong for compliance flags.
Auth
Authenticate with a private app token (pat-na1-*) or OAuth access token. Pass it on every request:
Authorization: Bearer {your-token}
Required scopes: crm.objects.contacts.read, , , . See the for token caching, OAuth refresh, and scope-drift detection.
Python 3.10+ (requests, phonenumbers, rapidfuzz) for the full pipeline
HubSpot Professional or Enterprise account (batch merge at scale)
Private app token with required scopes (above)
jq for shell examples
For catalogs >500K contacts: confirm daily quota with HubSpot support
Instructions
Step 1. Discover duplicates with search
Find exact duplicates by email using the search API. Never pull all contacts into memory for comparison — use the search endpoint with specific filter values.
For full-portal scans across millions of contacts use the four-stage Python pipeline in implementation-guide.md. The pipeline writes a local SQLite checkpoint so rate-limit interruptions do not require starting over.
Step 2. Select the primary (winner) contact
The oldest contact by createdate is the primary — its timeline is most historically complete. Two overrides apply:
If the oldest contact has hs_email_optout=true and the newer one does not, prefer the opted-in record as primary to avoid propagating unsubscribe status.
If the oldest contact has a test-domain email (@mailinator.com, @example.com, @test.com), always make the real-address contact the primary.
from datetime import datetime
defpick_primary(contacts: list[dict]) -> tuple[dict, list[dict]]:
"""Return (primary, secondaries). contacts is a list of HubSpot result dicts."""
TEST_DOMAINS = {"mailinator.com","example.com","test.com","yopmail.com"}
defis_test(email: str) -> bool:
return (email or"").split("@")[-1].lower() in TEST_DOMAINS
# Sort oldest first (default primary)
sorted_c = sorted(contacts, key=lambda c: c["properties"]["createdate"])
primary = sorted_c[0]
# Opt-out overrideif primary["properties"].get("hs_email_optout") == "true":
opted_in = next((c for c in sorted_c[1:] if c["properties"].get("hs_email_optout") != "true"), None)
if opted_in:
primary = opted_in
# Test email overrideif is_test(primary["properties"].get("email", "")):
real = next((c for c in sorted_c ifnot is_test(c["properties"].get("email", ""))), None)
if real:
primary = real
secondaries = [c for c in contacts if c["id"] != primary["id"]]
return primary, secondaries
Step 3. Normalize emails and phones for fuzzy matching
Exact-email dedup leaves a shadow population. Normalize before comparing:
DAILY_STOP_AT = 480_000# Stop at 96% of 500K quotadefcheck_quota(resp: requests.Response) -> None:
remaining = int(resp.headers.get("X-HubSpot-RateLimit-Daily-Remaining", 500_000))
if (500_000 - remaining) >= DAILY_STOP_AT:
raise SystemExit("Daily quota near limit — stopping. Resume after midnight UTC reset.")
Step 6. Post-merge verification and association repair
After merging, verify that the surviving contact's hs_email_optout matches the expected value (Step 4) and patch it if it drifted. Then audit associations that may not have transferred automatically:
# Check associations on surviving contact (replace 12345 with actual primary contact ID)
curl -s "https://api.hubapi.com/crm/v4/objects/contacts/12345/associations/deals" \
-H "Authorization: Bearer {your-token}" | jq '[.results[].toObjectId]'# Manually create a missing association (replace 12345 with primary ID, 67890 with deal ID)
curl -s -X PUT \
"https://api.hubapi.com/crm/v4/objects/contacts/12345/associations/deals/67890" \
-H "Authorization: Bearer {your-token}" \
-H "Content-Type: application/json" \
-d '[{"associationCategory":"HUBSPOT_DEFINED","associationTypeId":3}]'
The full four-stage Python pipeline (scan → pair → qualify → execute) with automatic association repair is in implementation-guide.md.
Error Handling
HTTP Status
Error
Root Cause
Action
400
CONTACT_ALREADY_MERGED
Secondary was already merged into another record
Re-fetch secondary; check hs_merged_object_ids for surviving primary ID
400
SAME_OBJECT_MERGE
Both IDs are identical
Remove self-merge pairs from candidate list before executing
400
INVALID_OBJECT_TYPE
One ID belongs to a different CRM object type
Verify via GET /crm/v3/objects/contacts/{id} before merging
404
OBJECT_NOT_FOUND
Contact was deleted between discovery and merge
Re-fetch to confirm existence; skip if deleted
409
MERGE_IN_PROGRESS
A concurrent merge is already running for this contact
