Also known as Record Linkage, Deduplication, or Data Matching, is the process of identifying, matching, and merging records that refer to the same real-world entity across one or more data sources. It is a critical task in data integration when records lack a unique identifier (such as a SSN or a Product SKU).
The ER Workflow
The process typically follows a specific pipeline to handle fuzzy or inconsistent data:
- Ingestion & Cleaning: Standardizing formats (e.g., converting St. to Street) and removing noise.
- Blocking: A performance-saving step where the dataset is divided into “blocks” based on similar attributes (e.g., matching only records with the same zip code) to avoid comparing every single record to every other record.
- Comparison: Applying similarity algorithms to compare attributes (Names, Addresses, DOBs).
- Classification: Using thresholds or machine learning to decide if a pair is a Match, Non-Match, or Potential Match (requiring human review).
Key Comparison Methods
| Method | How it Works | Use Case |
| Deterministic | Requires an exact match on specific fields. | When data is high-quality and contains unique IDs. |
| Probabilistic | Calculates a weights-based score (e.g., Fellegi-Sunter) to determine the likelihood of a match. | Handling typos, nicknames, or missing data. |
| Machine Learning | Uses training data to learn which attribute combinations constitute a match. | Complex datasets where rules are too difficult to manually define. |
Why ER Matters
- Customer 360: Creating a unified view of a customer who may have different profiles in a CRM, an email marketing tool, and a billing system.
- Fraud Detection: Identifying individuals opening multiple accounts under slightly different names or addresses.
- Healthcare: Linking patient records across different hospitals to ensure a complete medical history.
Common ER Challenges
- Scalability: As a dataset grows by N, the number of possible comparisons grows at O(N2), making blocking strategies essential.
- Data Quality: Variations like Jon Doe vs. Jonathan Doe or 123 Main St vs. 123 Main Street, Apt 4.
Additional Acronyms for ER
- ER - Entity-Relationship
