A critical task in data integration, Entity Resolution (ER) identifies, matches, and merges records that refer to the same real-world entity across one or more data sources. This process is essential when records lack a unique identifier, such as a Social Security Number (SSN) or a Product Stock Keeping Unit (SKU). ER is also known as Record Linkage, Deduplication, or Data Matching.
The ER Workflow
The process typically follows a specific pipeline to handle fuzzy or inconsistent data:
- Ingestion & Cleaning: Standardizing formats (e.g., converting St. to Street) and removing noise.
- Blocking: A performance-saving step where the dataset is divided into "blocks" based on similar attributes (e.g., matching only records with the same zip code) to avoid comparing every single record to every other record.
- Comparison: Applying similarity algorithms to compare attributes (Names, Addresses, Dates of Birth).
- Classification: Using thresholds or machine learning to decide if a pair is a Match, Non-Match, or Potential Match (requiring human review).
Key Comparison Methods
- Deterministic: Requires an exact match on specific fields. Used when data is high-quality and contains unique IDs.
- Probabilistic: Calculates a weights-based score (e.g., Fellegi-Sunter) to determine the likelihood of a match. Handles typos, nicknames, or missing data.
- Machine Learning: Uses training data to learn which attribute combinations constitute a match. Useful for complex datasets where rules are too difficult to manually define.
Applications of ER
- Customer 360: Creating a unified view of a customer who may have different profiles in a Customer Relationship Management (CRM), an email marketing tool, and a billing system.
- Fraud Detection: Identifying individuals opening multiple accounts under slightly different names or addresses.
- Healthcare: Linking patient records across different hospitals to ensure a complete medical history.
Common ER Challenges
- Scalability: As a dataset grows by N, the number of possible comparisons grows at O(N2), making blocking strategies essential.
- Data Quality: Variations like Jon Doe vs. Jonathan Doe or 123 Main St vs. 123 Main Street, Apt 4.
Additional Acronyms for ER
- ER - Entity-Relationship
