An advanced detection technology used in Data Loss Prevention (DLP) to protect unstructured data. While other methods focus on specific strings or database records, IDM identifies sensitive information by “fingerprinting” entire documents or specific sections of text. It allows organizations to protect intellectual property, such as legal contracts, design documents, or proprietary source code, even if the content has been partially modified or reformatted.
Core Components
- Document Crawling: The process where the DLP system scans a designated gold standard repository (e.g., a secure file share or SharePoint site) to ingest the files that require protection.
- Signature Generation (Fingerprinting): The system creates a mathematical representation (hash) of the document’s content. Unlike a simple file hash, IDM breaks the document into small fragments to detect partial matches.
- Proximity Analysis: A technique used to determine if the sensitive content within a monitored file is clustered in a way that matches the original indexed document, helping to distinguish between a few common phrases and actual data theft.
How It Works
- Collection: The organization identifies a folder containing sensitive unstructured files (PDFs, Word docs, CAD files, etc.).
- Indexing: The IDM engine reads these files and generates a set of unique fingerprints based on the text strings and their relative positions.
- Deployment: This index is distributed to the DLP enforcement points (email gateways, web proxies, or endpoint agents).
- Inspection: When a user attempts to upload or send a file, the system extracts the text from that file in real-time.
- Matching: The extracted text is compared against the index. If the percentage of matching content exceeds a predefined threshold (e.g., a 50% match), the policy is triggered.
DCM vs. EDM vs. IDM
| Feature | Described (DCM) | Exact (EDM) | Indexed (IDM) |
| Detection Logic | Patterns & Math | Specific Database Records | Document Similarity |
| Best For | Generic PII/PCI | Specific Customers/Employees | Proprietary Files/IP |
| Setup Effort | Low (Out-of-the-box) | Medium (Requires Hashing) | High (Requires Indexing) |
| Accuracy | Moderate (Prone to false positives) | Very High | High |
Key Benefits
- Format Agnostic: IDM can often detect sensitive content even if it is moved from a PDF into a plain text file or copied into the body of an email.
- Partial Match Detection: It is highly effective against derivative works, where a user takes an original sensitive document and changes a few sentences to try and bypass security.
- Automated Classification: Reduces the need for manual tagging or labeling of files, as the system learns what is sensitive based on the source location.
