Markdown

IDF

IDF is the Acronym for Inverse Document Frequency

Inverse document frequency is a weight applied to a word so that a rare word counts more than a word that appears in almost every document. The weight is high when the word shows up in few documents, and low when the word shows up in most of them.

The raw count this weight multiplies is a Term Frequency (TF). The product of that count and this weight is Term Frequency – Inverse Document Frequency (TF-IDF).

How the Weight Moves

  • A rare word. It appears in a handful of documents, so the weight is high and those documents stand out for that word.
  • A common word. It appears almost everywhere, so the weight is low and it barely separates one document from another.
  • A word in every document. The weight falls to nothing useful. The word cannot mark a subset of the set.

Rare Word and Common Word

ComparedRare WordCommon Word
How many documents contain itFewMost, or all
The weightHighLow
Effect on a searchThe few documents that contain it riseIt does little to pick one document over another

Articles Tagged IDF

View Additional Articles Tagged IDF