What does term frequency-inverse document frequency (TF-IDF) do?

The term frequency-inverse document frequency (TF-IDF) model plays a crucial role in text analysis, specifically in information retrieval and text mining. Its primary function is to quantify the importance of a word within a particular document relative to its prevalence across a broader collection of documents.

TF-IDF consists of two components:

1. Term Frequency (TF): This portion measures how frequently a term appears in a document. The more times a word occurs in the document, the higher its term frequency score.

2. Inverse Document Frequency (IDF): This factor helps adjust the term frequency based on how common or rare a term is across all documents in the corpus. If a term appears in many documents, its IDF score will be lower, indicating it is less unique and thus less indicative of the document's content. In contrast, terms that are rare (and appear in fewer documents) will have a higher IDF score, suggesting they are more significant in understanding the document's context.

By combining these two measures, TF-IDF helps identify words that are not only relevant within the specific document but also possess a degree of uniqueness when compared with the entire set of documents. This makes it particularly useful for tasks such as keyword extraction