What does TFIDF stand for
TF-IDF stands for Term Frequency-Inverse Document Frequency. It is a statistical measure used in information retrieval and text mining to evaluate the importance of a term (word or phrase) within a document relative to a collection of documents, typically a corpus.
Here's what each component of TF-IDF represents:
1. **Term Frequency (TF)**: Term Frequency measures how frequently a term occurs in a document. It is calculated by counting the number of times a term appears in a document and normalizing it by the total number of terms in the document. The idea behind TF is that terms with higher frequencies are more important to the document's content.
\[ \text{TF}(t, d) = \frac{\text{Number of occurrences of term } t \text{ in document } d}{\text{Total number of terms in document } d} \]
2. **Inverse Document Frequency (IDF)**: Inverse Document Frequency measures how important a term is across a collection of documents. It is calculated by dividing the total number of documents in the corpus by the number of documents containing the term, and then taking the logarithm of the result. The idea behind IDF is that terms that appear in fewer documents are more discriminative and carry more information.
\[ \text{IDF}(t, D) = \log\left(\frac{\text{Total number of documents in the corpus}}{\text{Number of documents containing term } t}\right) \]
3. **TF-IDF Score**: The TF-IDF score for a term in a document combines the TF and IDF measures. It is obtained by multiplying the term frequency (TF) of the term in the document by the inverse document frequency (IDF) of the term in the corpus.
\[ \text{TF-IDF}(t, d, D) = \text{TF}(t, d) \times \text{IDF}(t, D) \]
TF-IDF assigns higher scores to terms that are frequent within a document (high TF) but rare across the corpus (high IDF), indicating their importance in describing the content of the document. It is widely used in various NLP tasks such as information retrieval, text classification, and document clustering.
