Categories: NLP
Tags:

Understanding TF-IDF (Term Frequency-Inverse Document Frequency)

from sklearn.feature_extraction.text import TfidfVectorizer

texts = ["NLP is fun", "I love learning NLP"]
vectorizer = TfidfVectorizer()
tfidf_matrix = vectorizer.fit_transform(texts)

print(vectorizer.get_feature_names_out())  # Output: ['fun', 'is','learning', 'love', 'nlp']
print(tfidf_matrix.toarray())  # Numerical representation
--------------
Output
['fun' 'is' 'learning' 'love' 'nlp']
[[0.6316672  0.6316672  0.         0.         0.44943642]
 [0.         0.         0.6316672  0.6316672  0.44943642]]

The output you’ve provided represents the TF-IDF (Term Frequency-Inverse Document Frequency) transformation of the input text, which converts text into a numerical format suitable for machine learning models.

Breaking Down the Output

1. vectorizer.get_feature_names_out()

  • Result: ['fun', 'is', 'learning', 'love', 'nlp']
  • This is the vocabulary extracted from the input texts:
    • "NLP is fun"
    • "I love learning NLP"
  • Each word appears as a column in the transformed numerical matrix.

2. tfidf_matrix.toarray()

  • Result:
  [[0.6316672  0.6316672  0.         0.         0.44943642]
   [0.         0.         0.6316672  0.6316672  0.44943642]]
  • Rows:
    • Each row corresponds to a document.
    • Row 1: "NLP is fun"
    • Row 2: "I love learning NLP"
  • Columns:
    • Each column corresponds to a word from the vocabulary (['fun', 'is', 'learning', 'love', 'nlp']).
  • Values:
    • Each value represents the TF-IDF score for the word in the document.

How the TF-IDF Scores Are Computed

  1. TF (Term Frequency):
  • Counts how often a word appears in a document.
  • Example:
    • In "NLP is fun", the word NLP appears once, so its frequency is 1.
  1. IDF (Inverse Document Frequency):
  • Measures how unique a word is across all documents.
  • Words that appear in many documents get lower scores (e.g., “is”).
  1. TF-IDF Score:
  • Combines TF and IDF:

Interpreting the Matrix

  1. Row 1: Document "NLP is fun"
  • TF-IDF Scores:
    • fun: 0.6316672 (high importance in this document)
    • is: 0.6316672 (moderate importance, less unique)
    • nlp: 0.44943642 (shared across documents, so less unique)
    • learning and love: 0.0 (don’t appear in this document).
  1. Row 2: Document "I love learning NLP"
  • TF-IDF Scores:
    • learning: 0.6316672 (important in this document)
    • love: 0.6316672 (important in this document)
    • nlp: 0.44943642 (shared across documents, so less unique)
    • fun and is: 0.0 (don’t appear in this document).

Conclusion

  • High TF-IDF Scores:
  • Words unique to a document get high scores, indicating they are more important for that document.
  • Low TF-IDF Scores:
  • Words appearing in many documents or commonly used words like “is” get lower scores.

This process helps identify the most important words in each document, which is useful for tasks like text classification, clustering, and summarization.