To classify tweet sentiment, text first has to become numbers. Bag-of-Words (BoW) simply counts how many times each term appears in a tweet. TF-IDF (term frequency × inverse document frequency) rescales that count by how rare the term is across the whole corpus, so words that show up in almost every tweet — like "the", "amp" or "http" — get pushed toward zero, while distinctive sentiment words like "love" or "worst" keep their weight.
idf(term) = ln(N / df), where N is the corpus size and df is the number of tweets containing that term.tf-idf = tf × idf — a term that appears often in a tweet (tf) but rarely across the corpus (low df, high idf) gets the biggest weight.On short, noisy text like tweets, TF-IDF's down-weighting of boilerplate tokens (retweet markers, links, "amp" for &) is often what pushes a classifier's accuracy up over plain Bag-of-Words — even before any smarter model swap.
A 3D bar chart of term weights across negative, neutral and positive tweets, comparing raw Bag-of-Words counts against TF-IDF weighting so you can see exactly which words a TF-IDF classifier learns to trust.
Bag-of-Words treats every occurrence equally, so filler words like "the", "amp" and "http" tower over sentiment-bearing words. TF-IDF multiplies each count by an inverse-document-frequency term, shrinking ubiquitous words and preserving rare, class-distinctive ones.
Toggle between Bag-of-Words and TF-IDF, highlight a sentiment class, and drag the document-frequency and corpus-size sliders to see how the idf formula reshapes the bar for "the" in real time.
The "signal / stopword ratio" stat approximates why TF-IDF often beats plain Bag-of-Words on short, noisy text: it boosts the ratio of sentiment signal to boilerplate noise without any extra model complexity.