Bag-of-Words vs TF-IDF: Building a Tweet Sentiment Classifier

A practical walkthrough of building a three-class sentiment classifier for tweets, comparing Bag-of-Words and TF-IDF vectorisation and digging into where the model gets confused.

The task: classifying tweets as positive, negative or neutral

Given roughly 27,000 tweets, each labelled as positive, negative or neutral, the goal is to build a model that predicts sentiment from text alone. The labels are reasonably balanced: about 40% neutral, 31% positive and 28% negative, and tweet length averages around 68 characters, ranging from just 3 characters up to the old 140-character limit. Before any modelling, rows with missing text are dropped and the raw label distribution and length distribution are checked, because both directly affect how much signal a short-text classifier has to work with.

Preprocessing: from raw tweet to clean tokens

Tweets are noisy: mixed case, URLs, punctuation, misspellings and slang. The preprocessing pipeline lowercases everything, strips URLs, removes anything that isn't a letter or number, tokenises the remaining text, drops English stopwords and very short tokens, and finally applies a Snowball stemmer so that 'happy', 'happiness' and 'happier' all collapse to a common root like 'happi'. A tweet like "Sooo SAD I will miss you here in San Diego!!!" becomes sooo sad miss san diego — shorter, lowercase, and stripped of anything that doesn't carry sentiment information.

Vectorising text: Bag-of-Words vs TF-IDF

Machine learning models need numbers, not words, so the cleaned text is converted into vectors using two different approaches, each capped at the 5,000 most frequent tokens for a fair comparison:

Bag-of-Words (BoW), via CountVectorizer, simply counts how many times each vocabulary word appears in a tweet. It's fast, easy to interpret, but treats every word as equally important regardless of how common it is across the whole dataset.
TF-IDF (Term Frequency – Inverse Document Frequency) additionally down-weights words that appear in almost every document (which carry little discriminating power) and up-weights words that are frequent in a specific tweet but rare across the corpus.

Four classifiers — Logistic Regression, Decision Tree, Random Forest and Gradient Boosting — are trained on the BoW vectors first. Random Forest and Logistic Regression come out roughly tied as the strongest (around 69% accuracy, 0.688 weighted F1), comfortably ahead of the Decision Tree (62% accuracy) and Gradient Boosting (66%).

Does TF-IDF actually help?

Training the same best-performing model (Random Forest) on TF-IDF vectors instead of raw counts nudges accuracy from 68.9% to 69.8% and weighted F1 from 0.6885 to 0.6974 — a modest but real improvement. The per-class breakdown shows the classifier does best on the positive class (0.74 F1) and worst on negative (0.68 F1), with the biggest confusion happening between negative and neutral tweets rather than between positive and negative — sentiment classifiers most often fail at telling mild negativity from neutrality, not at confusing opposite extremes.

Comparing the top 20 most important words under each vectorisation reveals significant overlap — words like love, thank, good, miss, happy, sad dominate both — but the relative ranking shifts slightly, reflecting TF-IDF's tendency to boost words that are more distinctive across the corpus.

Where the model goes wrong

Analysing the roughly 30% of test tweets the model gets wrong is often more informative than the accuracy number itself. Sarcasm ('lol that`s great For you....' labelled negative but predicted neutral), understatement, and tweets that mix positive and negative sentiment within the same short text are the recurring failure patterns. Errors also tend to cluster around neutral-negative and neutral-positive boundaries rather than negative-positive flips, which makes sense: distinguishing 'flat, unemotional' text from 'mildly emotional' text is a genuinely harder linguistic judgement than distinguishing clearly opposite emotions.

Three concrete directions for improvement fall out of this error analysis: better handling of internet-specific text (elongated words like 'sooo', emoji, slang normalisation), moving beyond single words to bigrams and trigrams or word embeddings that capture some context, and addressing the negative/neutral boundary specifically rather than optimising for overall accuracy.

Frequently Asked Questions

Why does TF-IDF usually beat plain Bag-of-Words for text classification?

Bag-of-Words treats a word that appears in 90% of documents the same as a word that appears in only 2% of documents, even though the rare word is far more useful for distinguishing between classes. TF-IDF explicitly discounts very common words and boosts distinctive ones, which typically gives the downstream classifier cleaner signal to work with — though the improvement is often modest rather than dramatic, as seen here.

Why is negative-vs-neutral harder to classify than negative-vs-positive?

Positive and negative sentiment tend to use clearly opposite vocabulary (love vs hate, happy vs sad), which linear and tree-based models pick up easily from word frequency alone. Neutral text, by contrast, is defined by the *absence* of strong emotional language rather than by a distinctive vocabulary of its own, and mildly negative tweets can look lexically similar to flat neutral ones, which is exactly where this classifier makes most of its mistakes.

Would a modern transformer model do much better on this task?

Yes, typically. Models like BERT capture word order and context (so 'not good' is understood differently from 'good'), which bag-of-words and TF-IDF approaches cannot represent at all since they discard word order entirely. For short, informal text like tweets, contextual embeddings usually provide a meaningful accuracy boost over classic vectorisation, at the cost of much higher computational requirements.