Bag of Words vs TF-IDF: How Machines Turn Sentences into Numbers
An explainer on the two classic text-vectorisation methods behind spam filters and sentiment classifiers, showing exactly why weighting words by rarity usually beats simply counting them.
The problem: models need numbers, not words
Every classic machine learning classifier — logistic regression, random forests, gradient boosting — expects a fixed-length vector of numbers as input. Raw text is neither fixed-length nor numeric: a tweet might be eight words long or forty, and 'good' carries no arithmetic meaning a model can multiply by a weight. Before any classifier can be trained to detect sentiment, spam, or topic, the text has to be converted into vectors through a process called vectorisation. The two most widely used classic approaches are the Bag of Words model and TF-IDF (Term Frequency-Inverse Document Frequency), and the choice between them has a measurable effect on how well the downstream classifier performs.
Bag of Words: just count the words
Bag of Words builds a vocabulary from every distinct word across the training corpus, then represents each document as a vector recording how many times each vocabulary word appears in it — word order and grammar are discarded entirely, hence 'bag'. In practice the vocabulary is usually capped (limiting it to, say, the 5,000 most frequent terms) to keep the resulting vectors manageable, since an uncapped vocabulary across thousands of documents can run into the tens of thousands of dimensions, most of which would be zero for any given short document. The appeal of Bag of Words is its simplicity and speed: it's a straightforward frequency count with no additional statistics to compute. Its weakness is that it treats every word as equally important — 'the' and 'awful' get weighted the same way, even though 'the' appears in nearly every document and carries essentially no information about sentiment, while 'awful' is highly diagnostic.
TF-IDF: weighting words by how distinctive they are
TF-IDF fixes that blind spot by multiplying two quantities for each word in each document. Term Frequency (TF) is how often the word appears in that specific document (similar to Bag of Words). Inverse Document Frequency (IDF) is a corpus-wide statistic that grows larger the rarer a word is across all documents — a word that appears in nearly every document gets an IDF close to zero, while a word that appears in only a handful of documents gets a large IDF. Multiplying the two together means a word only receives a high TF-IDF score when it appears frequently in a specific document but rarely everywhere else — exactly the profile of a word that's actually diagnostic of that document's content, like 'refund' in a complaint or 'brilliant' in a positive review. Common connector words like 'and', 'the', and 'is' are automatically suppressed because their document frequency is so high, without needing a hand-built stop-word list to do all the work.
Preprocessing before vectorisation
Both methods perform better after the raw text is cleaned. A typical pipeline lower-cases everything so 'Good' and 'good' are treated as the same token, strips URLs and special characters that add noise without meaning, tokenises the text into individual words, removes stop words (common function words that carry little topical information), and applies stemming to collapse related word forms — 'running', 'runs', and 'ran' might all be reduced to the stem 'run' — so the vocabulary doesn't fragment into near-duplicate entries that dilute each other's counts. Skipping this preprocessing step is one of the most common reasons a first-pass text classifier underperforms, since the vocabulary balloons with noisy variants and the informative signal gets spread too thin.
Which one actually wins, and why
In head-to-head comparisons on classification tasks like sentiment labelling, TF-IDF vectors typically feed into slightly more accurate classifiers than raw Bag of Words counts, because down-weighting ubiquitous words lets the model's learned coefficients focus more of their signal on words that genuinely discriminate between classes. The improvement isn't dramatic — both methods are 'bag of words' in the sense that they ignore word order — but it's consistent enough that TF-IDF is the more common default in production text-classification pipelines. Where Bag of Words still holds an edge is interpretability of raw counts and marginally faster computation, since it skips the corpus-wide document-frequency pass that TF-IDF requires.
Where both methods eventually hit a wall
Because neither Bag of Words nor TF-IDF captures word order or semantic relationships, both treat 'not good' and 'good' as sharing the word 'good' with no signal that negation flipped the meaning, and neither recognises that 'excellent' and 'outstanding' mean similar things — each gets its own independent vocabulary slot with zero relationship to the other. This is the motivation for word embeddings (which place semantically similar words near each other in a dense vector space) and, further still, for sequence models like LSTMs and transformer-based models such as BERT, which read text in order and can represent context-dependent meaning. For many practical classification tasks with reasonably sized labelled datasets, though, a well-tuned TF-IDF plus logistic regression or gradient boosting pipeline remains a fast, surprisingly strong baseline that's worth building before reaching for anything heavier.
Frequently Asked Questions
Is TF-IDF always better than Bag of Words?
TF-IDF tends to produce slightly better classification accuracy in most benchmarks because it down-weights uninformative common words automatically. But the gap is often small, and Bag of Words remains useful when simplicity, speed, or direct interpretability of raw word counts matters more than squeezing out the last few points of accuracy.
Do I still need to remove stop words if I use TF-IDF?
TF-IDF already suppresses very common words through the inverse-document-frequency term, so it's somewhat more forgiving of leftover stop words than Bag of Words. Removing them explicitly beforehand still tends to help by shrinking the vocabulary and speeding up training, so most pipelines do both.
Why does the neutral sentiment class tend to be the hardest to classify?
Neutral text often contains a mix of mildly positive and mildly negative words, or wording that is genuinely ambiguous, which makes it sit closer to the decision boundary between positive and negative than either extreme does. Both Bag of Words and TF-IDF struggle here because they can't capture the subtle contextual cues a human reader uses to judge tone.
Can Bag of Words or TF-IDF handle sarcasm or negation?
Not well. Because both methods discard word order, a phrase like 'not bad' shares its main content word with 'bad' and gets no signal that the negation reverses the sentiment. Extending the vocabulary to include word pairs (bigrams) captures some of this, but robust handling of negation and sarcasm generally requires sequence-aware models.