How Does Sentiment Analysis Work? A Plain-English Guide

Sentiment analysis is one of the most practical applications of Natural Language Processing. From monitoring customer reviews to tracking brand mentions on social media, it powers countless real-world systems. But how does it actually work?

This guide walks through the full history — from simple word lists to transformer models — without assuming you have a machine learning background.

What Is Sentiment Analysis?

Sentiment analysis (also called opinion mining) is the task of automatically identifying the emotional tone of a piece of text — whether it is positive, negative, or neutral.

Examples: - "This product is absolutely fantastic!" → Positive - "The delivery was late and the packaging was damaged." → Negative - "The laptop has a 15-inch screen and weighs 1.8 kg." → Neutral

More advanced systems can detect emotions (joy, anger, fear, surprise) or measure intensity (strongly positive vs. mildly positive).

Approach 1: Rule-Based Sentiment Analysis

The oldest method uses manually curated word lists called lexicons.

How it works:

1. Tokenise the text (split into words)
2. Look up each word in a sentiment dictionary
3. Sum the scores: positive words add up, negative words subtract

# Simplified example
positive_words = {'great', 'excellent', 'love', 'fantastic', 'amazing'}
negative_words = {'terrible', 'horrible', 'hate', 'awful', 'disappointing'}

def rule_based_sentiment(text: str) -> str:
    tokens = text.lower().split()
    score = sum(
        1 if w in positive_words else -1 if w in negative_words else 0
        for w in tokens
    )
    if score > 0: return 'positive'
    if score < 0: return 'negative'
    return 'neutral'

print(rule_based_sentiment("This product is great and amazing"))  # positive
print(rule_based_sentiment("Terrible service, awful experience"))  # negative

Popular lexicons:

• VADER (Valence Aware Dictionary and sEntiment Reasoner) — tuned for social media
• AFINN — simple word list with scores −5 to +5
• SentiWordNet — WordNet + sentiment scores

Strengths and weaknesses:

VADER example (handles emojis and slang):

from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer

analyzer = SentimentIntensityAnalyzer()
scores = analyzer.polarity_scores("This is GREAT! 😊 Not bad at all 👍")
print(scores)
# {'neg': 0.0, 'neu': 0.262, 'pos': 0.738, 'compound': 0.7579}

Approach 2: Machine Learning Classifiers

Traditional ML models learn sentiment patterns from labelled training data.

Pipeline:

Raw text → Preprocessing → Feature extraction → Classifier → Label

1. Text preprocessing

import re
from nltk.corpus import stopwords
from nltk.stem import SnowballStemmer

stop_words = set(stopwords.words('english'))
stemmer = SnowballStemmer('english')

def preprocess(text: str) -> str:
    text = text.lower()
    text = re.sub(r'[^a-z\s]', '', text)   # remove punctuation
    tokens = [stemmer.stem(w) for w in text.split() if w not in stop_words]
    return ' '.join(tokens)

2. Feature extraction: TF-IDF

TF-IDF (Term Frequency–Inverse Document Frequency) converts text into numerical vectors:

from sklearn.feature_extraction.text import TfidfVectorizer

vectorizer = TfidfVectorizer(max_features=10_000, ngram_range=(1, 2))
X_train = vectorizer.fit_transform(train_texts)
X_test = vectorizer.transform(test_texts)

3. Training a Logistic Regression classifier

from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report

clf = LogisticRegression(max_iter=1000)
clf.fit(X_train, y_train)

y_pred = clf.predict(X_test)
print(classification_report(y_test, y_pred, target_names=['negative', 'positive']))

Typical accuracy on balanced datasets: 85–90%.

Common classifiers for sentiment analysis:

• Logistic Regression — fast baseline, often surprisingly strong
• Naive Bayes — efficient on text, good for short texts
• SVM — strong on small/medium datasets
• Random Forest — handles noise well, interpretable

Approach 3: Deep Learning (LSTMs and CNNs)

Before transformers, recurrent neural networks — particularly LSTMs (Long Short-Term Memory) — were state of the art because they process text sequentially and remember context.

import tensorflow as tf
from tensorflow.keras.layers import Embedding, LSTM, Dense, Bidirectional

model = tf.keras.Sequential([
    Embedding(input_dim=20_000, output_dim=128),
    Bidirectional(LSTM(64, return_sequences=True)),
    Bidirectional(LSTM(32)),
    Dense(64, activation='relu'),
    Dense(1, activation='sigmoid'),
])

model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])

Typical accuracy on IMDB or SST-2: 91–94%.

Limitation: LSTMs struggle with very long texts because the sequential pass compresses all information into a single hidden state.

Approach 4: Transformers (BERT, DistilBERT, RoBERTa)

Transformers are now the dominant approach. The key innovation is the attention mechanism — instead of reading left to right, the model weighs every token against every other token simultaneously.

Why transformers win:

• They understand context fully: "not bad" is positive, not negative
• They handle long-range dependencies
• Pre-trained on massive corpora — fine-tuning needs very little labelled data

Using DistilBERT with HuggingFace Transformers

from transformers import pipeline

# Single-inference, no GPU needed
classifier = pipeline(
    'sentiment-analysis',
    model='distilbert-base-uncased-finetuned-sst-2-english',
)

results = classifier([
    "Absolutely loved the product, highly recommend!",
    "Not what I expected. Disappointing quality.",
    "The item arrived on time.",
])

for r in results:
    print(f"{r['label']} ({r['score']:.2%})")
# POSITIVE (99.94%)
# NEGATIVE (99.81%)
# NEGATIVE (81.23%)  ← ambiguous neutral, model leans negative

Fine-tuning on custom data

from transformers import AutoTokenizer, AutoModelForSequenceClassification, Trainer, TrainingArguments
import torch

model_name = 'distilbert-base-uncased'
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name, num_labels=3)

# Tokenise dataset
def tokenize(batch):
    return tokenizer(batch['text'], truncation=True, padding='max_length', max_length=128)

# Training
args = TrainingArguments(
    output_dir='./results',
    per_device_train_batch_size=16,
    num_train_epochs=3,
    evaluation_strategy='epoch',
    save_strategy='epoch',
    load_best_model_at_end=True,
)

trainer = Trainer(
    model=model,
    args=args,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
)
trainer.train()

Fine-tuned BERT models routinely achieve 95–98% accuracy on standard benchmarks.

Common Challenges

1. Negation

"This is not good" — without context, "good" appears positive.

2. Sarcasm

"Oh great, another delay. Super impressed." — lexicon gives positive result, but intent is negative.

3. Domain specificity

The word "sick" is negative in general health contexts but positive in sports/slang ("That trick was sick!").

4. Aspect-based sentiment

"The battery life is poor but the screen is excellent." — overall sentiment is mixed, but you may need to track sentiment per aspect.

# Aspect-based example (simplified)
aspects = {
    'battery': 'negative',   # "battery life is poor"
    'screen': 'positive',    # "screen is excellent"
}

Practical Accuracy Benchmarks

Summary

Try it live

Explore the related model hands-on: open Sentiment Analysis and change its parameters while it runs. Everything stays in your browser.