Classifying Medical Transcriptions by Specialty with TF-IDF and Logistic Regression

Why unstructured text matters alongside structured clinical data

Most healthcare machine learning demonstrations lean on structured, tabular data — blood pressure readings, lab values, age, BMI — because it's clean and easy to model. But a large share of real clinical information lives in free text: transcribed dictations after a consultation, history-of-present-illness notes, discharge summaries. A model that only handles numbers is blind to everything a clinician wrote in prose. This project processes a public dataset of roughly 5,000 medical transcriptions spanning 40-plus specialties, aiming to classify each document by its medical specialty from the text alone — a proxy task that demonstrates the broader capability of triaging or routing unstructured clinical notes automatically.

Exploring the text before modelling it

Before vectorising anything, the exploratory pass looks at how the specialties are distributed (heavily skewed — a handful of specialties account for a disproportionate share of records, so the top 15 are visualised separately) and how long the transcriptions are (highly variable, from short notes to lengthy multi-paragraph reports, which matters because very long documents may need truncation for some model architectures). Word-frequency analysis and word clouds, generated both for the corpus as a whole and separately per specialty, confirm that each specialty has a recognisably different vocabulary — cardiology text clusters around heart, blood and cardiac terminology, for instance — which is the basic precondition for a text classifier to work at all: if every specialty used the same words, no vectoriser could possibly separate them.

Bigram and trigram analysis (looking at frequent two- and three-word phrases rather than single words) surfaces genuine clinical terminology that single-word frequency analysis misses — phrases like specific procedure names or diagnostic terms that only carry their full meaning as a unit.

Cleaning clinical text and picking a target

Clinical text preprocessing here follows a fairly standard NLP pipeline: lowercase everything, strip non-alphabetic characters, and normalise whitespace. Because 40-plus specialties would leave most classes with far too few examples to learn from, the target is deliberately narrowed to the five most frequent specialties, and any document under 50 characters is dropped as too short to carry a reliable classification signal. This kind of pragmatic scoping — trading breadth for enough data per class — is a common and defensible move in real healthcare NLP projects, better than attempting a 40-way classification with a hopelessly unbalanced, data-starved long tail.

TF-IDF vectorisation and two baseline classifiers

The cleaned text is split 70/15/15 into train, validation and test sets, stratified so each specialty is proportionally represented in every split. A TfidfVectorizer is fit only on the training set — a deliberate discipline to avoid data leakage, since fitting on the full dataset would let information about validation or test documents' vocabulary leak into the training vectoriser — with a 5,000-term vocabulary, unigrams and bigrams, and document-frequency filters (min_df=2, max_df=0.95) that drop both rare typos and near-universal filler words.

Two classifier families are compared on top of these TF-IDF vectors: Logistic Regression (both a default version and one tuned via grid search over regularisation strength and solver) and Multinomial Naive Bayes. Logistic Regression is chosen for its speed on sparse high-dimensional vectors and its interpretability (coefficients directly show which terms push a document toward or away from each specialty). Naive Bayes is included specifically because it is dramatically faster to train — a legitimate advantage when a system needs to be retrained frequently as new labelled documents arrive — even though its independence assumption between words is technically false for real language.

Choosing a model and validating generalisation

Models are compared using weighted F1-score rather than plain accuracy, because it properly accounts for the remaining class imbalance among even the top-5 specialties. The best-performing model on the validation set is selected automatically, evaluated once more on the held-out test set (used exactly once, specifically to get an unbiased read on generalisation), and its confusion matrix is inspected to see which specialties get mixed up — typically the ones with genuinely overlapping clinical vocabulary. The gap between training and validation F1-score is explicitly tracked as an overfitting check: a gap under roughly 0.1 is treated as healthy, while a larger gap signals the model has memorised training-set quirks rather than learning generalisable patterns. Both the trained classifier and its TF-IDF vectoriser are saved together, since a vectoriser fit on different vocabulary would silently produce meaningless features for any new document at inference time.