What Machine Learning Reveals About Online Course Pricing and Popularity

Why online course marketplaces are a good testbed for applied ML

Online education marketplaces sit at an unusually convenient intersection for data scientists: every listing is a structured bundle of numeric, categorical and text features (duration, price, instructor rating, number of enrolled students, language of instruction, subject category, course description) attached to two outcomes that businesses actually care about — how much a course sells for, and how popular it becomes. That makes the domain a natural case study for two of the most common applied-ML tasks: regression (predict a continuous price) and multi-class classification (predict a discrete popularity tier). A market-analysis project built around a scraped catalogue of courses from a national online-education market illustrates the workflow end to end, from raw HTML pages to a trained model bundle that can be queried for new listings.

From scraped listings to a model-ready table

The raw material is unstructured: course pages fetched with requests and parsed with BeautifulSoup, yielding free-text titles and descriptions, inconsistent duration formats ('20 годин', '3 місяці', '2 weeks'), and category labels that differ slightly across platforms. Turning that into a model-ready table requires the usual data-engineering steps — normalising duration into a single numeric unit (hours), extracting a clean price figure from currency-formatted strings, one-hot or target-encoding the category and language fields, and running lightweight NLP (via nltk) over descriptions to derive features like description length, keyword presence ('advanced', 'beginner', 'certificate') and basic sentiment or tone signals from any available reviews. Only after this cleaning stage does the dataset become usable for supervised learning — a reminder that in most real ML projects the modelling step is a small fraction of the total effort compared to data preparation.

Predicting price with gradient boosting

Course price turns out to be a genuinely learnable quantity rather than noise. A gradient-boosted regression model (XGBoost) trained on features such as subject category, duration, instructor rating, platform, and language of instruction achieved an R² above 0.8 on held-out data — meaning the model explained more than 80% of the variance in listed price using only structural attributes of the course, without ever reading the marketing copy. This is a useful result for a marketplace operator: it implies that price is driven largely by observable, structural factors (what the course teaches, how long it runs, who teaches it) rather than by idiosyncratic seller pricing whims. Practically, a model like this can flag listings that are priced far outside what their features predict — either underpriced courses leaving money on the table, or overpriced ones likely to underperform on conversion.

Feature-importance analysis on this kind of model typically surfaces subject category as the dominant driver, followed by duration and instructor/platform reputation signals — which aligns with a broader pattern seen across many e-learning markets: technical and professional-skills content commands a premium over general-interest content, and that premium is large enough to be predicted well before a single sale happens.

Popularity as a classification problem

Where price is naturally continuous, popularity is better modelled as an ordinal or multi-class classification problem, since 'exactly how many students enrolled' is a noisier, more skewed target than a coarse tier. Binning enrolment counts (or a composite popularity score combining enrolments, ratings and review volume) into four tiers — for example low, moderate, high and top-seller — and training a classifier to predict tier membership from the same structural features achieved better than 85% accuracy in this project's setup. That accuracy figure needs the usual grain of salt: if one tier (say 'low popularity') dominates the dataset, a naive baseline that always predicts the majority class could already score well, so any credible version of this analysis needs to be checked against class-balanced metrics like macro-F1 rather than raw accuracy alone. Still, the qualitative finding is consistent with intuition and worth taking seriously: shorter courses (roughly 10–30 hours) and courses taught in the learner's native language systematically classify into the higher-popularity tiers, suggesting that accessibility and time-commitment are at least as important to demand as subject matter.

What the models imply about market structure

Combining the two models gives a more interesting picture than either alone. A course can be expensive and popular (premium professional certifications), cheap and popular (short practical skills content), expensive and unpopular (niche, long-form expert courses with thin demand), or cheap and unpopular (low-quality or poorly marketed listings). Mapping predicted price against predicted popularity tier for every course in the catalogue effectively produces a two-dimensional market map without any explicit clustering step — it's a direct byproduct of having two independently validated predictive models. In the underlying dataset, IT and programming courses cluster toward the high-price end, native-language instruction correlates with the popular end, and the sweet spot for volume — courses that are both moderately priced and land in the top popularity tiers — sits around the 20–40 hour duration range.

Limitations worth being explicit about

Any model trained on scraped marketplace data inherits the marketplace's own biases: courses that are easier to scrape (larger, better-established platforms) are overrepresented, self-reported enrolment and rating figures can be inflated, and the R²/accuracy figures above describe fit to a specific historical snapshot rather than a guarantee of future predictive power, since course pricing and demand shift with new competitors and content trends. A production version of this pipeline would need periodic retraining, out-of-time validation (training on older listings, testing on newer ones), and ideally A/B-style validation against actual conversion data rather than static ratings alone.