EN 
Українська
Polski
Español
Deutsch
日本語
Italiano
Português (BR)
Français
العربية
Nederlands
中文
한국어
हिन्दी
Türkçe
Tiếng Việt
Русский
Bahasa Indonesia
فارسی
বাংলা
ไทย
Filipino
Kiswahili
اردو
Bahasa Melayu
தமிழ்
עברית
Ελληνικά
ਪੰਜਾਬੀ
አማርኛ
မြန်မာဘာသာ
ខ្មែរ
ქართული
नेपाली
Монгол хэл
Հայերեն
සිංහල
ລາວ
Čeština
Română
Magyar
Български
Svenska
Dansk
Suomi
Norsk
Slovenčina
Hrvatski🛡️ Content Moderation Classifier — Toxicity Scoring Live
Watch a text-toxicity classifier score a live stream of synthetic social posts token by token, highlighting exactly which words drove each moderation decision and threshold.
About the Content Moderation Classifier
Real trust-and-safety pipelines score every piece of user-generated content against a text classifier before it reaches an automated approve/flag/remove decision, usually a neural network trained on millions of labeled examples. This simulation builds a small but completely genuine version of that same scoring pipeline: a synthetic stream of social posts is generated from templates spanning calm, mixed and hostile tones, each post is tokenized into lowercase words, every token is looked up in a fixed vocabulary of learned toxicity weights, and the weighted sum (scaled by a sensitivity slider) is passed through a logistic sigmoid to produce a 0–1 toxicity score — exactly the same weighted-sum-plus-sigmoid math that sits at the output layer of real linear and neural text classifiers.
The top panel renders token-level attribution: each word in the current post is highlighted red or green in proportion to how much it pushed the score toward removal or approval, so you can see precisely which words the model is reacting to rather than trusting an opaque number. The bottom panel streams every scored post's toxicity score over time as a bar chart against two live-adjustable threshold lines — drag the flag and remove thresholds to see the trade-off between under-moderation and over-moderation play out in real time, and switch the scenario mix between Calm, Mixed and Hostile to change how often toxic language actually appears in the stream.
Frequently Asked Questions
How does this classifier actually score a post?
Every incoming synthetic post is tokenized by splitting on whitespace and punctuation into lowercase word tokens. Each token is looked up in a fixed vocabulary of learned toxicity weights — hostile words like "idiot" or "threaten" carry a positive weight, supportive words like "thank" or "kind" carry a negative weight, and unknown tokens contribute zero, exactly like a real bag-of-words model would. The weights (scaled by the sensitivity slider) are summed, a fixed bias is added, and the total is passed through a logistic sigmoid to produce a toxicity score between 0 and 1. This is a genuine linear/logistic scoring model, not a decorative animation.
What is token attribution and why does the top panel highlight individual words?
Token attribution means showing how much each individual token contributed to the final score, rather than just presenting the score as a black box. This simulation computes each token's contribution as its vocabulary weight multiplied by the sensitivity setting, then colors that token's chip red (pushing toward toxic) or green (pushing toward approved) with an intensity proportional to the size of its contribution. This mirrors how real explainability techniques for text classifiers, such as LIME or SHAP-style attributions or simple linear-model coefficient inspection, let a human moderator see which specific words drove an automated decision instead of just trusting a single opaque number.
What do the flag and remove thresholds control?
The toxicity score is a continuous number between 0 and 1, but moderation systems need discrete actions. This simulation compares the score against two live-adjustable thresholds: any post scoring below the flag threshold is approved automatically, anything at or above the flag threshold but below the remove threshold is flagged for human review, and anything at or above the remove threshold is removed automatically. Dragging either slider immediately reclassifies future posts in the stream and shifts the threshold lines on the live chart, letting you see the real trade-off between under-moderation (thresholds too high) and over-moderation (thresholds too low).
Why use a logistic sigmoid instead of just the raw weighted sum?
A raw weighted sum of token weights is unbounded — a long, heavily loaded post could sum to a very large positive or negative number. The logistic sigmoid function, score = 1 / (1 + e^-x), squashes any real-valued input into a bounded 0–1 range that behaves like a probability, so scores stay comparable across posts of very different lengths and severities. This is exactly the final layer used in real logistic-regression text classifiers and in the output layer of many neural toxicity-detection models, converting an internal "log-odds" score into an interpretable probability-like number a threshold can act on.
How realistic is this compared to production moderation systems like Perspective API?
Google Jigsaw's Perspective API, and most production toxicity classifiers, are trained on millions of human-labeled real comments using deep neural networks (often transformer-based) that capture context, negation and sarcasm far beyond a bag-of-words sum. This simulation uses a small, hand-assigned vocabulary of a few dozen tokens and a single linear-plus-sigmoid layer rather than a trained network, so it should be read as a transparent, inspectable mechanism demonstration of the scoring math those systems ultimately reduce to, not a production-grade moderation tool.
What are the limitations of a weighted-vocabulary approach to toxicity detection?
Because the score only sums fixed per-token weights, this approach cannot detect negation ("not stupid" scores the same direction as "stupid"), sarcasm, coded harassment that avoids listed words, or context that changes meaning across a whole sentence. It can also misfire on reclaimed or quoted language and on any token outside its fixed vocabulary, which simply contributes zero regardless of how it's actually used. Real systems mitigate this with contextual language models, but every neural classifier still ultimately produces a single score that gets thresholded exactly the way this simulation demonstrates.
Each synthetic post is tokenized, every token's learned toxicity weight is scaled by sensitivity and summed with a fixed bias, and a logistic sigmoid converts that sum into a 0–1 toxicity score — which is then thresholded live into approved / flagged / removed while the top panel renders per-token attribution.
3D · Three.js / WebGL renderer · 60 FPS target · runs fully client-side, no install