Document Extraction Confidence Routing and Review Queue Lab (2D)

Invoices are scanned, read by OCR and labelled by a layout-aware model; a confidence threshold decides which go straight through and which wait for a person.
Switch to 3D
1. Field confidence: correct vs wrong fields
2. Risk vs coverage as tau varies
3. Cost per document vs threshold
4. Calibration (reliability diagram)
5. Live review queue vs Erlang C

The science

What is modelled

An intelligent document processing (IDP) pipeline turns scanned invoices into structured records. Each invoice has ten fields (invoice number, dates, supplier, VAT ID, amounts, currency, IBAN, PO number). OCR reads the characters; a layout-aware model (in the spirit of LayoutLM-type models that combine text, position and image) decides which field each text region is; a confidence threshold decides whether the whole document is accepted automatically or sent to a person.

1. Field labelling as a Gaussian decision

z_j = d * [j = true class] + e_j,   e_j ~ N(0, 1),  j = 1..K
P(correct label) = integral phi(x) Phi(x + d)^(K-1) dx
P(class j | z) = softmax(d z)_j          (exact Bayes posterior for this model)
reported confidence = max_j softmax(d z / T)_j

d is the separation between the true label and the K - 1 alternatives (signal-detection d'). With T = 1 the reported confidence is the true posterior, i.e. perfectly calibrated; T < 1 makes the model overconfident, T > 1 underconfident. The "layout difficulty" slider lowers the effective separation, d_eff = d (1 - 0.6 difficulty): an illustrative mapping, not a fitted law.

2. OCR errors and OCR confidence

each character is wrong with probability CER (independent)  ->  P(field exact) = (1 - CER)^L
OCR score s ~ N(d_ocr, 1) if right, N(0, 1) if wrong
P(right | s) = 1 / (1 + r exp(-s d_ocr + d_ocr^2 / 2)),  r = CER / (1 - CER)
field OCR confidence = product of P(right | s) over its L characters

A calibrated posterior has the property that its average equals the true probability, so the mean OCR confidence of a field must equal (1 - CER)L: one of the checks.

3. Routing

field score = label confidence x OCR confidence
document straight-through  <=>  every field score >= tau
cost per document = P(review) * review_time * wage + P(wrong document leaves the system) * error_cost

Raising tau sends more documents to people (higher cost, longer queues) but lets fewer wrong documents through silently. Graph 2 is the risk-coverage curve; graph 3 finds the cheapest tau. Reviewers are assumed to miss 1 % of wrong documents.

4. The human review queue (M/M/c)

lambda = documents per hour * P(review),  mu = 60 / review minutes,  rho = lambda / (c mu)
P(wait) = Erlang C(c, a = lambda / mu),   W_q = P(wait) / (c mu - lambda),   L_q = lambda W_q   (Little's law)

If rho ≥ 1 there is no steady state: the backlog grows by about (rho - 1) hours of work per hour. The live stream in the scene is an event simulation of the same queue; graph 5 compares it with the Erlang C mean.

Honest simplifications

Independent checks (bottom of the controls)

2D view adds: the share of documents that have to wait, event simulation vs the Erlang C probability. Monte Carlo label accuracy vs the closed-form integral; Monte Carlo OCR exact-match rate vs (1 - CER)L; calibration of the OCR confidence; an event-by-event simulation of the review queue vs the Erlang C formula and the utilisation rho (or, when overloaded, the backlog growth rate vs rho - 1).

Pipeline icons rendered with Cycles in Blender from the 3D hall model; drawn on a 2D canvas. Same model as the 3D view.
Made with 🟠 Blender