An NLP pipeline that reads a restaurant menu, infers hidden ingredients, and flags dishes against vegan, gluten-free, halal, kosher, and allergen profiles
Before any dietary reasoning can happen, the AI needs the menu as text. Most restaurant menus exist as photographs, scanned PDFs, or loosely structured web pages — none of which are natively machine-readable. This first stage converts an image or document into structured dish records the rest of the pipeline can process.
Menu text reaches the AI system through one of two main routes:
• Optical Character Recognition (OCR): a diner photographs a physical menu, or the restaurant supplies a scanned PDF. Modern OCR engines (often vision-capable LLMs rather than classic OCR) segment the image into text blocks, correct for skew and glare, and output raw strings for each dish name, description, and price.
• Structured scraping: for restaurant websites, delivery-app listings (DoorDash, Uber Eats), or POS-integrated digital menus, the system can often pull semi-structured HTML or JSON directly — skipping OCR entirely and yielding cleaner text with fewer transcription errors.
Both paths converge on the same output: a list of raw dish name + description strings, ready for the parsing stage.
Errors introduced at this very first step propagate through the entire pipeline, so ingestion quality is a hard ceiling on final accuracy:
• Stylized fonts and low-contrast printing (white text on light backgrounds, decorative chalk fonts) reduce OCR confidence sharply • Glare, shadows, and curled laminated menus in a photo distort character shapes • Abbreviations and shorthand ("GF", "V", "w/", "chz") need to be expanded correctly or they silently disappear from the ingredient text • Menus in a second language, or with a mix of languages (common in ethnic cuisine menus), require language detection before parsing • Seasonal insert cards and handwritten specials are frequently missed entirely because they are not part of the main scanned document
A dish description that OCR mis-reads or drops silently (for example missing the word "buttermilk" from a fried chicken description) is invisible to every later stage — the ingredient parser simply never sees it. Ingestion errors are the least visible but most consequential failure mode in the whole system.
Once raw text is extracted, it is normalized into a common schema: {section, dish_name, description, price, raw_text}. Prices and currency symbols are stripped from the ingredient stream. Dish names and descriptions are kept separate but linked, since restriction-relevant information often appears in only one of the two (a name like "Chicken Tikka Masala" already implies dairy-based cream sauce even before reading any description text).
This is where an LLM does the heavy lifting: turning "Caesar Salad" into a structured ingredient list that includes not just romaine and parmesan, but anchovy and raw egg — ingredients the menu never actually names. Named-entity recognition finds explicit ingredients; world-knowledge inference fills in the implicit ones a human diner would assume from culinary convention.
The parser performs two distinct jobs on every dish description:
• Explicit extraction (named-entity recognition): identifying ingredient words that are literally printed — "grilled chicken," "goat cheese," "walnuts." This is the easier, higher-confidence half of the task, similar to standard NER in any NLP pipeline.
• Implicit inference (culinary world knowledge): recognizing that certain dish names or preparation styles conventionally include ingredients never printed on the menu. This requires the model to draw on training data about recipes and cuisines rather than the text in front of it — a fundamentally different, riskier kind of reasoning.
Example inferences a well-trained model should make: "Caesar salad" → anchovy and raw/coddled egg in the dressing; "Pad Thai" → fish sauce and often shrimp paste; "Tiramisu" → raw or lightly-cooked egg; "Refried beans" → lard unless marked vegetarian; "Naan" → dairy (yogurt/ghee) and often egg wash.
Because inference is not certainty, each extracted ingredient carries a confidence tag rather than a flat yes/no:
• Explicit — the word appears verbatim in the menu text • Standard-recipe inferred — the ingredient is part of the dish's near-universal traditional recipe (anchovy in Caesar dressing) but restaurants do sometimes substitute or omit it • Regional/variant inferred — common in some versions of the dish but far from universal (e.g., some pad thai recipes are tofu-only with no fish sauce) • Preparation-risk inferred — not an ingredient in the dish itself but a likely kitchen practice (shared fryer oil, shared cutting boards, butter finish on a "vegetable" side)
This confidence tier is what later feeds the yellow "ask the server" flag rather than a blanket red or green — the system is explicitly modeling its own uncertainty instead of guessing silently.
The Menu Description Detail slider in this simulation models a real effect: the sparser a menu's wording, the more the parser must lean on inference rather than explicit text — and the more confidence tiers shift from "explicit" toward "inferred," directly increasing how many dishes need a human check.
An LLM's inference quality is only as good as its exposure to a given cuisine's conventions in training data. Well-documented Western dishes (Caesar salad, carbonara, French onion soup) are inferred reliably because thousands of recipes and food-safety articles describe their standard ingredients online. Regional, home-style, or less-documented cuisines are inferred far less reliably — increasing the risk of silent false negatives (an unlisted allergen the model simply never flags) precisely for the diners who may have the least other information available to them.
With a structured ingredient list in hand, the system compares every tag — explicit or inferred — against the user's stated dietary profile: vegan, gluten-free, halal, kosher, or one or more specific allergens. Each ingredient-to-restriction comparison inherits the confidence level assigned during parsing, and low-confidence matches are what later become "ask the server" flags rather than automatic decisions.
Each restriction profile is encoded as a rule set, not a single keyword list:
• Vegan: excludes any animal-derived ingredient (meat, dairy, eggs, honey, gelatin) and animal-derived processing aids (lard, fish sauce, Worcestershire sauce, some refined sugars) • Gluten-free: excludes wheat, barley, rye, and their derivatives (soy sauce is often wheat-based; some spice blends use flour as an anti-caking agent) • Halal: excludes pork and its derivatives, alcohol as an ingredient, and any meat not prepared per halal slaughter method — the AI can flag the ingredient conflict but generally cannot verify slaughter method from a menu alone • Kosher: excludes pork and shellfish, forbids mixing meat and dairy in the same dish, and requires certified preparation — again largely unverifiable from text alone • Allergens (tree nut, peanut, shellfish, sesame, etc.): excludes the specific allergen and its common hidden carriers (almond milk in a "dairy-free" dessert, sesame oil as a finishing oil)
Each parsed ingredient tag is checked against the active profile's exclusion rules, producing a per-ingredient verdict of conflict, no-conflict, or needs-verification.
The matching stage does not collapse everything into a binary safe/unsafe — it preserves and propagates three tiers:
1. Explicit match — the conflicting (or clearing) ingredient was directly stated on the menu. Highest confidence in either direction.
2. Inferred match — the ingredient was not printed but confidently inferred from dish convention (anchovies in Caesar dressing). Correct most of the time, but restaurant-to-restaurant variation means it is not certain.
3. Ambiguous / needs-verification — the model cannot determine with reasonable confidence whether a conflict exists (e.g., "made with our house dressing" with no further detail, or a described "vegetable stock" that may or may not actually be vegetable-only in practice).
This three-tier structure is what allows the next stage to separate a hard "no" from a "probably not, but ask" — a distinction that matters enormously for someone managing a medical condition versus someone following a general preference.
Crucially, the underlying ingredient evidence does not change with the user's strictness setting — only how conservatively it is interpreted does. A "preference" vegan user might accept a yellow-tier inferred risk as good enough to order confidently. A user with a life-threatening tree-nut allergy needs the exact same inferred risk treated as an automatic red flag requiring verification, because for them the cost of a false "safe" is categorically higher than the cost of a false alarm.
This is why the matching stage keeps confidence tiers separate from the final flag color — the flagging decision in the next stage is a policy applied on top of the evidence, not baked into the evidence itself.
The flagging stage applies the user's strictness setting as a policy on top of the matching evidence, producing one clear signal per dish: green (safe to order), yellow (ask the server — a hidden-ingredient or cross-contact risk exists), or red (a conflict is confirmed or judged too likely to risk). The same dish can land on a different color for two different users with the same restriction but different stakes.
Green (safe): every ingredient relevant to the active restriction was matched with explicit or high-confidence inferred evidence, and none of it conflicts. Example: a plain steamed vegetable side with a clearly stated olive-oil dressing, for a vegan profile.
Yellow (ask the server): a hidden-ingredient or preparation risk exists at inferred or ambiguous confidence — shared fryer oil that may have cooked breaded meat, a "house dressing" of unknown composition, a stock base that is commonly but not certainly meat-derived, a butter finish mentioned nowhere on the menu. The dish might be fine, but the AI cannot say so with the confidence the user's strictness setting requires.
Red (unsafe): an explicit, high-confidence conflict exists — a named ingredient the restriction directly excludes (bacon in a "vegetarian" club sandwich description, shellfish for a shellfish allergy) or an inferred conflict so standard to the dish that treating it as anything but certain would be reckless (anchovies in Caesar dressing, for a strict vegan or fish allergy).
Turning up strictness does not change what the AI knows about a dish — it lowers the bar for what counts as "risky enough to flag." Concretely, in this simulation:
• Preference-level strictness treats standard-recipe inferred risks as usually acceptable — most inferred-hidden-ingredient dishes stay green or move to yellow only when the risk is substantial. • Strict-avoidance treats most inferred risks as yellow, reserving red for explicit conflicts. • Life-threatening-allergy strictness treats almost any non-trivial hidden-ingredient probability as red or yellow — because for a severe allergy, an inferred 10% chance of exposure is not a rounding error, it is a hospital visit.
This mirrors real clinical guidance: allergists consistently advise that "probably fine" is not an acceptable standard for anaphylaxis-risk allergens.
A confidently wrong green flag is the single most dangerous failure mode in this entire system. An honest yellow ("uncertain — verify") that turns out to have been safe costs the diner a question to the server. A confident green that turns out to be wrong can cost far more. Well-designed dietary AI should be tuned to fail toward yellow, not toward green.
Not every risk is an ingredient in the dish itself. Shared fryer oil, shared grills, shared cutting boards, and shared prep surfaces introduce trace-level cross-contact risk that has nothing to do with the recipe as written — "vegetarian" fries cooked in the same oil as breaded chicken, or a gluten-free pizza crust baked on a floured surface. This risk is almost never disclosed on a menu, and a text-only AI system has no reliable way to detect it at all — it can only apply a general prior ("fried items at non-dedicated kitchens carry elevated cross-contact risk") and flag accordingly.
No matter how well-tuned the pipeline, a text-based AI system is reasoning over a menu description — not the actual dish as prepared today, in this kitchen, by this cook. Hidden ingredients, recipe substitutions, and cross-contact risk routinely fall outside what any menu, however well parsed, discloses. This final stage is about knowing exactly where the system's confidence should stop and a human conversation should start.
A handful of ingredient classes are responsible for most real-world dietary-filter failures, precisely because they are rarely spelled out on a menu:
• Umami/savory bases: Worcestershire sauce and Caesar dressing both commonly contain anchovies; many Asian sauces contain fish sauce or shrimp paste even in ostensibly vegetable dishes • Stocks and broths: a "vegetable soup" or risotto is frequently built on a chicken or beef stock base — the printed vegetables are real, but so is the animal-derived liquid underneath them • Legacy fats: refried beans are traditionally made with lard; some pie crusts and older-recipe pastries use lard or beef tallow rather than butter or shortening • Baked goods: bread, pastry, and pizza dough frequently contain dairy (butter, milk) and egg wash on the crust, none of which is obvious from the name "bread basket" • Shared cooking equipment: a plain order of fries can pick up gluten, dairy, or meat protein residue from a shared fryer, even though potatoes and salt are the only "ingredients"
Every one of these is a case where the printed menu text under-describes the true ingredient list — the exact gap the parsing stage tries, imperfectly, to close.
Large language models can produce fluent, confident-sounding answers that are simply wrong — a well-documented failure mode called hallucination. In a general-knowledge chatbot, a hallucinated fact is an inconvenience. In a dietary-filter context, a hallucinated "this dish is vegan" or "this contains no tree nuts" is a safety claim, and a wrong one can trigger a real allergic reaction or violate a genuinely non-negotiable religious or ethical restriction.
The core asymmetry: an admitted "I don't know, please ask the server" costs the user a small amount of friction. A confidently wrong "safe" costs nothing until it very suddenly costs a great deal. Any dietary-AI system should be evaluated not just on overall accuracy, but specifically on its false-negative rate for its most severe risk category — how often it says "safe" when the dish was not.
For non-negotiable restrictions — anaphylaxis-risk allergies, and religious dietary law that requires verified preparation method rather than just excluded ingredients — no text-parsing system, however accurate, can fully replace a conversation with kitchen staff. Waitstaff and cooks have information no menu text carries: today's actual recipe, substitutions, cross-contact practices, and certification status.
The realistic role for AI here is triage, not final authority: quickly narrowing a 60-item menu down to a handful of clearly safe options and a handful that need a specific, well-informed question ("does the Caesar dressing contain anchovy or egg?" rather than a vague "any allergens?"). Several real products already operate in this space, including vegan-focused restaurant/menu discovery apps like HappyCow, allergy-focused dining apps like Spokin, and a growing set of AI-powered menu-scanning tools built into delivery and restaurant-discovery platforms — all of which, to varying degrees, still point users back toward confirming with the restaurant for serious restrictions.
The safest design pattern for this category of tool is not "trust the AI" but "let the AI do the reading, and let the human do the final check" — exactly the same division of labor recommended for AI use in medicine, law, and other high-stakes domains: AI narrows the search space, a qualified human confirms the specific, consequential decision.
| Product | Indication | Trial Design | Key Result |
|---|---|---|---|
| Vegan | Preference to strict ethical | Hidden animal-derived processing aids: lard, fish sauce, gelatin, honey, Worcestershire, animal-based stock, refined sugar bone char | Moderate risk — mostly inferable from cuisine and dish convention |
| Gluten-Free | Sensitivity to celiac disease | Wheat flour as thickener/anti-caking agent, soy sauce, shared fryer oil and prep surfaces, malt-based ingredients | High risk for celiac-level strictness — cross-contact is rarely disclosed at all |
| Halal / Kosher | Religious dietary law | Slaughter method and certification are unverifiable from menu text; alcohol as a cooking ingredient often unlisted; meat/dairy mixing (kosher) | Highest risk — ingredient matching alone cannot confirm preparation compliance |
| Tree-Nut / Peanut Allergy | Life-threatening anaphylaxis risk | Nut oils and pastes in sauces/desserts, cross-contact from shared equipment, nut-derived thickeners in ethnic cuisines | Highest stakes — even low-probability inferred risk should default to a flag |