❓ Question Answering System

Understanding Question Answering

Question Answering (QA) is the task of automatically answering questions posed in natural language. QA systems can extract answers from text passages (extractive QA), generate answers (generative QA), or retrieve answers from knowledge bases.

Types of Question Answering

• Extractive QA (Reading Comprehension):
  • Answer is a span of text from the passage
  • Find start and end positions
  • SQuAD dataset popularized this
  • Most common type
• Abstractive QA:
  • Generate answer in own words
  • May paraphrase or summarize
  • More flexible but harder
• Open-Domain QA:
  • Answer from large corpus (e.g., Wikipedia)
  • No specific passage given
  • Retrieve relevant documents first
• Knowledge-Based QA:
  • Query structured knowledge bases
  • Convert to database queries
  • Freebase, Wikidata
• Visual QA:
  • Answer questions about images
  • Multimodal reasoning

How Extractive QA Works

Modern extractive QA (BERT-based) follows this process:

• Step 1 - Input Encoding:
  • Concatenate question + context
  • Tokenize: [CLS] question [SEP] context [SEP]
  • Create token embeddings
• Step 2 - Contextual Encoding:
  • Pass through BERT/transformer
  • Get contextualized representations
  • Each token vector considers full context
• Step 3 - Span Prediction:
  • Predict start position probabilities
  • Predict end position probabilities
  • Answer = tokens from start to end
• Step 4 - Scoring:
  • Score = P(start) × P(end)
  • Return highest scoring span
  • Confidence = max score

Popular QA Models

• BERT for QA:
  • Fine-tuned on SQuAD dataset
  • Two additional output layers (start, end)
  • State-of-the-art on many benchmarks
• RoBERTa, ALBERT, XLNet:
  • BERT variants with improvements
  • Better pre-training procedures
  • Stronger performance
• T5 (Text-to-Text):
  • Treats QA as text generation
  • Flexible framework
• DPR (Dense Passage Retrieval):
  • For open-domain QA
  • Efficient retrieval from millions of documents

Training Datasets

• SQuAD (Stanford Question Answering Dataset):
  • 100K+ questions on Wikipedia articles
  • Extractive answers
  • SQuAD 2.0 includes unanswerable questions
• Natural Questions (Google):
  • Real Google search queries
  • 300K+ questions
  • Long and short answers
• TriviaQA:
  • 650K question-answer pairs
  • Trivia questions
  • Evidence from web and Wikipedia
• HotpotQA:
  • Multi-hop reasoning required
  • Need to combine information from multiple passages
• MS MARCO:
  • Bing search queries
  • 1M+ queries
  • Real-world distribution

Evaluation Metrics

• Exact Match (EM):
  • Predicted answer exactly matches ground truth
  • Very strict metric
• F1 Score:
  • Token-level overlap
  • More forgiving than EM
  • Precision and recall of tokens
• Mean Reciprocal Rank (MRR): For retrieval-based QA
• BLEU/METEOR: For generative QA

Challenges

• Unanswerable Questions:
  • No answer in context
  • Model must know when to abstain
  • SQuAD 2.0 addresses this
• Multi-Hop Reasoning:
  • Need to combine info from multiple sentences
  • "Who is the CEO of the company that made iPhone?"
  • Requires reasoning chains
• Numerical Reasoning:
  • Arithmetic operations
  • "How many more X than Y?"
  • Special numerical reasoning modules needed
• Ambiguous Questions:
  • Multiple valid interpretations
  • Context-dependent meaning
• Common Sense:
  • Implicit knowledge not stated in text
  • Requires world knowledge

Architecture Components

• For Extractive QA:
  • Encoder: BERT, RoBERTa, etc.
  • Start Position Head: Linear layer → softmax
  • End Position Head: Linear layer → softmax
  • Training: Cross-entropy on start/end positions
• For Open-Domain QA:
  • Retriever: Find relevant passages (BM25, dense retrieval)
  • Reader: Extract answer from passages (BERT-based)
  • End-to-end training or separate training

Applications

• Search Engines:
  • Direct answer snippets
  • Google featured snippets
• Virtual Assistants:
  • Alexa, Siri, Google Assistant
  • Answer factual questions
• Customer Support:
  • Automatic FAQ responses
  • Chatbots
• Education:
  • Intelligent tutoring systems
  • Homework help
• Legal/Medical:
  • Research assistance
  • Case law search
  • Medical diagnosis support

Beyond Simple QA

• Conversational QA:
  • Multi-turn conversations
  • Coreference resolution
  • Context carries across turns
• Multi-Document QA:
  • Synthesize from multiple sources
  • Verify consistency
• Generative QA:
  • GPT-based models
  • Generate free-form answers
  • More natural but less controllable

Implementation Tips

• Use pre-trained BERT/RoBERTa models
• Fine-tune on SQuAD or domain data
• Implement "unanswerable" detection
• Use post-processing for answer validation
• Consider n-best answers with re-ranking
• Add confidence thresholding
• Test on diverse question types

Retrieval-Augmented Generation (RAG)

Modern approach combining retrieval and generation:

• Retrieve relevant documents from corpus
• Feed to language model as context
• Generate answer based on retrieved context
• Reduces hallucination
• Keeps knowledge up-to-date

Experiment with the Demo

Use the interactive tool above to:

• Ask questions about provided passages
• See how answers are extracted
• Try different question types (who, what, when, where, why)
• Observe confidence scores
• Understand answer highlighting in context

Question answering is one of the most practical NLP applications, powering search engines, virtual assistants, and countless other systems we use daily!