LLM Integration Patterns

Large Language Models are powerful but expensive and unpredictable. Production integration requires careful architecture - choosing between prompting, fine-tuning, and RAG; managing costs and latency; and building guardrails against hallucinations and misuse.

Where It Is Used

LLM integration powers new capabilities across industries:

• Customer support: Chatbots, ticket summarization, response drafting
• Content: Generation, editing, summarization, translation
• Development: Code completion, documentation, review assistance
• Legal: Contract analysis, document summarization
• Healthcare: Clinical note summarization, patient communication
• E-commerce: Product descriptions, review summarization

Products: ChatGPT, GitHub Copilot, Notion AI, Jasper, Grammarly, various enterprise copilots.

Typical Data You Will See

Data sources for LLM applications: - User queries and conversation history - Documents for retrieval (RAG) - Few-shot examples for prompting - Training data for fine-tuning

What You Do in Practice

1. Define the task - What exactly should the LLM do? What is success?
2. Choose integration approach - Prompting, fine-tuning, or RAG?
3. Select model - Balance capability vs cost vs latency
4. Design prompts - System prompts, few-shot examples, output formatting
5. Build retrieval - If RAG, implement document retrieval pipeline
6. Add guardrails - Input validation, output filtering, rate limiting
7. Handle failures - Timeouts, fallbacks, retry logic
8. Evaluate quality - Human evaluation, automated metrics, A/B tests
9. Monitor costs - Track token usage, set budgets, optimize prompts
10. Iterate continuously - Prompts and retrieval need ongoing tuning

Baseline First

Start with zero-shot prompting before complex solutions:

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": user_query}
    ]
)

Why prompting baseline: - No training data required - Immediate iteration - Low initial cost - Easy to experiment

Only move to fine-tuning or RAG when prompting limitations are clear.

Integration Approaches

Decision tree: 1. Can zero-shot work? Try it first. 2. Need specific format? Add few-shot examples. 3. Need external knowledge? Add RAG. 4. Need consistent style/domain? Consider fine-tuning. 5. Need multi-step actions? Consider agents.

Prompting Patterns

System prompt design: - Define role and constraints - Specify output format (JSON, markdown) - Include examples of good outputs - Add safety instructions

Few-shot structure:

System: You are a classifier...

User: [Example input 1]
Assistant: [Example output 1]

User: [Example input 2]
Assistant: [Example output 2]

User: [Actual input]

Output formatting: - Request JSON for structured data - Use XML tags for sections - Specify length constraints - Include field explanations

Chain of thought: - "Let's think step by step" - Request reasoning before answer - Useful for complex tasks

Evaluation and Metrics

LLM evaluation is challenging - combine approaches:

Evaluation framework: 1. Define criteria (relevance, accuracy, safety, style) 2. Create evaluation dataset (100-500 examples) 3. Run automated metrics as baseline 4. Sample human evaluation (10-20%) 5. Track business metrics in production

Pitfalls: - Automated metrics poorly correlate with human judgment - LLM-as-judge has biases (verbosity, position) - Human evaluation is expensive and slow - Not testing edge cases and adversarial inputs

Failure Modes and Debugging Checklist

• [ ] Hallucinations: Model invents facts not in context
• [ ] Prompt injection: User input manipulates system behavior
• [ ] Format failures: Output does not match expected structure
• [ ] Refusals: Model refuses valid requests
• [ ] Context overflow: Input too long, truncation issues
• [ ] Latency spikes: Variable response times
• [ ] Cost explosion: Unexpected token usage
• [ ] Quality drift: Model updates change behavior

Production Checklist

Before deployment: - [ ] Prompt tested on diverse inputs - [ ] Output parsing handles edge cases - [ ] Rate limiting configured - [ ] Cost tracking in place - [ ] Fallback behavior defined - [ ] Content filtering for outputs - [ ] Input validation for injections

After deployment: - [ ] Token usage monitored daily - [ ] Quality sampled and reviewed - [ ] Error rates tracked - [ ] Latency percentiles monitored - [ ] A/B testing framework ready - [ ] Prompt versioning in place - [ ] Model update testing process

Security: - [ ] Prompt injection protection - [ ] PII filtering on inputs/outputs - [ ] Output content moderation - [ ] Audit logging for compliance

Cost Optimization

Strategies: - Use cheaper models where possible (GPT-4o-mini vs GPT-4) - Cache repeated queries - Batch requests when latency allows - Compress prompts (remove redundancy) - Limit output tokens - Use fine-tuning to reduce prompt length

Cost formula:

Monthly cost = (input_tokens + output_tokens) * price_per_token * requests

Typical cost breakdown: | Volume | Model | Estimated Monthly Cost | |--------|-------|------------------------| | 10K requests | GPT-4o-mini | $5-20 | | 10K requests | GPT-4o | $50-200 | | 100K requests | GPT-4o-mini | $50-200 | | 100K requests | GPT-4o | $500-2,000 |

Tools and Stack

Example Mini-Case

Scenario: Customer support team needs AI to draft email responses based on ticket content and knowledge base.

Approach: 1. Baseline: Zero-shot GPT-4o-mini with simple prompt - Quality: 3.2/5 2. Add few-shot: 5 examples of good responses - Quality: 3.8/5 3. Add RAG: Retrieve relevant KB articles - Quality: 4.2/5 4. Guardrails: Filter PII, check for hallucinations - Safety improved

Production architecture:

User ticket → Retrieve KB articles → Construct prompt → LLM → 
Filter output → Draft review queue → Agent edits → Send

Metrics: - Draft acceptance rate: 65% - Average edit time: 2 min → 45 sec - Cost per ticket: $0.03 - Customer satisfaction: unchanged (validation passed)

Key learnings: - RAG critical for accurate product information - Human review essential for first 3 months - Prompt iteration reduced hallucinations by 80%

Related Pages

• Index: ML Field Guide
• Retrieval-Augmented Generation
• Text Classification at Scale
• Semantic Search Systems

Further Reading

• OpenAI API Documentation
• Anthropic Claude Documentation
• LangChain Documentation
• Prompt Engineering Guide

Try it live

Everything above runs in your browser — open LLM Evaluation Pipeline: Continuous Testing & Regression Gate Simulator and change the parameters while it is running. Nothing is installed, nothing is uploaded, the whole model lives in one tab.