Deep Learning: Neural Network Architectures & Training Techniques

1. Feedforward Neural Networks

Multi-layer perceptrons (MLPs) consist of input, hidden, and output layers. Each neuron: z = Wx + b, a = σ(z). Activation functions: ReLU max(0,x) (most common), Leaky ReLU, GELU (used in Transformers), SiLU/Swish, Softmax (output layer for classification). Universal approximation theorem: single hidden layer can approximate any continuous function (given sufficient width). Training via backpropagation: chain rule computes ∂L/∂w through computational graph. Batch normalization: normalizes layer inputs, accelerates training. Layer normalization: preferred for Transformers. Dropout: randomly zeros neurons during training (p=0.1–0.5) for regularization.

2. Convolutional Neural Networks

CNNs exploit spatial hierarchy via local receptive fields and weight sharing. Convolution: output[i,j] = Σ kernel[m,n] · input[i+m, j+n]. Typical architecture: Conv → BN → ReLU → Pool, repeated. Pooling: max pooling (most common), average pooling, global average pooling. Key architectures: LeNet-5 (1998), AlexNet (2012, 60M params), VGG-16 (138M), GoogLeNet/Inception (6.8M, 1×1 convolutions), ResNet (skip connections, 152 layers), DenseNet (dense connections), EfficientNet (compound scaling). Modern: ConvNeXt matches ViT with pure convolutions. Object detection: YOLO (real-time), Faster R-CNN (two-stage). Semantic segmentation: U-Net (encoder-decoder with skip connections).

3. Recurrent Networks & Sequence Models

RNNs process sequential data: h_t = σ(W_hh h_{t-1} + W_xh x_t + b). Vanishing gradient problem limits long-range dependencies. LSTM (Hochreiter & Schmidhuber, 1997): forget gate, input gate, output gate, cell state — captures dependencies over 100+ timesteps. GRU: simplified LSTM with 2 gates (update, reset), comparable performance. Bidirectional RNNs: process sequence forward and backward. Encoder-decoder with attention: Bahdanau attention (2014) weights source positions. Seq2seq applications: machine translation, text summarization, speech recognition. Largely superseded by Transformers for most NLP tasks but remain relevant for streaming/online applications.

4. Transformers & Attention

Transformer architecture (Vaswani et al., 2017, "Attention Is All You Need"). Self-attention: Attention(Q,K,V) = softmax(QKᵀ/√d_k)V. Multi-head attention: parallel attention with different learned projections. Positional encoding: sinusoidal or learned embeddings. Encoder-only: BERT (bidirectional, 340M params, MLM + NSP pretraining). Decoder-only: GPT series (autoregressive, GPT-4 estimated 1.8T params). Encoder-decoder: T5, BART. Vision Transformers (ViT): image patches as tokens, 16×16 patches. Scaling laws: L ∝ N^{-0.076} (loss vs parameters). Flash Attention: IO-aware exact attention, 2-4× faster. Mixture of Experts (MoE): Mixtral 8×7B, sparse activation.

5. Training at Scale

Optimization: Adam (β₁=0.9, β₂=0.999) most common, AdamW (decoupled weight decay). Learning rate schedules: warmup + cosine decay. Gradient clipping: prevents exploding gradients (max norm 1.0). Mixed precision: FP16/BF16 computation with FP32 master weights — 2× speedup, 50% memory reduction. Distributed training: data parallelism (DDP), model parallelism (pipeline, tensor), ZeRO optimizer stages (DeepSpeed). Hardware: NVIDIA H100 (989 TFLOPS FP16), Google TPU v5p. Training cost: GPT-4 estimated $100M+, Llama 3 405B: 30.8M GPU-hours on H100. Techniques: gradient accumulation, gradient checkpointing (trade compute for memory), distillation.

Try it live

Everything above runs in your browser — open Dimensionality Reduction: PCA, t-SNE & UMAP and change the parameters while it is running. Nothing is installed, nothing is uploaded, the whole model lives in one tab.