Select Activation Functions to Compare
Activation Functions
Derivatives (Gradients)
Visualization Range
| Function | Formula | Range | Best Use Cases |
|---|---|---|---|
| ReLU | f(x) = max(0, x) | [0, ∞) | Hidden layers, CNNs, most deep networks |
| Sigmoid | f(x) = 1 / (1 + e^(-x)) | (0, 1) | Binary classification output, gates in LSTMs |
| Tanh | f(x) = (e^x - e^(-x)) / (e^x + e^(-x)) | (-1, 1) | Hidden layers, RNNs, zero-centered outputs |
| Leaky ReLU | f(x) = max(αx, x) | (-∞, ∞) | Preventing dying ReLU, GANs |
| ELU | f(x) = x if x>0 else α(e^x - 1) | (-α, ∞) | Deep networks, faster convergence |
| Swish | f(x) = x · sigmoid(x) | (-∞, ∞) | Deep networks, smooth non-linearity |
| Softplus | f(x) = ln(1 + e^x) | (0, ∞) | Smooth ReLU alternative |
| GELU | f(x) = x · Φ(x) | (-∞, ∞) | Transformers, BERT, modern NLP |
Understanding Activation Functions
Activation functions are the heart of neural networks. They introduce non-linearity into the network, allowing it to learn complex patterns beyond simple linear relationships. Without activation functions, even the deepest neural network would be equivalent to a single-layer linear model!
Why Do We Need Activation Functions?
Neural networks learn by composing multiple layers of transformations. If each layer only performed linear operations (y = wx + b), stacking multiple layers would still result in a linear transformation. Activation functions break this linearity, enabling networks to approximate any continuous function (universal approximation theorem).
ReLU (Rectified Linear Unit)
f'(x) = 1 if x > 0, else 0
✓ Advantages
- Computationally efficient (simple max operation)
- Reduces vanishing gradient problem
- Sparse activation (many neurons output 0)
- Empirically works well in practice
- Enables faster training than sigmoid/tanh
✗ Disadvantages
- Dying ReLU problem (neurons can "die")
- Not zero-centered (can cause zig-zagging)
- Unbounded output can lead to explosions
- Not differentiable at x = 0
When to use: Default choice for hidden layers in CNNs and feedforward networks. Start with ReLU unless you have a specific reason to use something else.
Sigmoid
f'(x) = f(x) · (1 - f(x))
✓ Advantages
- Smooth gradient, preventing jumps
- Output bounded between 0 and 1
- Clear probabilistic interpretation
- Perfect for binary classification
✗ Disadvantages
- Vanishing gradient problem (gradients near 0 for large |x|)
- Not zero-centered (slows convergence)
- Computationally expensive (exponential)
- Outputs saturate at extremes
When to use: Output layer for binary classification, gates in LSTM/GRU cells. Avoid in hidden layers of deep networks.
Tanh (Hyperbolic Tangent)
f'(x) = 1 - f(x)²
✓ Advantages
- Zero-centered (better than sigmoid)
- Stronger gradients than sigmoid
- Output bounded between -1 and 1
- Symmetric around origin
✗ Disadvantages
- Still suffers from vanishing gradients
- Computationally expensive
- Saturates for large |x|
When to use: Preferred over sigmoid for hidden layers in shallow networks, RNNs. Better than sigmoid but still inferior to ReLU for deep networks.
Leaky ReLU
f'(x) = 1 if x > 0, else α
✓ Advantages
- Solves dying ReLU problem
- Allows small gradient when x < 0
- Computationally efficient
- No saturation in positive region
✗ Disadvantages
- Results inconsistent across problems
- Need to tune α hyperparameter
- Not always better than ReLU
When to use: When experiencing dying ReLU problem, GANs, networks with many negative inputs.
ELU (Exponential Linear Unit)
f'(x) = 1 if x > 0 else f(x) + α
✓ Advantages
- Smoothly approaches negative values
- Mean activation closer to zero
- Can produce negative outputs
- Faster learning than ReLU in some cases
✗ Disadvantages
- Exponential computation (slower than ReLU)
- May cause explosions for large negative inputs
When to use: Deep networks requiring faster convergence, when computational cost isn't critical.
Swish (SiLU - Sigmoid Linear Unit)
f'(x) = f(x) + sigmoid(x)(1 - f(x))
✓ Advantages
- Smooth, non-monotonic function
- Self-gating property
- Outperforms ReLU in deep networks
- Unbounded above, bounded below
✗ Disadvantages
- More computationally expensive
- Requires more memory for backprop
When to use: Very deep networks, when computational resources allow, modern architectures like EfficientNet.
GELU (Gaussian Error Linear Unit)
✓ Advantages
- Smooth, differentiable everywhere
- Non-monotonic with curvature
- Empirically strong in transformers
- Stochastic regularizer interpretation
✗ Disadvantages
- Computationally expensive
- Complex formula
When to use: Transformers, BERT, GPT, modern NLP models. State-of-the-art for language models.
Choosing the Right Activation Function
Here's a practical decision guide:
- Default choice: ReLU for hidden layers - simple, fast, and effective
- Binary classification output: Sigmoid - gives probabilities between 0 and 1
- Multi-class classification: Softmax (not shown here) - gives probability distribution
- Regression output: Linear/Identity - no bounds on output
- Deep networks (>50 layers): Consider ELU, Swish, or GELU for better gradient flow
- RNNs/LSTMs: Tanh and sigmoid for internal gates
- GANs: Leaky ReLU often works better than ReLU
- Transformers/NLP: GELU is becoming standard
The Vanishing Gradient Problem
Sigmoid and tanh suffer from the vanishing gradient problem. When inputs are large (positive or negative), these functions saturate (flatten out), producing gradients close to zero. During backpropagation, these tiny gradients get multiplied across layers, becoming exponentially smaller. In deep networks, early layers receive almost no gradient signal and fail to learn.
ReLU helps because its gradient is either 0 or 1 - no multiplication makes gradients smaller. However, ReLU has its own "dying ReLU" problem where neurons can get stuck outputting 0.
The Dying ReLU Problem
When a neuron using ReLU receives large negative weighted inputs, it outputs 0 and has gradient 0. If this persists, the neuron stops learning entirely - it "dies." This can happen to many neurons, especially with high learning rates. Solutions include:
- Use Leaky ReLU or ELU (allow small gradients for negative inputs)
- Lower learning rate
- Proper weight initialization
- Batch normalization
Modern Trends and Research
Recent years have seen exciting developments:
- Swish/SiLU: Discovered through automated search, often outperforms ReLU
- GELU: Became dominant in transformers and language models
- Mish: Self-regularizing, smooth, non-monotonic (f(x) = x·tanh(softplus(x)))
- Adaptive activations: Learning activation functions as part of training
- Periodic activations: For specialized tasks like coordinate networks
Practical Tips
- Start with ReLU - it's the safe default choice
- If training is unstable, try Leaky ReLU or ELU
- For very deep networks, experiment with Swish or GELU
- Don't use sigmoid/tanh in hidden layers of deep networks
- Match activation to task: sigmoid for binary, softmax for multi-class
- Consider computational cost - sigmoid/tanh are expensive
- Monitor dead neurons - if >20% neurons always output 0, try alternatives
- Use proper weight initialization (He for ReLU, Xavier for tanh)
Mathematical Properties to Consider
When evaluating activation functions, consider:
- Range: Bounded functions prevent explosions but can saturate
- Zero-centered: Helps optimization, prevents zig-zagging
- Monotonicity: Most are monotonic; non-monotonic (Swish, GELU) can help
- Smoothness: Smooth functions have better gradient properties
- Computation cost: Exponentials are expensive; max operations are cheap
- Gradient properties: Non-vanishing gradients enable deeper networks
Experiment and Learn!
Use the interactive visualizations above to:
- Compare activation functions side-by-side
- Observe how derivatives (gradients) behave
- Understand why some functions saturate
- See the effect of hyperparameters (alpha values)
- Build intuition about gradient flow
The best way to understand these functions is to experiment with them in your own projects. Try different combinations, monitor training dynamics, and develop a feel for which works best for your specific tasks!