Activation Functions: Full Guide

Activation Functions for Nonlinearity in Neural Networks

Activation functions are nonlinear transformations that add complexity to neural networks, allowing them to learn complex patterns and make sophisticated predictions.

1. Core Principles of Activation Functions

Exploring Nonlinearity, Implementing Basic Functions

ReLU, Leaky ReLU, PReLU, experiments

Studying saturating activations, comparing

live demo · related simulation● LIVE

□ Planning for Monitoring Dead Neurons

12. Curriculum

Fundamentals: Understanding nonlinearity

Frequently asked questions

What is a dead neuron in neural networks?

Neurons that output 0 for all training examples become ‘dead’, receiving zero gradient and never updating.

Why does Leaky ReLU have a small gradient for negative values?

Leaky ReLU has a small gradient for negative values (αx), preventing the dying ReLU problem and improving gradient flow.

What is the vanishing gradient problem associated with Sigmoid?

Sigmoid has a vanishing gradient problem (gradient → 0 for extreme values) and is centered around 0.5, which hinders learning.

What is Swish(x) = x · sigmoid(x)?

Swish(x) = x · sigmoid(x). Smooth, non-monotonic for negative values, often better than ReLU on some tasks.

▶ Try it live

Everything above runs in your browser — open Hash Function Avalanche Visualizer and change the parameters while it is running. Nothing is installed, nothing is uploaded, the whole model lives in one tab.

▶ Open Hash Function Avalanche Visualizer simulation