🧠 Neural Network Architecture Hyperparameters

About this simulation

Before a neural network ever sees training data, its designer must fix a handful of architecture hyperparameters: how many layers to stack, how wide each layer should be, which activation function fires between them, and how much of the network to randomly silence with dropout. These choices are not learned from data — they define the hypothesis space the network can even represent, and they are usually the first thing tuned during model design.

How it works

• Hidden layers — depth of the network. More layers let the model compose more abstract features, but add training difficulty (vanishing/exploding gradients) and parameters.
• Neurons per layer — width of each hidden layer. Wider layers increase capacity to memorize patterns but raise compute cost and overfitting risk.
• Activation function — the nonlinearity applied at each neuron. ReLU is fast and avoids vanishing gradients for positive inputs; Sigmoid and Tanh squash outputs into a fixed range; Linear removes nonlinearity entirely, collapsing the network to a single linear map.
• Dropout — the fraction of neurons randomly zeroed each forward pass during training, a regularizer that discourages co-adapted features and reduces overfitting.
• Parameter count — the total learnable weights and biases, computed as the sum of (inputs × outputs + outputs) across every connected layer pair.

Controls

• Hidden layers — sets the number of hidden layers between input and output (1–6).
• Neurons per layer — sets the width of every hidden layer (2–14).
• Dropout rate — randomly deactivates this fraction of hidden neurons on each pass (0–60%).
• Activation function — choose the nonlinearity used at every hidden neuron.
• Animate forward pass — toggles a traveling signal pulse that lights up connections layer by layer.
• ↺ Rebuild network — regenerates dropout mask and restarts the pulse. Drag to rotate, scroll to zoom.

Did you know?

Unlike weights and biases, architecture hyperparameters are not learned by gradient descent — they are chosen by the designer (or by automated search methods like grid search, Bayesian optimization, or neural architecture search) before training even begins.