Neural Network Architecture Hyperparameters

Learn about neural network architecture hyperparameters. Understanding layer sizes, activation functions, and network design parameters.

▶ Open the simulation

Introduction

Neural network architecture hyperparameters determine the structure and capacity of the network. These parameters significantly impact model performance, training efficiency, and generalization ability. Understanding how to tune architecture parameters is crucial for building effective neural networks.

Network Depth

Number of Layers

Controls the depth of the network:

Shallow networks: 1-3 hidden layers
Deep networks: 4+ hidden layers
Very deep: 10+ layers (ResNet, DenseNet)

Depth Considerations

Deeper networks can learn more complex patterns
Risk of vanishing/exploding gradients
Requires more data and computation
May need skip connections (ResNet)

Layer Width

Hidden Units

Number of neurons in each hidden layer:

# Common patterns
hidden_units = [512, 256, 128]  # Decreasing
hidden_units = [256, 256, 256]  # Constant
hidden_units = [128, 256, 512]  # Increasing
            

Width Guidelines

Start with 2-4x input dimension
Use powers of 2 for efficiency
Consider computational constraints
Balance capacity vs overfitting

Activation Functions

Common Activations

ReLU: f(x) = max(0, x)
Leaky ReLU: f(x) = max(0.01x, x)
ELU: f(x) = x if x > 0, else α(e^x - 1)
Sigmoid: f(x) = 1/(1 + e^(-x))
Tanh: f(x) = (e^x - e^(-x))/(e^x + e^(-x))

Selection Guidelines

ReLU: Default choice for most cases
Leaky ReLU: When ReLU causes dead neurons
Sigmoid/Tanh: Output layers for classification
ELU: When smooth gradients are important

Skip Connections

Residual Connections

Add input to layer output:

output = activation(linear(input)) + input

Dense Connections

Connect each layer to all subsequent layers:

x1 = layer1(input)
x2 = layer2(input + x1)
x3 = layer3(input + x1 + x2)
            

Convolutional Parameters

Filter Size

3x3: Most common, good balance
5x5: Larger receptive field
1x1: Pointwise convolution
7x7: Very large receptive field

Stride

1: Preserves spatial dimensions
2: Reduces spatial dimensions by half
Larger: More aggressive downsampling

Padding

Same: Preserves input size
Valid: No padding, reduces size
Custom: Specific padding amount

Recurrent Parameters

Hidden State Size

Number of hidden units in RNN/LSTM/GRU:

Start with 64-256 units
Increase for complex sequences
Consider computational cost
Balance capacity vs overfitting

Number of Layers

1-2 layers: Simple sequences
3-4 layers: Complex sequences
5+ layers: Very complex patterns

Attention Mechanisms

Attention Heads

Number of attention heads in transformer:

8 heads: Standard for BERT
16 heads: Large models
Multi-head allows different attention patterns

Key/Query/Value Dimensions

Usually hidden_size / num_heads
Affects model capacity
Balance between efficiency and performance

Key Insight

Architecture hyperparameters should be tuned based on problem complexity, available data, and computational resources. Start with proven architectures and gradually modify based on validation performance.

Architecture Search Strategies

Grid Search

depths = [2, 3, 4, 5]
widths = [64, 128, 256, 512]
activations = ['relu', 'leaky_relu', 'elu']
            

Progressive Search

Start with simple architecture
Add complexity gradually
Stop when performance plateaus
Use early stopping

Frequently Asked Questions

How do I choose the number of layers?

Start with 2-3 layers, increase if underfitting, decrease if overfitting. Consider problem complexity, data size, and computational resources. Use cross-validation to find optimal depth.

What's the best activation function?

ReLU is the default choice for most cases. Use Leaky ReLU if you have dead neurons, sigmoid/tanh for output layers, and ELU for smooth gradients. Test multiple options.

How do I determine layer width?

Start with 2-4x input dimension, use powers of 2 for efficiency, consider computational constraints, and balance capacity with overfitting risk.

Should I use skip connections?

Use skip connections (ResNet, DenseNet) for deep networks (>10 layers) to help with gradient flow. For shallow networks, they may not be necessary.

How do I choose CNN filter sizes?

Use 3x3 filters as default, 1x1 for pointwise convolution, 5x5 or 7x7 for larger receptive fields. Consider input image size and desired feature complexity.