Model Selection Hyperparameters

Learn about model selection hyperparameters in machine learning. Understanding cross-validation, ensemble methods, and model comparison parameters.

▶ Open the simulation

Introduction

Model selection hyperparameters control how different models are compared, validated, and selected. These parameters determine the evaluation strategy, ensemble methods, and model comparison criteria. Understanding how to tune these parameters is crucial for building robust and reliable machine learning systems.

Cross-Validation Parameters

K-Fold Cross-Validation

Splits data into k folds for validation:

cv = KFold(n_splits=5, shuffle=True, random_state=42)

N splits: number of folds (typical: 5-10)
Shuffle: randomize data before splitting
Random state: for reproducibility

Stratified K-Fold

Maintains class distribution in each fold:

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

N splits: number of folds
Shuffle: randomize data
Use for classification with imbalanced classes

Time Series Split

Respects temporal order in time series:

cv = TimeSeriesSplit(n_splits=5)

N splits: number of splits
No shuffle (maintains order)
Use for time series data

Ensemble Parameters

Random Forest

Ensemble of decision trees:

n_estimators = 100  # number of trees
max_depth = None    # maximum tree depth
min_samples_split = 2  # minimum samples to split
min_samples_leaf = 1   # minimum samples per leaf
            

N estimators: number of trees
Max depth: tree depth limit
Min samples: splitting criteria

Gradient Boosting

Sequential ensemble method:

n_estimators = 100    # number of boosting stages
learning_rate = 0.1   # shrinkage parameter
max_depth = 3         # maximum tree depth
subsample = 1.0       # fraction of samples used
            

N estimators: number of boosting stages
Learning rate: shrinkage parameter
Max depth: tree depth limit
Subsample: sampling fraction

Model Comparison Parameters

Scoring Metrics

Metrics for model evaluation:

scoring = 'accuracy'  # for classification
scoring = 'neg_mean_squared_error'  # for regression
scoring = ['accuracy', 'precision', 'recall', 'f1']  # multiple metrics
            

Accuracy: overall correctness
Precision: true positives / (true positives + false positives)
Recall: true positives / (true positives + false negatives)
F1: harmonic mean of precision and recall

Validation Strategy

How to split data for validation:

Train/validation/test: 60/20/20 split
Cross-validation: k-fold validation
Holdout: single train/test split
Nested CV: outer CV for model selection, inner CV for hyperparameter tuning

Hyperparameter Search Parameters

Grid Search

Exhaustive search over parameter grid:

param_grid = {
    'C': [0.1, 1, 10, 100],
    'gamma': [0.001, 0.01, 0.1, 1],
    'kernel': ['rbf', 'linear', 'poly']
}
            

Parameter grid: dictionary of parameters
CV: cross-validation strategy
Scoring: evaluation metric

Random Search

Random sampling from parameter space:

param_dist = {
    'C': uniform(0.1, 100),
    'gamma': uniform(0.001, 1),
    'kernel': ['rbf', 'linear', 'poly']
}
            

Parameter distribution: parameter ranges
N iterations: number of random samples
CV: cross-validation strategy

Model Selection Criteria

Information Criteria

Balance model fit and complexity:

AIC: Akaike Information Criterion
BIC: Bayesian Information Criterion
Lower values indicate better models

Cross-Validation Score

Average performance across folds:

Mean CV score: average performance
Std CV score: performance variability
Use for model comparison

Ensemble Selection Parameters

Voting Classifier

Combines predictions from multiple models:

voting = 'hard'  # or 'soft'
weights = [1, 2, 1]  # model weights
            

Voting: 'hard' or 'soft' voting
Weights: model importance weights
Models: list of base models

Stacking

Uses meta-model to combine predictions:

meta_model = LogisticRegression()
base_models = [RandomForestClassifier(), GradientBoostingClassifier()]
            

Meta model: final prediction model
Base models: first-level models
CV: cross-validation for meta-model training

Model Persistence Parameters

Model Saving

Parameters for saving trained models:

Format: pickle, joblib, ONNX
Compression: gzip, lz4
Version: model versioning

Model Loading

Parameters for loading saved models:

Format: match saving format
Validation: check model compatibility
Fallback: handle version mismatches

Key Insight

Model selection parameters should be chosen based on your data characteristics, problem type, and computational constraints. Use appropriate validation strategies and consider ensemble methods for improved performance.

Parameter Tuning Strategies

Nested Cross-Validation

outer_cv = StratifiedKFold(n_splits=5)
inner_cv = StratifiedKFold(n_splits=3)
            

Model Comparison

Use same CV strategy for all models
Compare using statistical tests
Consider computational cost
Validate on holdout test set

Frequently Asked Questions

How do I choose the right cross-validation strategy?

Use KFold for balanced data, StratifiedKFold for imbalanced classification, TimeSeriesSplit for time series. Choose 5-10 folds based on data size and computational constraints.

What's the difference between grid search and random search?

Grid search exhaustively tests all parameter combinations, while random search samples randomly from parameter space. Random search is often more efficient for high-dimensional parameter spaces.

How do I compare different models fairly?

Use the same CV strategy, same evaluation metrics, and same data splits. Use statistical tests to determine if performance differences are significant. Validate on a holdout test set.

When should I use ensemble methods?

Use ensembles when you have multiple good models, want to reduce overfitting, or need more robust predictions. Random Forest and Gradient Boosting are good starting points.

How do I choose the right scoring metric?

Use accuracy for balanced classification, F1 for imbalanced data, precision/recall for specific requirements, RMSE/MAE for regression. Consider your business objectives.