Evaluation Metrics Hyperparameters

Learn about evaluation metrics hyperparameters in machine learning. Understanding classification, regression, and custom metric parameters.

▶ Open the simulation

Introduction

Evaluation metrics hyperparameters control how model performance is measured and compared. These parameters determine the evaluation criteria, threshold settings, and metric calculations. Understanding how to tune these parameters is crucial for accurate model assessment and selection.

Classification Metrics Parameters

Precision and Recall

Measures of classification performance:

precision = TP / (TP + FP)
recall = TP / (TP + FN)
            

Threshold: decision threshold for binary classification
Average: 'micro', 'macro', 'weighted' for multi-class
Zero division: how to handle division by zero

F1 Score

Harmonic mean of precision and recall:

f1 = 2 * (precision * recall) / (precision + recall)

Beta: weight for recall vs precision
Average: 'micro', 'macro', 'weighted'
Labels: specific classes to include

ROC AUC

Area under the ROC curve:

Multi class: 'ovr', 'ovo' for multi-class
Average: 'macro', 'weighted' for multi-class
Max fpr: maximum false positive rate

Regression Metrics Parameters

Mean Squared Error (MSE)

Average squared prediction errors:

mse = mean((y_true - y_pred)²)

Sample weight: weights for each sample
Multi output: 'uniform_average', 'raw_values'
Squared: return MSE (True) or RMSE (False)

Mean Absolute Error (MAE)

Average absolute prediction errors:

mae = mean(|y_true - y_pred|)

Sample weight: weights for each sample
Multi output: 'uniform_average', 'raw_values'
No additional parameters

R² Score

Coefficient of determination:

r2 = 1 - (SS_res / SS_tot)

Sample weight: weights for each sample
Multi output: 'uniform_average', 'raw_values'
Force finite: handle infinite values

Custom Metrics Parameters

Custom Loss Functions

User-defined evaluation metrics:

def custom_metric(y_true, y_pred, param1=1.0, param2=0.5):
    return calculated_metric
            

Parameters: custom hyperparameters
Sample weight: weights for each sample
Return: metric value or dict of metrics

Business Metrics

Metrics aligned with business objectives:

Revenue weight: weight by revenue impact
Cost weight: weight by cost impact
Threshold: business decision threshold

Threshold Parameters

Binary Classification Threshold

Decision threshold for binary classification:

threshold = 0.5  # default threshold
optimal_threshold = find_optimal_threshold(y_true, y_proba)
            

Threshold: decision boundary
Optimization: 'f1', 'precision', 'recall'
Method: 'grid', 'optimize'

Multi-Class Thresholds

Thresholds for multi-class classification:

Per class: different threshold per class
Global: same threshold for all classes
Optimization: optimize for specific metric

Cross-Validation Metrics

CV Scoring

Metrics for cross-validation:

scoring = 'accuracy'  # single metric
scoring = ['accuracy', 'precision', 'recall', 'f1']  # multiple metrics
            

Scoring: metric or list of metrics
CV: cross-validation strategy
Return: train scores, test scores

Learning Curves

Metrics vs training size:

Train sizes: array of training sizes
CV: cross-validation strategy
Scoring: evaluation metric

Statistical Test Parameters

Paired T-Test

Compare two model performances:

t_stat, p_value = ttest_rel(scores1, scores2)

Alternative: 'two-sided', 'less', 'greater'
Alpha: significance level
Correction: multiple comparison correction

McNemar's Test

Compare two binary classifiers:

Continuity: continuity correction
Exact: exact p-value calculation
Alpha: significance level

Metric Aggregation Parameters

Micro Average

Calculate metrics globally:

Counts: global true/false positives/negatives
Use: when class imbalance is important
Formula: global calculation

Macro Average

Calculate metrics per class then average:

Per class: calculate metric for each class
Average: unweighted average across classes
Use: when all classes are equally important

Weighted Average

Weight by class support:

Support: number of samples per class
Weight: support as weight
Use: when class distribution matters

Metric Selection Guidelines

Classification Tasks

Balanced classes: accuracy, F1
Imbalanced classes: precision, recall, F1
Multi-class: macro F1, weighted F1
Ranking: NDCG, MAP

Regression Tasks

Normal errors: MSE, RMSE
Outlier presence: MAE, Huber
Percentage errors: MAPE, sMAPE
Relative errors: R², adjusted R²

Key Insight

Metric selection should align with your business objectives and problem characteristics. Use multiple metrics to get a complete picture of model performance, and consider threshold optimization for classification tasks.

Parameter Tuning Strategies

Threshold Optimization

from sklearn.metrics import precision_recall_curve
precision, recall, thresholds = precision_recall_curve(y_true, y_proba)
optimal_threshold = thresholds[np.argmax(precision + recall)]
            

Metric Validation

Use cross-validation
Compare multiple metrics
Consider business impact
Validate on holdout set

Frequently Asked Questions

How do I choose the right evaluation metric?

Consider your problem type, class distribution, and business objectives. Use accuracy for balanced classification, F1 for imbalanced data, precision/recall for specific requirements, RMSE/MAE for regression.

What's the difference between micro and macro averaging?

Micro averaging calculates metrics globally, macro averaging calculates per class then averages. Use micro when class imbalance matters, macro when all classes are equally important.

How do I optimize classification thresholds?

Use precision-recall curves, ROC curves, or grid search to find optimal thresholds. Optimize for your specific metric (F1, precision, recall) using validation data.

When should I use custom metrics?

Use custom metrics when standard metrics don't capture your business objectives. Consider revenue impact, cost implications, or domain-specific requirements. Ensure metrics are differentiable for optimization.

How do I handle multi-class evaluation?

Use appropriate averaging (micro, macro, weighted), consider per-class metrics, and use confusion matrices for detailed analysis. Choose metrics based on class importance and distribution.