Search⌘ K
Home
Courses
Understanding Deep Learning Applications in Rare Event Prediction

Optimizing and Evaluating the Final Neural Network Model

Explore how to optimize and evaluate Multi-Layer Perceptron models by comparing performance metrics, addressing overfitting risks, and selecting appropriate activation functions. Learn key strategies for data preprocessing, class weighting, and parameter tuning relevant to rare event prediction with imbalanced data.

We'll cover the following...

The final model selection is a critical step, relying not just on performance metrics but also on the model’s ability to generalize effectively to unseen data. Our comparative analysis of models, as summarized in the table, reveals the nuanced trade-offs between model complexity, performance metrics, and the risk of overfitting. The selection is made by comparing the validation results.

MLP Models Comparison

Model

Validation

Loss

f1-score

Recall

fpr

Baseline

Increasing

0.13

0.08

0.001

Dropout

Non-increasing

0.00

0.00

0.000

Class weights

Non-increasing

0.12

0.31

0.102

selu

Nonincreasing

0.04

0.02

0.001

telu (custom)

Non-increasing

0.12

0.08

0.001

The baseline model has higher accuracy measures but increasing validation loss indicating potential overfitting. Dropout resolves the overfitting, but the accuracy goes to zero. Class weights boosted the accuracy but also had a high false-positive rate. The selu activation attempted had significantly lower f1-score than the baseline.

The custom activation telu performed relatively better than the others. It has a non-increasing validation loss and is close to the baseline accuracy. Therefore, the telu activated model is selected as the final model.

This does not mean telu will be the best choice in other cases, too. The process of building multiple models and selection should be followed for every problem.

The final selected model is evaluated using the model.evaluate() function. The evaluate function applies the trained model on test data and returns the performance measures defined in the model.compile() function.

The evaluation and test results are shown in the code below.

Python 3.8
# Final model
loss,accuracy,recall,f1_score,fpr = model.evaluate( x=X_test_scaled ,
                                                    y=y_test , batch_size=128,
                                                    verbose =1)
print(' Loss: ',loss, '\n Acuuracy: ',accuracy, 
        '\n recall_5: ',recall, '\n f1_score: ',f1_score,
        '\n false_positive_rate: ',fpr)
# loss: 0.0796 - accuracy: 0.9860 -
# recall_5: 0.0755 - f1_score: 0.1231 - # false_positive_rate: 0.0020

MLPs are elementary deep learning models. Any reasonable results from MLPs act as a preliminary screening that there are some predictive patterns in the data. This lays down a path for further development with different network architectures.

MLP conclusion

In practice, there are many choices, but they could be overwhelming. Let’s take a look at them.

Number of layers and nodes

  • Number of layers: Start with two hidden layers, excluding the output layer.

  • Number of nodes of intermediate layers: This is a number from a geometric series of 2:1,2,4,8,16,32, ...