...
/Optimizing and Evaluating the Final Neural Network Model
Optimizing and Evaluating the Final Neural Network Model
Learn the process of selecting the best neural network model based on validation results.
The final model selection is a critical step, relying not just on performance metrics but also on the model’s ability to generalize effectively to unseen data. Our comparative analysis of models, as summarized in the table, reveals the nuanced trade-offs between model complexity, performance metrics, and the risk of overfitting. The selection is made by comparing the validation results.
MLP Models Comparison
Model | Validation | |||
Loss | f1-score | Recall | fpr | |
Baseline | Increasing | 0.13 | 0.08 | 0.001 |
Dropout | Non-increasing | 0.00 | 0.00 | 0.000 |
Class weights | Non-increasing | 0.12 | 0.31 | 0.102 |
| Nonincreasing | 0.04 | 0.02 | 0.001 |
| Non-increasing | 0.12 | 0.08 | 0.001 |
The baseline model has higher accuracy measures but increasing validation loss indicating potential overfitting. Dropout resolves the overfitting, but the accuracy goes to zero. Class weights boosted the accuracy but also had a high false-positive rate. The selu activation attempted had significantly lower f1-score than the baseline.
The custom activation telu performed relatively better than the others. It has a non-increasing validation loss and is close to the baseline accuracy. Therefore, the telu activated model is selected as the final model.
This does not mean telu will be the best choice in other cases, too. The process of building multiple models and selection should be followed for every problem.
The final selected model is evaluated using the model.evaluate() function. The evaluate function applies the trained model on test data and returns the performance measures defined in the model.compile() function.
The evaluation and test results are shown in the code below.
# Final modelloss,accuracy,recall,f1_score,fpr = model.evaluate( x=X_test_scaled ,y=y_test , batch_size=128,verbose =1)print(' Loss: ',loss, '\n Acuuracy: ',accuracy,'\n recall_5: ',recall, '\n f1_score: ',f1_score,'\n false_positive_rate: ',fpr)# loss: 0.0796 - accuracy: 0.9860 -# recall_5: 0.0755 - f1_score: 0.1231 - # false_positive_rate: 0.0020
MLPs are elementary deep learning models. Any reasonable results from MLPs act as a preliminary screening that there are some predictive patterns in the data. This lays down a path for further development with different network architectures.
MLP conclusion
In practice, there are many choices, but they could be overwhelming. Let’s take a look at them.
Number of layers and nodes
-
Number of layers: Start with two hidden layers, excluding the output layer.
-
Number of nodes of intermediate layers: This is a number from a geometric series of ...