Model Evaluation: Part 2
Explore key evaluation metrics for classification models including balanced accuracy, which addresses imbalanced data, and ROC and PR curves that analyze performance across varying thresholds. Understand the calculation and interpretation of these metrics to better gauge model effectiveness in real-world scenarios.
We'll cover the following...
Balanced accuracy
This measure is intended for datasets where the class distribution is skewed, i.e., one class label (e.g., 1) has more instances than the other (e.g., 0). Here, the class label with more instances is referred to as the majority class, and the class label with fewer instances is called the minority class.
-
The model trained on imbalance dataset and evaluated using the accuracy measure cannot be trusted.
-
This model tends to predict the majority class all the time, and the accuracy comes out to be 98%, which seems good, but is misleading. The model is unable to learn the minority class.
To evaluate such models, other evaluation measures come to the rescue, and balanced accuracy is one of them.
Balanced accuracy is accuracy where each instance is weighted according to the inverse prevalence of its true class. It is computed by taking the average of the recall for each class. This score, for balanced datasets, depicts the accuracy score.
For binary class classification, balanced accuracy is given as:
...