Search⌘ K
AI Features
Home
Courses
An Introductory Guide to Data Science and Machine Learning

Model Evaluation: Part 1

Explore important classification model evaluation metrics including the confusion matrix, accuracy, precision, recall, and F1 score. Understand how these measures help assess your model's performance and their applications in binary and multi-class classification. This lesson equips you to calculate and interpret these evaluation statistics effectively.

We'll cover the following...

Model evaluation measures

In this lesson, we will observe model evaluation for classification. We predict a discrete-valued output, and it has its own evaluation measures.

Confusion matrix

The confusion matrix serves as an evaluation measure for classification problems. The confusion matrix for a binary classification problem having only two class labels (0 and 1) would look like this:

Actual Class Label (1) Actual Class Label (0)
Predicted Class Label (1) TP FP
Predicted Class Label (0) FN TN

Here

TP (True Positive): We predicted a positive Class Label (1), and it is True.

TN (True Negative): We predicted a negative Class Label (0), and it is True.

FP (False Positive): It is also called Type 1 Error. We predicted a positive Class Label (1), and it is False.

FN (False Negative): It is also called a Type 2 Error. We predicted a negative Class Label (0), and it is True.

Note that positive and negative refer to the predicted class labels, whereas True and False refer to the actual class labels.

This confusion matrix is the basis of many evaluation measures we are calculating below:

Python 3.5
def confusion_matrix_calculation(y_true, y_pred):
    """
    This function calculates the following values (TP, FP, TN, FN).
    It takes in y_true (which are actual class labels) and y_pred(which 
    are predicted class labels).
    """
    TP = 0
    FP = 0
    TN = 0
    FN = 0
    for i in range(len(y_pred)): 
        if y_true[i]==y_pred[i]==1:          
           TP += 1
        if y_pred[i]==1 and y_true[i]!=y_pred[i]:
           FP += 1
        if y_true[i]==y_pred[i]==0:
           TN += 1
        if y_pred[i]==0 and y_true[i]!=y_pred[i]:
           FN += 1
    return(TP, FP, TN, FN)
y_true = [1, 0, 1, 0, 0, 1]
y_pred = [0, 0, 1, 1, 0, 1]
TP, FP, TN, FN = confusion_matrix_calculation(y_true, y_pred)
print("The True Class Labels are {}.".format(y_true))
print("The Predicted Class Labels are {}.".format(y_pred))
print("The number of True Positives are {}.".format(TP))
print("The number of False Positives are {}.".format(FP))
print("The number of False Negatives are {}.".format(FN))
print("The number of True Negatives are {}.".format(TN))

  • Line 1 defines the function to calculate the values inside a confusion matrix.

  • Lines 12 to 20 calculate the values that are present inside a confusion matrix. Things to be learned are the logic used to calculate the values.

  • Lines 27 and 28, print the actual and the predicted class labels.

  • Lines 29 to 32 print the respective statistics.

Accuracy

Accuracy is calculated as the ratio of the correctly classified instances and the total number of instances.

Accuracy=NumberofCorrectPredictionsTotalNumberofPredictionsAccuracy = \frac{Number of Correct Predictions}{Total Number of Predictions} ...