Business Machine Learning/

...

/

Performance of the Trained Models on Unseen Data

Performance of the Trained Models on Unseen Data

Explore how trained models perform on unseen data.

We'll cover the following...

In reality, we don’t have the labels available for the unseen data. So, we have a separate unseen dataset that we can use to evaluate our trained models. Let’s read the unseen data and see the performance of our trained models.

Python 3.8
unseen=pd.read_csv("BioAssay_AID362red_test.csv")
print(unseen.head())

We can use our custom function to check the missing data columns.

Python 3.8
# Utility function to check the missing columns
def missing_col_check(df):
    # Checking if there is any missing data 
    emp_col_list=[]
    for col_ in df.columns:
        if ((df[col_].isnull().sum()/df.shape[0]*100)>0.0):
            emp_col_list.append(col_)
            #print(col_)
    print("The dataset scanned!")
    if len(emp_col_list)==0:
        print("No missing data column found!")
    else:print("Missing data found in: {}".format(emp_col_list))
    return None
missing_col_check(unseen)

Let’s separate our features and targets and observe the class balance in unseen data.

Python 3.8
X_unseen=unseen.drop("Outcome",axis=1)
y_unseen=unseen.Outcome
print("The class balance in unseen data is:")
print(y_unseen.value_counts(normalize=True))

We have trained the following three models from the previous lesson:

  • logR on the original data.

  • logR_os on the data after oversampling (creating copies), the minority class.

  • logR_smote using SMOTE to develop synthetic data.

Let's check their performance on the unseen data.

Accuracy score

Let’s get the accuracy scores of the individual models first and proceed with logR, which was trained on the original data.

Python 3.8
print("************************************************************")
print("Trained on imbalanced data.")
print("************************************************************")
print("Accuracy Score :",logR.score(X_unseen,y_unseen))
print("The confusion matrix")
print(metrics.confusion_matrix(y_unseen,logR.predict(X_unseen)))
print("The classification report.")
print(metrics.classification_report(y_unseen,logR.predict(X_unseen)))

With logR_os, which is trained on the data after oversampling (creating copies), the scores for the minority class will be fetched as follows:

Python 3.8
print("************************************************************")
print("Trained on oversampled with replacement.")
print("************************************************************")
print("Accuracy Score :",logR_os.score(X_unseen,y_unseen))
print("The confusion matrix")
print(metrics.confusion_matrix(y_unseen,logR_os.predict(X_unseen)))
print("The classification report.")
print(metrics.classification_report(y_unseen,logR_os.predict(X_unseen)))

Finally, we’ll use logR_smote, which was trained using SMOTE to create synthetic data.

Python 3.8
print("************************************************************")
print("Trained on oversampled using SMOTE")
print("************************************************************")
print("Accuracy Score :",logR_smote.score(X_unseen,y_unseen))
print("The confusion matrix")
print(metrics.confusion_matrix(y_unseen,logR_smote.predict(X_unseen)))
print("The classification report.")
print(metrics.classification_report(y_unseen,logR_smote.predict(X_unseen)))

Changing the performance matrix is helpful, and for general purposes, AUC-ROC is useful.

Area under ROC

Let's start with predicting class probabilities of the logR.

Python 3.8
from sklearn.metrics import roc_auc_score
# Imbalanced data
logR_prob = logR.predict_proba(X_unseen) # Predicting class probabilities
logR_prob_1 = [p[1] for p in logR_prob] # Keep only for class 1
print("************************************************************")
print("Trained on imbalanced data.")
print("The AUC-ROC: {}".format(roc_auc_score(y_unseen, logR_prob_1)))

Now, let’s get the class probabilities of the model (that is, logR_os) trained on oversampling (copies only).

Python 3.8
logR_os_prob = logR_os.predict_proba(X_unseen) # Predicting class probabilities
logR_os_prob_1 = [p[1] for p in logR_os_prob] # Keep only for class 1
print("************************************************************")
print("Trained on oversampled with replacement.")
print("The AUC-ROC: {}".format(roc_auc_score(y_unseen, logR_os_prob_1)))

Finally, the class probabilities of the logR_smote.

Python 3.8
logR_smote_prob = logR_smote.predict_proba(X_unseen) # Predicting class probabilities
logR_smote_prob_1 = [p[1] for p in logR_smote_prob] # Keep only for class 1
print("************************************************************")
print("Trained on oversampled using SMOTE")
print("The AUC-ROC: {}".format(roc_auc_score(y_unseen, logR_smote_prob_1)))

Cohen's kappa coefficient ...