Search⌘ K
Home
Courses
Fundamentals of Machine Learning: A Pythonic Introduction

Validation

Get introduced to the importance of data splits and the process of cross-validation.

We'll cover the following...

Since regularization is a method to fine-tune the subject model by introducing an additional penalty in the error function, we need to validate its impact. Several hyperparameters need to be set before optimizing the objective function. The hyperparameters include model fwf_\bold w, loss function LL, regularization function RR, and the scale of regularization λ\lambda. Validation is the process of testing the accuracy of the trained model, which also measures the validity of the hyperparameters.

Note: An accurate indicator of generalization is the performance of the trained model on unseen dataThis is the data that isn’t used in the training process..

Data splits

Where to get the unseen data for validation? One way is to hold out a percentage of available data and use the rest for training. Once the training is complete, the validation can be carried out on the subset of available data that was kept for validation, known as the hold-out set.

Note: The more popular term used for hold-out set is test set.

Train-test split
Train-test split

How large should the test set be? To assess the generalization, we need the test set to be large. But we also need the training set to be large to avoid overfitting. There’s no exact workaround to this trade-off. A rule of thumb, however, is to use an 80/2080 percent for training and 20 percent for testing split.

To improve the performance after validation, the hyperparameters can be tuned. The validation and tuning cycle continues until the desired performance is achieved.

Validation cycle
Validation cycle

Validation set

After validation on the test set, if the hyperparameters are tuned and the training is carried out again, the test set is no longer unseen. It’s used in the training process but not directly as the training set.

If the test set must be unseen, then how to tune the hyperparameters?

A compromise in this situation is to make another split of the data called the validation set and use it for tuning the hyperparameters. This way, the test set can be kept unseen, and the performance of the final model can be reported on it.

Train-validation-test split
Train-validation-test split

Implementing data splits

It’s handy to have an understanding of how a dataset can be randomly split into three splits. In the code below, the variable X contains feature vectors as rows, and the variable y contains the targets as rows (assuming multi-target modeling).

Using numpy

There can be several ways of doing this data split; one way is to randomly permute the indices of the data points and then pick the percentages. This is easy to understand and implement. A handy way to do a validation split is to split the dataset into two splits, train and test, and then split the test set to get a validation set. The code below implements this idea:

Python 3.10.4
import numpy as np
# This function is responsible for splitting the given dataset in train and test segments
def split_data(X, y, test_size = 0.2):
  n_total = X.shape[0] # 0th index represents size of X
  n_test = int(test_size * n_total)
  idx_permuted = np.random.permutation(np.arange(n_total))
  X_test, y_test = X[idx_permuted[:n_test],:], y[idx_permuted[:n_test]]
  X_train, y_train = X[idx_permuted[n_test:],:], y[idx_permuted[n_test:]]
  return X_train, X_test, y_train, y_test
X = np.random.rand(100,10) # 100 data points in 10 features
y = np.random.rand(100,3) # 3 targets
# Dividing the whole dataset in train and test splits
X_train, X_test, y_train, y_test = split_data(X, y, test_size = 0.2)
# Further dividing the test split into two equal proportions that are test and validation
X_test, X_val, y_test, y_val = split_data(X_test, y_test, test_size = 0.5)
# Cross checking the train, test, and validation sets shapes
print ("{:<4} {:<10} {:<10} {:<10}".format(' ', 'Train','Test','Validation'))
print ("{:<4} {:<10} {:<10} {:<10}".format('X', str(X_train.shape), str(X_test.shape), str(X_val.shape)))
print ("{:<4} {:<10} {:<10} {:<10}".format('Y', str(y_train.shape), str(y_test.shape), str(y_val.shape)))

Here is the explanation for the code above:

  • Lines 4–7: We define the split_data function that splits the input arrays X and y into training and testing sets, where the testing set size is determined by the test_size argument (default 0.20.2 ...