Train/Validation/Test Splits
Understand how to correctly split datasets into training, validation, and test sets to build reliable machine learning models. Explore the role of these splits in preventing data leakage, guiding model selection, and preserving unbiased evaluation. Learn practices to maintain separation of data during preprocessing and model tuning, ensuring trustworthy results on unseen data.
We'll cover the following...
A churn model can appear to improve after every tweak when we repeatedly inspect its test accuracy and keep the version with the highest score. The score stops being an independent check because model choices now reflect the particular customers in the test set. We have effectively learned from data that was meant to represent future, unseen customers.
A disciplined split assigns each row one role:
The training set supplies the examples used to fit model parameters.
The validation set supplies the examples used to compare choices, such as several values of
Cfor logistic regression.The test set stays isolated until development ends and provides the final estimate of performance on unseen data.
Accuracy measures the fraction of correctly predicted churn labels. It is sufficient for this workflow example, although a real churn decision may need metrics that reflect the cost of contacting a customer who would not leave or missing a customer who would leave.
The contract is strict. We may fit and revise models using training and validation data. We touch the test set once, after choosing the final model.
Build a split before fitting
The churn-like dataset below contains numeric customer information and the binary target churned. The complete pipeline creates the data, makes the test set unavailable during development, fits a baseline logistic regression model on ...