Model Performance on the Test Set
Explore how to rigorously evaluate machine learning model performance on a reserved test set to estimate future predictive accuracy. Understand the importance of preventing data leakage, calculating ROC AUC metrics, and consider methods like learning curves to decide on training data usage. This lesson also guides you on preparing your tested model for client delivery and monitoring over time.
Rigorous estimate of expected future performance
We already have some idea of the out-of-sample performance of the XGBoost model from the validation set. However, the validation set was used in model fitting via early stopping. The most rigorous estimate of expected future performance we can make should be created with data that was not used at all for model fitting. This was the reason for reserving a test dataset from the model-building process.
You may notice that we did examine the test set to some extent already, for example, in the chapter "Data Exploration and Cleaning," when assessing data quality and cleaning data. The gold standard for predictive modeling is to set aside a test set at the very beginning of a project and not examine it at all until the model is finished. This is the easiest way to ensure that none of the knowledge from the test set has "leaked" into the training set during model development. When this happens, it opens up the possibility that the test set is no ...