Search⌘ K
Home
Courses
Data Science Projects with Python

Cross-Validation

Explore the concept of cross-validation and understand its role in selecting hyperparameters like the regularization parameter for logistic regression. Learn how to use scikit-learn's KFold and StratifiedKFold to create training and test folds that help evaluate model performance reliably. Understand the importance of data splitting, shuffling, stratification, and how these practices reduce overfitting and improve model generalization. Gain practical skills for implementing k-fold and leave-one-out cross-validation to optimize predictive models.

We'll cover the following...

Choosing the regularization parameter

By now, you may suspect that we could use regularization in order to decrease the overfitting we observed when we tried to model the synthetic data in Exercise: Generating and Modeling Synthetic Classification Data. The question is, how do we choose the regularization parameter CC?, CC is an example of a model hyperparameter. Hyperparameters are different from the parameters that are estimated when a model is trained, such as the coefficients and the intercept of a logistic regression. Rather than being estimated by an automated procedure like the parameters are, hyperparameters are input directly by the user as keyword arguments, typically when instantiating the model class. So, how do we know what values to choose?

Hyperparameters are more difficult to estimate than parameters. This is because it is up to the data scientist to determine what the best value is, as opposed to letting an optimization algorithm find it. However, it is possible to programmatically choose hyperparameter values, which could be viewed as an optimization procedure in its own right. Practically speaking, in the case of the regularization parameter CC, this is most commonly done by fitting the model on one set of data with a particular value of CC, determining model training performance, and then assessing the out-of-sample performance on another set of data.

We are already familiar with the concept of using model training and test sets. However, there is a key difference here; for instance, what would happen if we were to use the test set multiple times ...