How Does Grid Search Work

The following list is the optimization steps and procedures that the grid search method must follow to find the combination of hyperparameter values that produce the best ML model performance:

1. Define the hyperparameters and their possible values.

2. Create a grid of all possible combinations of hyperparameter values.

3. Train a model using every possible combination of hyperparameter values.

4. Evaluate and record the performance of the ML model.

5. Select the best-performing combination of hyperparameters.

6. Train the final ML model with selected hyperparameters.

1. Define the hyperparameters and their possible values

This is the first step for the grid search method; we need to specify the range of values for each hyperparameter selected for the optimization process. First, it is important to identify the specific ML algorithm for which the hyperparameters need optimization to produce the best performance. The easy way is to read the documentation of the selected ML algorithm. For example, sklearn’s random forest algorithm has about 18 hyperparameters that can be combined. You can find its documentation provided by the scikit-learn library.

Examples of hyperparameters from the random forest algorithm are as follows:

• n_estimators: This is the number of trees in the forest.

• criterion: This is the function to measure the quality of a split.

• max_depth: This is the maximum depth of the tree.

• min_sample_split: This represents the minimum number of samples required to split an internal node.

• max_features : This is the number of features to consider when looking for the best split.

2. Create a grid of all possible combinations of hyperparameter values

The second step is to create a grid of possible combinations of values for each hyperparameter. This can be done by specifying a range of values or a set of discrete values. For example, n_estimators and criterion from the random forest algorithm can have a range of values as follows:

• n_estimators = [50,100,250]

• criterion= ["gini", "entropy", "log_loss"]

With these two hyperparameters, we can have a total of nine combinations of hyperparameters for the optimization process. These combinations are (50, "gini"), (50, "entropy"), (50, "log_loss"), (100, "gini"), (100, "entropy"), (100, "log_loss"), (250, "gini") , and (250, "entropy"), (250, "log_loss"). Keep in mind that the more hyperparameters and their values we add, the more combinations of hyperparameters we create for the optimization process.