Optimization: Bayesian Methods

What is Bayesian optimization?

Bayesian optimization is a powerful and efficient method for optimizing black box functions that are expensive to evaluate. It combines techniques from Bayesian inference and optimization to sequentially search for the optimal solution by intelligently selecting the next set of parameters to evaluate based on past observations.

The key idea behind Bayesian optimization is to build a probabilistic surrogate model, typically a GP, that captures the underlying unknown function. The GP model is initially fitted to a few observations or evaluations of the objective function. It provides not only predictions of the function values at unexplored locations, but also estimates of the uncertainty or confidence in those predictions.

The algorithm proceeds by iteratively selecting the next set of parameters to evaluate based on an acquisition function, which balances the exploration of uncertain regions of the search space and the exploitation of promising areas where the function is expected to perform well. The acquisition function is derived from the surrogate model and takes into account both the predicted function values and their uncertainties.

Once a new evaluation is made, the surrogate model is updated to incorporate the new observation, and the process continues. By efficiently selecting new evaluations based on the acquisition function, Bayesian optimization can quickly narrow down the search space and converge to the optimal solution with a minimal number of evaluations.