Gradient Boosting Tree

In this lesson, we show you another type of ensemble method, the gradient boosting tree.

What is an ensemble method?

Ensemble methods are a group of Machine Learning methods that use multiple learning algorithms to gain better predictive performance than one single method. Generally speaking, there are three types of ensembleboosting, bagging, and stacking. In this course, we show ensemble and boosting. In this lesson, we will learn one method of boosting, gradient boosting tree (GBDT).

The core principle of bagging is to build many weak estimators; Each estimator was trained and predicted independently. The final result is the combination of their predictions. If it’s a regression, it’s the average of the results. If it’s a category, it’s a vote for the results. Random Forest is one of the methods, the normal decision tree is the weak estimator.

By contrast, in the boosting method, base estimators are built sequentially, and one tries to reduce the bias of the combined estimator. There is an interdependence between the weak estimators. The motivation is to combine several weak models to produce a powerful ensemble.

From the perspective of learning theory, bagging reduces the variance of the model, while boosting reduces the deviation of the model.

Before we get hands-on, let’s look at the GBDT first. It helps us understand the algorithm a little bit better.

Create a free account to view this lesson.

By signing up, you agree to Educative's Terms of Service and Privacy Policy