Boosting is a potent and widespread class of ensemble learning strategies. 

Historically, boosting algorithms were a challenge to implement, and it was not till AdaBoost illustrated how to implement boosting that the strategy could be leveraged effectively. 

AdaBoost and modern gradient boosting work by sequentially including models that rectify the residual prediction errors of the model. As such, boosting methods are popular to be effective, but developing models can be slow, particularly for big datasets. 

More lately, extensions developed for computational efficiency have made the methods quick enough for wider adoption. Open-source implementations like XGBoost and LightGBM, have implied that booting algorithms have become the preferred and often top-performing strategy within machine learning competitions for classification and regression on tabular information. 

In this guide, you will find out about the essence of boosting to ML ensembles. 

After going through this guide, you will be aware of: 

1] The boosting ensemble method for ML incrementally includes weak learners trained on weighted variants of the training dataset. 

2] The basic idea that underlies all boosting algorithms and the critical approach leveraged within every boosting algorithm. 

3] How the basic ideas that underlie boosting could be looked into on new forecasting modelling projects. 

Tutorial Summarization 

This tutorial is subdivided into four portions, which are: 

1] Boosting Ensembles 

2] Essence of Boosting Ensembles 

3] Family of Boosting Ensemble Algorithms 

• AdaBoost Ensembles
• Classic Gradient Boosting Ensembles
• Modern Gradient Boosting Ensembles 

4] Customized Boosting Ensembles 

Boosting Ensembles 

Boosting is a potent ensemble learning strategy. 

As such, boosting is widespread and might be the most broadly leveraged ensemble strategies at the time of writing. 

Boosting is one of the most capable learning ideas put forth over the course of the previous two decades. 

As an ensemble strategy, it can read and sound more complicated than sibling techniques, like bootstrap aggregation (bagging) and stacked generalization (stacking). The implementations, can, as a matter of fact, be quite complex, however the concepts that are behind boosting ensembles are very simplistic. 

Boosting can be comprehended by comparing it to bagging. 

Within bagging, an ensemble is developed by making several differing samples of the same training dataset and fitting a decision tree on each. Provided that every sample of the training dataset is differing, every decision tree is differing, in turn making slightly differing forecasts and prediction errors. The forecasts for all of the created decision trees are brought together, having the outcome of reduced errors than fitting a singular tree. 

Boosting functions in a similar fashion. Several trees are fit on differing versions of the training dataset and the forecasts from the trees are brought together leveraging simple voting for classification or averaging for regression to have the outcome of a better prediction than fitting a singular decision tree. 

Boosting brings together an ensemble of weak classifiers leveraging simple majority voting. 

There are a few critical variations, which are: 

• Examples within the training set are allocated a weight on the basis of difficulty 
• Learning algorithms might pay attention to instance weights. 
• Ensemble members are included sequentially. 

The first difference is the same training dataset is leveraged to train every decision tree. No sampling of the training dataset is executed. Rather, every instance within the training dataset (every row of data) is allocated a weight on the basis of how simple or difficult the ensemble identifies that instance to forecast. 

The main concept underlying this algorithm is to provide more focus to patterns that are more difficult to categorize. The amount of focus is quantified through a weight that is allocated to each pattern within the training set. 

This implies that rows that are simple to forecast leveraging the ensemble possess a small weight and rows that are tough to forecast accurately will possess a lot larger weight. 

Boosting functions in a similar fashion, except that the trees are grown in sequence, every tree is grown leveraging data from historically grown trees. Boosting does not consist of bootstrap sampling, rather every tree is fit on a modified version of the original data set. 

The second differentiation from bagging is that the base learning algorithm, for example, the decision tree must pay focus to the weightings of the training dataset. In turn, it implies that boosting is particularly developed to leverage decision trees as the base learner, or other algorithms that support a weighting of rows when building the model. 

The building of the model must allocate more focus to training instances proportional to their allocated weight. This implies that ensemble members are built in a biased fashion to make (or work hard to make) correct forecasts on heavily weighted instances. 

Lastly, the boosting ensemble is built slowly. Ensemble members are included in sequence, one, then another, and so on, till the ensemble has the desirable number of members. 

Critically, the weighting of the training dataset is updated on the basis of the capacity of the complete ensemble following every ensemble member is included. This makes sure that every member that is subsequently included functions hard to rectify errors made by the whole model on the training dataset. 

In boosting, though, the training dataset for every subsequent classifier increasingly concentrates on examples misclassified by prior generated classifiers. 

The contribution of every model to the final prediction is a weighted total of the performance of every model, e.g., a weighted average or weighted vote. 

This incremental inclusion of ensemble members to rectify errors within the training dataset sound like it would ultimately overfit the training dataset. Practically, boosting ensembles could overfit the training dataset, but usually, the impact is subtle and overfitting is not a dominant issue. 

Not like bagging and random forests, boosting can overfit if [the number of the trees] is too big, even though this overfitting has a tendency to happen gradually if at all. 

This is a high-level summarization of the boosting ensemble method and closely is like the AdaBoost algorithm, however, we can go about generalizing the approach and extract the basic elements. 

Essence of Boosting Ensembles 

The essence of boosting sounds a lot like it might have been about rectifying forecasts. 

This is how all modern boosting algorithms are implemented, and it is a fascinating and critical concept. Nonetheless, rectifying forecasting errors might be viewed an implementation detail for accomplishing boosting (a large and critical detail) instead of the essence of the boosting ensemble approach. 

The essence of boosting is the combo of multiple weak learners into a strong learner. 

The technique of boosting is the combo of several weak learners into a robust learner. 

The technique of boosting, and ensembles of classifiers, it to learn several weak classifiers and bring them together in some way, rather than attempting to learn a singular strong classifier. 

A weak learner is a model that has a really modest skill, often implying that its performance is a bit above an arbitrary classifier for binary classification or forecasting the mean value for regression. Conventionally, this implies a decision stump, which is a decision tree that takes up one value of a singular variable and makes a forecast. 

A weak learner (WL) is a learning algorithm with the potential to generate classifiers with probability of error strictly (but only merely) less than that of arbitrary guessing. 

A weak learner can be compared to a robust learner that has good performance on a predictive modelling problem. Generally, we look for a strong learner to address a classification or regression problem. 

A strong learner (SL) is capable (provided adequate training data) to yield classifiers with randomly small error probability, 

Even though we look for a strong learner for a provided predictive modelling problem, they are a challenge to train. While weak learners are very quick and simple to train. 

Boosting identifies this distinction and proposes explicitly building a strong learner from several weak learners. 

Boosting is a class of ML methods on the basis of the concept that a combo of simple classifiers (gotten by a weak learner) can feature improved performance than any of the simple classifiers alone. 

Several approaches to boosting have been looked into, but only one has been really successful. That is the method detailed in the prior section where weak learners are included in sequence to the ensemble to particularly tackle or rectify the residual errors for regression trees or class label prediction errors for classification. The outcome is a strong learner. 

Let’s take a deeper look at ensemble methods that might be considered a part of the boosting family. 

Family of Boosting Ensemble Algorithms 

There are a massive number of boosting ensemble learning algorithms, even though all work in the generally a similar way. 

Specifically, they consist of sequentially including simplistic base learner models that are trained on (re-)weighted variants of the training dataset. 

The term boosting is a reference to a grouping of algorithms that are capable of converting weak learners to strong learners. 

We might take up three primary families of boosting methodologies, they are: AdaBoost, Classic Gradient Boosting, and Modern Gradient Boosting. 

The division is to an extent random, as there are strategies that may span all groups or implementations that can be configured to realize an instance from every group and even bagging-based methods. 

AdaBoost Ensembles 

Initially, naïve boosting methods looked into training weak classifiers on separate samples of the training dataset and bringing together the predictions. 

These methods were not successfully contrasted to bagging. 

Adaptive Boosting, or AdaBoost for short, were the first successful implementations of boosting. 

Researchers had issues for a duration of time to identify an effective implementation of boosting theory, until Freund and Schapire collaborated to generate the AdaBoost algorithm. 

It was not just a successful realization of the boosting principle, it was an efficient algorithm with regards to classification.  

Boosting, particularly in the form of the AdaBoost algorithm, was demonstrated to be a capable forecasting tool, typically outperforming any individual model. Its success brought attention from the modelling community and its use became broader spread. 

Even though AdaBoost was produced initially for binary classification, it was later extended for multi-class classification, regression, and a myriad of other extensions and specialized versions. 

They were the first to maintain the same training dataset and to put forth weighting of training instances and the sequential addition of models that have received training to rectify prediction errors of the ensemble. 

Classic Gradient Boosting Instances 

After the success of AdaBoost, a ton of focus was paid to boosting strategies. 

Gradient boosting was a generalization of the AdaBoost grouping of strategies that enabled the training of every subsequent base learning to be accomplished leveraging arbitrary loss functions. 

Instead of deriving new variants of boosting for each differing loss function, it is doable to derive a generic version, referred to as gradient boosting. 

The gradient in gradient boosting is a reference to the prediction error from a selected loss function, which is reduced by including base learners.  

The fundamental principles of gradient boosting are as follows: provided a loss function, for instance, squared error for regression – and a weak learner – for example, regression trees, the algorithm looks to identify an additive model that reduces the loss function. 

Following the preliminary reframing of AdaBoost as gradient boosting and the leveraging of alternate loss functions, there was a ton of subsequent innovation, like Multivariate Adaptive Regression Trees (MART), Tree Boosting, and Gradient Boosting Machines (GBM) 

If we bring together the gradient boosting algorithm with shallow regression trees, we obtain a model referred to as MART […] after fitting a regression tree to the residual (negative gradient), we re-estimate the parameters at the leaves of the tree to reduce the loss. 

The strategy was extended to add in regularization in an effort to further slow down the learning and sampling of rows and columns for every decision tree in order to include some independence to the ensemble members, on the basis of concepts from bagging, referenced to as stochastic gradient boosting. 

Modern Gradient Boosting Ensembles 

Gradient boosting and variances of the method were demonstrated to be really effective, yet were typically slow to train, particularly for big training datasets. 

This was primarily owing to the sequential training of ensemble members, which could not be parallelized. This was unlucky, as training ensemble members in parallel and the computational speed-up it furnishes is an often described desirable attribute of leveraging ensembles. 

As such, a lot of effort was invested into enhancing the computational efficiency of the method. 

This had the outcome of highly optimised open-source implementations of gradient boosting that put forth innovative strategies that both accelerated the training of the model and offered further enhanced forecasting performance. 

Notable instances consisted of both Extreme Gradient Boosting (XGBoost) and the Light Gradient Boosting Machine (LightGBM) projects. Both were so efficient that they became de facto strategies leveraging in machine learning competitions when operating with tabular data. 

Customized Boosting Ensembles 

We have reviewed the canonical types of boosting algorithms in brief. 

Modern implementations of algorithms like XGBoost and LightGBM furnish adequate configuration hyperparameters to realize several differing variants of boosting algorithms. 

Even though boosting was at the start a challenge to realize in contrast to simpler to implement methods such as bagging, the basic ideas might be useful in looking into or extending ensemble methods on your own predictive modelling projects in the future. 

The basic concept of building a strong learner from weak learners could have implementation in several differing fashions. For instance, bagging with decision stumps or other similarly weak learner configurations of conventional machine learning algorithms could be taken up as a realization of this approach. 

• Bagging weak learners 

It also furnishes a contrast to other ensemble variants, like stacking, that makes an effort bring together several strong learners into a bit stronger learner. Even in that case, probably alternate success could be accomplished on a project by stacking diverse weak learners instead. 

• Stacking weak learners 

The path to efficient boosting consists of particular implementation details of weighted training instances, models that can honour the weightings, and the sequential inclusion of models fit under some loss minimization strategy. 

Nonetheless, these principles could be leveraged in ensemble models more typically. 

For instance, probably members can be included to a bagging or stacking ensemble in sequence and only kept if they have the outcome of a useful elevation in skill, drop in prediction error, or alteration in the distribution of predictions made by the model. 

• Sequential bagging or stacking 

In a few senses, stacking provides a realization of the concept of correcting forecasts of other models. The meta-model (level-1 learner) makes an effort to efficiently bring together the predictions of the base models (level-0 learners). In a sense, it is making an effort to rectify the predictions of those models.  

Levels of the stacked model could be included to tackle particular necessities, like the minimization of some or all forecasting errors. 

• Deep Stacking of models 

These are a few perhaps overt instances of how the essence of the boosting strategy can be looked into, hopefully serving as inspiration to further ideas.  

Further Reading 

This section furnishes additional resources on the topic if you desire to delve deeper. 

Books 

Pattern Classification Using Ensemble Methods, 2010 

Ensemble Methods, 2012 

Ensemble Machine Learning, 2012 

The Elements of Statistical Learning, 2016 

An intro to Statistical Learning with Applications in R, 2014 

Machine Learning: A Probabilistic Perspective, 2012 

Applied Predictive Modelling, 2013 

Articles 

Boosting (machine learning), Wikipedia 

Conclusion 

In this guide, you found out about the essence of boosting to machine learning ensembles. 

Particularly, you learned: 

• The boosting ensemble method for machine learning incrementally includes weak learners trained on weighted variants of the training dataset 
• The basic idea that underlies all boosting algorithms and the critical approach leveraged within every boosting algorithm 
• How the basic ideas that are behind boosting could be looked into on fresh predictive modelling projects.