Ensemble member selection references to algorithms that optimize the composition of an ensemble.

This may consist of growing an ensemble from available models or pruning members from a fully defined ensemble.

The objective is typically to reduce the model or computational complexity of an ensemble with minimal or no impact on the performance of an ensemble, and in a few scenarios identify a combo of ensemble members that has the outcome of improved performance than blindly leveraging all contributing models directly.

In this guide, you will find out how to develop ensemble selection algorithms from the ground up.

After going through this guide, you will be aware of:

• Ensemble selection consists of selecting a subset of ensemble members that has the outcome of reduced complexity than leveraging all members and at times improved performance.
• How to develop and assess a greedy ensemble pruning algorithm for classification.
• How to develop and assess an algorithm for greedily growing an ensemble from the ground up.

Tutorial Summarization

This guide is subdivided into four portions, which are:

1] Ensemble Member Selection

2] Baseline Models and Voting

3] Ensemble Pruning Example

4] Ensemble Growing Example

Ensemble Member Selection

Voting and stacking ensembles usually bring together the predictions from a heterogenous group of model variants.

Even though the ensemble might possess a large number of ensemble members, it is tough to know that the ideal combo of members is being leveraged by the ensemble. For instance, rather than merely leveraging all members, it is possible that improved results could be accomplished by including one more differing model type or removing one or additional models.

This can be tackled leveraging a weighted average ensemble and leveraging an optimization algorithm to identify an appropriate weighting for every member, enabling some members to have a zero weight, which essentially removes them from the ensemble. The issue with a weighted average ensemble is that all models stay part of the ensemble, probably needing an ensemble of bigger complexity than is needed to be generated and maintained.

An alternate strategy is to optimize the composition of the ensemble itself. The general strategy of automatically selecting or optimizing the members of ensembles is referenced to as ensemble selection.

Two typical approaches consist of ensemble growing and ensemble pruning.

• Ensemble Growing: Include members to the ensemble till no further improvement is observed.
• Ensemble Pruning: Remove members from the ensemble till no further improvement is observed.

Ensemble Growing is a strategy where the model begins with no members and consists of including new members till no further improvement is observed. This could be carried out in a greedy fashion where members are included one at a time just if they have the outcome of an improvement in model performance.

Ensemble pruning is a strategy where the model begins with all potential members that are being considered and removes members from the ensemble till no further enhancement is observed. This could be carried out in a greedy manner where members are removed one at a time and only if their removal outcomes in a lift in the performance of the overall ensemble.

Provided a grouping of trained individual learners, rather than combining all of them, ensemble pruning attempts to choose a subset of individual learners to make up the ensemble.

A benefit of ensemble pruning and growing is that it might have the outcome of an ensemble with a smaller size (reduced complexity) and/or an ensemble with improved predictive performance. At times a small drop in performance is desirable if it can be accomplished in a large drop in model complexity and resulting maintenance burden. Alternately, on some projects, predictive ability is more critical than all other concerns, and ensemble selection furnishes one additional technique to attempt and obtain the most out of the contributing models.

There are two primary reasons for minimizing the ensemble size: a) Reducing computational overhead: Smaller ensembles need reduced computational overhead and b) enhancing precision. Some members in the ensemble might reduce the predictive performance of the whole.

Ensemble growing may be preferred for computational efficiency purposes in scenarios where a small number of ensemble members are expected to feature better performance, while ensemble pruning would be more effective and efficient in scenarios where a major number of ensemble members might be expected to feature better performance.

Simplistic greedy ensemble growing and pruning possess a lot in common with stepwise feature selection strategies, like those leveraged in regression (for example, so-called stepwise regression).

More sophisticated strategies might be leveraged, like choosing members for addition to or removal from the ensemble based on their standalone performance on the dataset, or even via the leveraging of a global search process that attempts to identify a combo of ensemble members that has the outcome of improved global performance.

One can carry out heuristic search in the space of the possible different ensemble subsets while assessing the collective merit of a candidate subset.

Now that we are acquainted with ensemble selection strategies, let’s look into how we might implement ensemble pruning and ensemble growing within scikit-learn.

Baseline Models and Voting

Prior to diving into developing growing and pruning ensembles, let’s first go about establishing a dataset and baseline.

We will leverage a synthetic binary classification problem as the foundation for this investigation, defined by the make_classification() function with 5,000 instances and 20 numerical input features.

The instance below gives definition to the dataset and summarizes its size.

Running the instance develops the dataset in a repeatable fashion and reports the number of rows and input features, matching our expectations.

(5000, 20) (5000,)

Then, we can select some candidate models that will furnish the foundation for our ensemble.

We will leverage five standard ML models, which includes logistic regression, naïve Bayes, decision tree, support vector machine, and a k-nearest neighbour algorithm.

To start with, we can give definition to a function that will develop every model with default hyperparameters. Every model will be defined as a tuple with a name and the model object, then included to a list. This is a useful structure both for enumeration of the models with their names for standalone evaluation and for subsequent leveraging within an ensemble.

The get_models() function below implements this and returns the list of models to consider:

We can then go about defining a function that takes a singular model and the dataset and assesses the performance of the model on the dataset. We will assess a model leveraging repeated stratified k-fold cross-validation with ten folds and three repeats, best practice within machine learning.

The evaluate_model() function below implements this and gives back a listing of scores across all folds and repeats.

We can then develop the list of models and enumerate them, reporting the performance of each on the synthetic dataset in turn.

Connecting this together, the complete instance is detailed below.

# evaluate standard models on the synthetic dataset

from numpy import mean

from numpy import std

from sklearn.datasets import make_classification

from sklearn.model_selection import cross_val_score

from sklearn.model_selection import RepeatedStratifiedKFold

from sklearn.linear_model import LogisticRegression

from sklearn.neighbors import KNeighborsClassifier

from sklearn.tree import DecisionTreeClassifier

from sklearn.svm import SVC

from sklearn.naive_bayes import GaussianNB

from matplotlib import pyplot

# get the dataset

def get_dataset():

X, y = make_classification(n_samples=5000, n_features=20, n_informative=10, n_redundant=10, random_state=1)

return X, y

# get a list of models to evaluate

def get_models():

models = list()

models.append((‘lr’, LogisticRegression()))

models.append((‘knn’, KNeighborsClassifier()))

models.append((‘tree’, DecisionTreeClassifier()))

models.append((‘nb’, GaussianNB()))

models.append((‘svm’, SVC(probability=True)))

return models

# evaluate a give model using cross-validation

def evaluate_model(model, X, y):

# define the model evaluation procedure

cv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)

# evaluate the model

scores = cross_val_score(model, X, y, scoring=’accuracy’, cv=cv, n_jobs=-1)

return scores

# define dataset

X, y = get_dataset()

# get the models to evaluate

models = get_models()

# evaluate the models and store results

results, names = list(), list()

for name, model in models:

# evaluate model

scores = evaluate_model(model, X, y)

# store results

results.append(scores)

names.append(name)

# summarize result

print(‘>%s %.3f (%.3f)’ % (name, mean(scores), std(scores)))

# plot model performance for comparison

pyplot.boxplot(results, labels=names, showmeans=True)

pyplot.show()

Running the instance evaluates every standalone ML algorithm on the synthetic binary classification dataset.

Your outcomes might demonstrate variance provided the stochastic nature of the algorithm or evaluation process, or variations in numerical accuracy. Take up running the instance a few times and contrast the average outcome.

In this scenario, we can observe that both the KNN and SVM models have the best performance on this dataset, accomplishing a mean classification precision of approximately 95.3%.

These outcomes furnish a baseline in performance that we need any ensemble to exceed in order to be considered useful on this dataset.

A figure is developed demonstrating box and whisker plots of the distribution of precision scores for every algorithm.

We can observe that the KNN and SVM algorithms have much better performance than the other algorithms, even though all algorithms are skilful in differing ways. This might make them good candidates to consider within an ensemble.

Then, we are required to establish a baseline ensemble that leverages all models. This will furnish a point of comparison with growing and pruning strategies that look for improved performance with a smaller subset of models.

In this scenario, we will leverage a voting ensemble with soft voting. This implies that every model will forecast probabilities and the probabilities will be summed by the ensemble model to select a final output prediction for every input sample.

This can be accomplished leveraging the VotingClassifier class where the members are set through the “estimators” argument, which expects a listing of models where every model is a tuple with a name and configured model object, just as we defined in the prior section.

We can then set the type of voting to perform through the “voting” argument, which in this scenario is set to “soft”

Connecting this together, the instance below evaluates a voting ensemble of all five models on the synthetic binary classification dataset.

Running the instance evaluates the soft voting ensemble of all models leveraging repeated stratified k-fold cross-validation and reports the mean precision across all folds and repeats.

Your outcomes may demonstrate variance provided the stochastic nature of the algorithm or evaluation procedures, or variations in numerical accuracy. Consider running the instance a few times and contrast the average outcome.

In this scenario, we can observe that the voting ensemble accomplished a mean precision of about 92.8%. This is lower than SVM and KNN models leveraged alone that accomplished a precision of approximately 95.3%.

This outcome highlights that a simple voting ensemble of all models have the outcome of a model with increased complexity and worse performance in this scenario. Probably we can identify a subset of members that have better performance than any singular model and has reduced complexity than merely leveraging all models.

Mean Accuracy: 0.928 (0.012)

Then, we will explore pruning members from the ensemble.

Ensemble Pruning Instance

In this portion of the blog, we will look into how to develop a greedy ensemble pruning algorithm from the ground up.

We will leverage a greedy algorithm in this scenario, which is straightforward to implement. This consists of removing a single member from the ensemble and evaluating the performance and repeating this for every member within the ensemble. The member, that, if removed, has the outcome of the best improvement in performance is then permanently removed from the ensemble and the procedure repeats. This goes on till no subsequent improvements can be accomplished.

It is referenced to as a “greedy” algorithm as it seeks the best improvement at every step, it is possible that the best combo of members is not on the path of greedy enhancements, in which scenario the greedy algorithm will not identify it and a global optimization algorithm could be leveraged instead.

To start with, we can give definition to a function that evaluates a candidate list of models. This function will take the listing of models and the dataset and develop a voting ensemble from the listing of models and evaluate its performance leveraging repeated stratified k-fold cross-validation, returning the mean classification precision.

This function can be leveraged to evaluate every candidate’s removal from the ensemble. The evaluate_ensemble() function below implements this.

Then, we can define a function that performs a singular round of pruning.

To start with, a baseline in performance is determined with all models that are presently in the ensemble. Then, the listing of models is enumerated and every one is removed in turn, and the impact of removing the model is evaluated on the ensemble. If the removal has the outcome of an improvement in performance, the new score and particular model that was removed is recorded.

Critically, the trial removal is carried out on a copy of the listing of models, not on the primary listing of models itself. This is to make sure we just remove an ensemble member from the listing once we know it will have the outcome of the best potential improvement from all the members that could possibly be removed at the present step.

The prune_round() function below implements this provided the listing of present models in the ensemble and dataset, and returns the improvement in score (if any) and the ideal model to remove to accomplish that score.

Then, we are required to drive the pruning process.

This consists of running a round of pruning till no further improvement in precision is accomplished by calling the prune_round() function repeatedly.

If the function returns none for the model to be removed, we know that no singular greedy improvement is doable and we can return the final listing of models. Otherwise, the selected model is removed from the ensemble and the process goes on.

The prune_ensemble() function below implements this and returns the models to leverage in the final ensemble and the score that it accomplished through our evaluation procedure.

We can connect all of this together into an instance of ensemble pruning on the synthetic binary classification dataset.

# example of ensemble pruning for classification

from numpy import mean

from numpy import std

from sklearn.datasets import make_classification

from sklearn.model_selection import cross_val_score

from sklearn.model_selection import RepeatedStratifiedKFold

from sklearn.linear_model import LogisticRegression

from sklearn.neighbors import KNeighborsClassifier

from sklearn.tree import DecisionTreeClassifier

from sklearn.svm import SVC

from sklearn.naive_bayes import GaussianNB

from sklearn.ensemble import VotingClassifier

from matplotlib import pyplot

# get the dataset

def get_dataset():

X, y = make_classification(n_samples=5000, n_features=20, n_informative=10, n_redundant=10, random_state=1)

return X, y

# get a list of models to evaluate

def get_models():

models = list()

models.append((‘lr’, LogisticRegression()))

models.append((‘knn’, KNeighborsClassifier()))

models.append((‘tree’, DecisionTreeClassifier()))

models.append((‘nb’, GaussianNB()))

models.append((‘svm’, SVC(probability=True)))

return models

# evaluate a list of models

def evaluate_ensemble(models, X, y):

# check for no models

if len(models) == 0:

return 0.0

# create the ensemble

ensemble = VotingClassifier(estimators=models, voting=’soft’)

# define the evaluation procedure

cv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)

# evaluate the ensemble

scores = cross_val_score(ensemble, X, y, scoring=’accuracy’, cv=cv, n_jobs=-1)

# return mean score

return mean(scores)

# perform a single round of pruning the ensemble

def prune_round(models_in, X, y):

# establish a baseline

baseline = evaluate_ensemble(models_in, X, y)

best_score, removed = baseline, None

# enumerate removing each candidate and see if we can improve performance

for m in models_in:

# copy the list of chosen models

dup = models_in.copy()

# remove this model

dup.remove(m)

# evaluate new ensemble

result = evaluate_ensemble(dup, X, y)

# check for new best

if result > best_score:

# store the new best

best_score, removed = result, m

return best_score, removed

# prune an ensemble from scratch

def prune_ensemble(models, X, y):

best_score = 0.0

# prune ensemble until no further improvement

while True:

# remove one model to the ensemble

score, removed = prune_round(models, X, y)

# check for no improvement

if removed is None:

print(‘>no further improvement’)

break

# keep track of best score

best_score = score

# remove model from the list

models.remove(removed)

# report results along the way

print(‘>%.3f (removed: %s)’ % (score, removed[0]))

return best_score, models

# define dataset

X, y = get_dataset()

# get the models to evaluate

models = get_models()

# prune the ensemble

score, model_list = prune_ensemble(models, X, y)

names = ‘,’.join([n for n,_ in model_list])

print(‘Models: %s’ % names)

print(‘Final Mean Accuracy: %.3f’ % score)

Running the instance performs the ensemble pruning procedure, reporting which model was removed every round and the precision of the model after the model was removed.

Your outcomes may demonstrate variance provided the stochastic nature of the algorithm or evaluation procedure, or variations in numerical accuracy. Consider running the instance a few times and contrast the average outcome.

In this scenario, we can observe that three rounds of pruning were carried out, removing the naïve Bayes, decision tree, and logistic regression algorithms, leaving just the SVM and KNN algorithms that accomplished a mean classification precision of approximately 95.7%. This is better than the 95.3% accomplished by SVM and KNN leveraged in a standalone fashion, and clearly better than bringing all models together.

The final listing of models could then be leveraged in a new final voting ensemble model through the “estimators” argument, fit on the entire dataset and leverage to make a prediction on new data.

Now that we are acquainted with developing and evaluating an ensemble pruning strategy, let’s observe the reverse case of growing the ensemble members.

Ensemble Growing Instance

In this portion of the blog, we will look into how to develop a greedy ensemble growing algorithm from the ground up.

The structure of greedily growing an ensemble is a lot like the greedy pruning of members, even though in reverse. We begin with an ensemble with no models and include a singular model that has the best performance. Models are then included one by one if they have the outcome of a lift in the performance over the ensemble prior to the inclusion of the model.

A lot of the code is the same as the pruning scenario so we can concentrate on the differences.

To start with, we must go about defining a function to carry out one round of growing the ensemble. This consists of enumerating all candidate models that could be included and evaluating the impact of including each in turn to the ensemble. The singular model that has the outcome of the biggest improvement is then returned by the function, combined with its score.

The grow_round() function below implements this behaviour.

Then, we require a function to drive the growing procedure.

This consists of a loop that runs rounds of growing till no further additions can be made having the outcome of an improvement in model performance. For every addition that can be made, the primary listing of models in the ensemble is updated and the listing of models presently in the ensemble is reported along with the performance.

The grow_ensemble() function implements this and returns the listing of models greedily determined to have the outcome of the best performance along with the expected mean precision.

Connecting this together, the complete instance of greedy ensemble growing on the synthetic binary classification dataset is detailed here.

Running the instance incrementally includes a single model at a time to the ensemble and reports the mean classification precision of the ensemble of the models.

Your outcomes may demonstrate variance provided the stochastic nature of the algorithm or evaluation procedure, or variations in numerical accuracy. Take up running the instance a few times and contrast the average outcome.

In this scenario, we can observe that ensemble growing identified the same answer as greedy ensemble pruning where an ensemble of SVM and KNN accomplished a mean classification precision of approximately 95.6%, an enhancement over any singular standalone model and over combining all models.

Further Reading

Books

Ensemble Methods, Foundations and Algorithms, 2012

Pattern Classification Using Ensemble Methods, 2010

APIs

sklearn.ensemble.VotingClassifer API

Conclusion

In this guide, you found out how to develop ensemble selection algorithms from the ground up.

Particularly, you learned:

• Ensemble methods consists of selecting a subset of ensemble members that has the outcome of reduced complexity than leveraging all members and at times improved performance.
• How to generate and evaluate a greedy ensemble pruning algorithm for classification.
• How to develop and evaluate an algorithm for greedily growing an ensemble from the ground up.