Growing and pruning ensembles in Python
Ensemble member selection references to algorithms that optimize the composition of an ensemble.
This may consist of growing an ensemble from available models or pruning members from a fully defined ensemble.
The objective is typically to reduce the model or computational complexity of an ensemble with minimal or no impact on the performance of an ensemble, and in a few scenarios identify a combo of ensemble members that has the outcome of improved performance than blindly leveraging all contributing models directly.
In this guide, you will find out how to develop ensemble selection algorithms from the ground up.
After going through this guide, you will be aware of:
- Ensemble selection consists of selecting a subset of ensemble members that has the outcome of reduced complexity than leveraging all members and at times improved performance.
- How to develop and assess a greedy ensemble pruning algorithm for classification.
- How to develop and assess an algorithm for greedily growing an ensemble from the ground up.
Tutorial Summarization
This guide is subdivided into four portions, which are:
1] Ensemble Member Selection
2] Baseline Models and Voting
3] Ensemble Pruning Example
4] Ensemble Growing Example
Ensemble Member Selection
Voting and stacking ensembles usually bring together the predictions from a heterogenous group of model variants.
Even though the ensemble might possess a large number of ensemble members, it is tough to know that the ideal combo of members is being leveraged by the ensemble. For instance, rather than merely leveraging all members, it is possible that improved results could be accomplished by including one more differing model type or removing one or additional models.
This can be tackled leveraging a weighted average ensemble and leveraging an optimization algorithm to identify an appropriate weighting for every member, enabling some members to have a zero weight, which essentially removes them from the ensemble. The issue with a weighted average ensemble is that all models stay part of the ensemble, probably needing an ensemble of bigger complexity than is needed to be generated and maintained.
An alternate strategy is to optimize the composition of the ensemble itself. The general strategy of automatically selecting or optimizing the members of ensembles is referenced to as ensemble selection.
Two typical approaches consist of ensemble growing and ensemble pruning.
- Ensemble Growing: Include members to the ensemble till no further improvement is observed.
- Ensemble Pruning: Remove members from the ensemble till no further improvement is observed.
Ensemble Growing is a strategy where the model begins with no members and consists of including new members till no further improvement is observed. This could be carried out in a greedy fashion where members are included one at a time just if they have the outcome of an improvement in model performance.
Ensemble pruning is a strategy where the model begins with all potential members that are being considered and removes members from the ensemble till no further enhancement is observed. This could be carried out in a greedy manner where members are removed one at a time and only if their removal outcomes in a lift in the performance of the overall ensemble.
Provided a grouping of trained individual learners, rather than combining all of them, ensemble pruning attempts to choose a subset of individual learners to make up the ensemble.
A benefit of ensemble pruning and growing is that it might have the outcome of an ensemble with a smaller size (reduced complexity) and/or an ensemble with improved predictive performance. At times a small drop in performance is desirable if it can be accomplished in a large drop in model complexity and resulting maintenance burden. Alternately, on some projects, predictive ability is more critical than all other concerns, and ensemble selection furnishes one additional technique to attempt and obtain the most out of the contributing models.
There are two primary reasons for minimizing the ensemble size: a) Reducing computational overhead: Smaller ensembles need reduced computational overhead and b) enhancing precision. Some members in the ensemble might reduce the predictive performance of the whole.
Ensemble growing may be preferred for computational efficiency purposes in scenarios where a small number of ensemble members are expected to feature better performance, while ensemble pruning would be more effective and efficient in scenarios where a major number of ensemble members might be expected to feature better performance.
Simplistic greedy ensemble growing and pruning possess a lot in common with stepwise feature selection strategies, like those leveraged in regression (for example, so-called stepwise regression).
More sophisticated strategies might be leveraged, like choosing members for addition to or removal from the ensemble based on their standalone performance on the dataset, or even via the leveraging of a global search process that attempts to identify a combo of ensemble members that has the outcome of improved global performance.
One can carry out heuristic search in the space of the possible different ensemble subsets while assessing the collective merit of a candidate subset.
Now that we are acquainted with ensemble selection strategies, let’s look into how we might implement ensemble pruning and ensemble growing within scikit-learn.
Baseline Models and Voting
Prior to diving into developing growing and pruning ensembles, let’s first go about establishing a dataset and baseline.
We will leverage a synthetic binary classification problem as the foundation for this investigation, defined by the make_classification() function with 5,000 instances and 20 numerical input features.
The instance below gives definition to the dataset and summarizes its size.
1 2 3 4 5 6 | # test classification dataset from sklearn.datasets import make_classification # define dataset X, y = make_classification(n_samples=5000, n_features=20, n_informative=10, n_redundant=10, random_state=1) # summarize the dataset print(X.shape, y.shape) |
Running the instance develops the dataset in a repeatable fashion and reports the number of rows and input features, matching our expectations.
(5000, 20) (5000,)
Then, we can select some candidate models that will furnish the foundation for our ensemble.
We will leverage five standard ML models, which includes logistic regression, naïve Bayes, decision tree, support vector machine, and a k-nearest neighbour algorithm.
To start with, we can give definition to a function that will develop every model with default hyperparameters. Every model will be defined as a tuple with a name and the model object, then included to a list. This is a useful structure both for enumeration of the models with their names for standalone evaluation and for subsequent leveraging within an ensemble.
The get_models() function below implements this and returns the list of models to consider:
1 2 3 4 5 6 7 8 9 | # get a list of models to evaluate def get_models(): models = list() models.append((‘lr’, LogisticRegression())) models.append((‘knn’, KNeighborsClassifier())) models.append((‘tree’, DecisionTreeClassifier())) models.append((‘nb’, GaussianNB())) models.append((‘svm’, SVC(probability=True))) return models |
We can then go about defining a function that takes a singular model and the dataset and assesses the performance of the model on the dataset. We will assess a model leveraging repeated stratified k-fold cross-validation with ten folds and three repeats, best practice within machine learning.
The evaluate_model() function below implements this and gives back a listing of scores across all folds and repeats.
1 2 3 4 5 6 7 | # evaluate a give model using cross-validation def evaluate_model(model, X, y): # define the model evaluation procedure cv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1) # evaluate the model scores = cross_val_score(model, X, y, scoring=’accuracy’, cv=cv, n_jobs=-1) return scores |
We can then develop the list of models and enumerate them, reporting the performance of each on the synthetic dataset in turn.
Connecting this together, the complete instance is detailed below.
# evaluate standard models on the synthetic dataset
from numpy import mean
from numpy import std
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import RepeatedStratifiedKFold
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.svm import SVC
from sklearn.naive_bayes import GaussianNB
from matplotlib import pyplot
# get the dataset
def get_dataset():
X, y = make_classification(n_samples=5000, n_features=20, n_informative=10, n_redundant=10, random_state=1)
return X, y
# get a list of models to evaluate
def get_models():
models = list()
models.append((‘lr’, LogisticRegression()))
models.append((‘knn’, KNeighborsClassifier()))
models.append((‘tree’, DecisionTreeClassifier()))
models.append((‘nb’, GaussianNB()))
models.append((‘svm’, SVC(probability=True)))
return models
# evaluate a give model using cross-validation
def evaluate_model(model, X, y):
# define the model evaluation procedure
cv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)
# evaluate the model
scores = cross_val_score(model, X, y, scoring=’accuracy’, cv=cv, n_jobs=-1)
return scores
# define dataset
X, y = get_dataset()
# get the models to evaluate
models = get_models()
# evaluate the models and store results
results, names = list(), list()
for name, model in models:
# evaluate model
scores = evaluate_model(model, X, y)
# store results
results.append(scores)
names.append(name)
# summarize result
print(‘>%s %.3f (%.3f)’ % (name, mean(scores), std(scores)))
# plot model performance for comparison
pyplot.boxplot(results, labels=names, showmeans=True)
pyplot.show()
Running the instance evaluates every standalone ML algorithm on the synthetic binary classification dataset.
Your outcomes might demonstrate variance provided the stochastic nature of the algorithm or evaluation process, or variations in numerical accuracy. Take up running the instance a few times and contrast the average outcome.
In this scenario, we can observe that both the KNN and SVM models have the best performance on this dataset, accomplishing a mean classification precision of approximately 95.3%.
These outcomes furnish a baseline in performance that we need any ensemble to exceed in order to be considered useful on this dataset.
1 2 3 4 5 | >lr 0.856 (0.014) >knn 0.953 (0.008) >tree 0.867 (0.014) >nb 0.847 (0.021) >svm 0.953 (0.010) |
A figure is developed demonstrating box and whisker plots of the distribution of precision scores for every algorithm.
We can observe that the KNN and SVM algorithms have much better performance than the other algorithms, even though all algorithms are skilful in differing ways. This might make them good candidates to consider within an ensemble.
Then, we are required to establish a baseline ensemble that leverages all models. This will furnish a point of comparison with growing and pruning strategies that look for improved performance with a smaller subset of models.
In this scenario, we will leverage a voting ensemble with soft voting. This implies that every model will forecast probabilities and the probabilities will be summed by the ensemble model to select a final output prediction for every input sample.
This can be accomplished leveraging the VotingClassifier class where the members are set through the “estimators” argument, which expects a listing of models where every model is a tuple with a name and configured model object, just as we defined in the prior section.
We can then set the type of voting to perform through the “voting” argument, which in this scenario is set to “soft”
1 2 3 | … # create the ensemble ensemble = VotingClassifier(estimators=models, voting=’soft’) |
Connecting this together, the instance below evaluates a voting ensemble of all five models on the synthetic binary classification dataset.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 | # example of a voting ensemble with soft voting of ensemble members from numpy import mean from numpy import std from sklearn.datasets import make_classification from sklearn.model_selection import cross_val_score from sklearn.model_selection import RepeatedStratifiedKFold from sklearn.linear_model import LogisticRegression from sklearn.neighbors import KNeighborsClassifier from sklearn.tree import DecisionTreeClassifier from sklearn.svm import SVC from sklearn.naive_bayes import GaussianNB from sklearn.ensemble import VotingClassifier
# get the dataset def get_dataset(): X, y = make_classification(n_samples=5000, n_features=20, n_informative=10, n_redundant=10, random_state=1) return X, y
# get a list of models to evaluate def get_models(): models = list() models.append((‘lr’, LogisticRegression())) models.append((‘knn’, KNeighborsClassifier())) models.append((‘tree’, DecisionTreeClassifier())) models.append((‘nb’, GaussianNB())) models.append((‘svm’, SVC(probability=True))) return models
# define dataset X, y = get_dataset() # get the models to evaluate models = get_models() # create the ensemble ensemble = VotingClassifier(estimators=models, voting=’soft’) # define the evaluation procedure cv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1) # evaluate the ensemble scores = cross_val_score(ensemble, X, y, scoring=’accuracy’, cv=cv, n_jobs=-1) # summarize the result print(‘Mean Accuracy: %.3f (%.3f)’ % (mean(scores), std(scores))) |
Running the instance evaluates the soft voting ensemble of all models leveraging repeated stratified k-fold cross-validation and reports the mean precision across all folds and repeats.
Your outcomes may demonstrate variance provided the stochastic nature of the algorithm or evaluation procedures, or variations in numerical accuracy. Consider running the instance a few times and contrast the average outcome.
In this scenario, we can observe that the voting ensemble accomplished a mean precision of about 92.8%. This is lower than SVM and KNN models leveraged alone that accomplished a precision of approximately 95.3%.
This outcome highlights that a simple voting ensemble of all models have the outcome of a model with increased complexity and worse performance in this scenario. Probably we can identify a subset of members that have better performance than any singular model and has reduced complexity than merely leveraging all models.
Mean Accuracy: 0.928 (0.012)
Then, we will explore pruning members from the ensemble.
Ensemble Pruning Instance
In this portion of the blog, we will look into how to develop a greedy ensemble pruning algorithm from the ground up.
We will leverage a greedy algorithm in this scenario, which is straightforward to implement. This consists of removing a single member from the ensemble and evaluating the performance and repeating this for every member within the ensemble. The member, that, if removed, has the outcome of the best improvement in performance is then permanently removed from the ensemble and the procedure repeats. This goes on till no subsequent improvements can be accomplished.
It is referenced to as a “greedy” algorithm as it seeks the best improvement at every step, it is possible that the best combo of members is not on the path of greedy enhancements, in which scenario the greedy algorithm will not identify it and a global optimization algorithm could be leveraged instead.
To start with, we can give definition to a function that evaluates a candidate list of models. This function will take the listing of models and the dataset and develop a voting ensemble from the listing of models and evaluate its performance leveraging repeated stratified k-fold cross-validation, returning the mean classification precision.
This function can be leveraged to evaluate every candidate’s removal from the ensemble. The evaluate_ensemble() function below implements this.
1 2 3 4 5 6 7 8 9 10 11 12 13 | # evaluate a list of models def evaluate_ensemble(models, X, y): # check for no models if len(models) == 0: return 0.0 # create the ensemble ensemble = VotingClassifier(estimators=models, voting=’soft’) # define the evaluation procedure cv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1) # evaluate the ensemble scores = cross_val_score(ensemble, X, y, scoring=’accuracy’, cv=cv, n_jobs=-1) # return mean score return mean(scores) |
Then, we can define a function that performs a singular round of pruning.
To start with, a baseline in performance is determined with all models that are presently in the ensemble. Then, the listing of models is enumerated and every one is removed in turn, and the impact of removing the model is evaluated on the ensemble. If the removal has the outcome of an improvement in performance, the new score and particular model that was removed is recorded.
Critically, the trial removal is carried out on a copy of the listing of models, not on the primary listing of models itself. This is to make sure we just remove an ensemble member from the listing once we know it will have the outcome of the best potential improvement from all the members that could possibly be removed at the present step.
The prune_round() function below implements this provided the listing of present models in the ensemble and dataset, and returns the improvement in score (if any) and the ideal model to remove to accomplish that score.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 | # perform a single round of pruning the ensemble def prune_round(models_in, X, y): # establish a baseline baseline = evaluate_ensemble(models_in, X, y) best_score, removed = baseline, None # enumerate removing each candidate and see if we can improve performance for m in models_in: # copy the list of chosen models dup = models_in.copy() # remove this model dup.remove(m) # evaluate new ensemble result = evaluate_ensemble(dup, X, y) # check for new best if result > best_score: # store the new best best_score, removed = result, m return best_score, removed |
Then, we are required to drive the pruning process.
This consists of running a round of pruning till no further improvement in precision is accomplished by calling the prune_round() function repeatedly.
If the function returns none for the model to be removed, we know that no singular greedy improvement is doable and we can return the final listing of models. Otherwise, the selected model is removed from the ensemble and the process goes on.
The prune_ensemble() function below implements this and returns the models to leverage in the final ensemble and the score that it accomplished through our evaluation procedure.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 | # prune an ensemble from scratch def prune_ensemble(models, X, y): best_score = 0.0 # prune ensemble until no further improvement while True: # remove one model to the ensemble score, removed = prune_round(models, X, y) # check for no improvement if removed is None: print(‘>no further improvement’) break # keep track of best score best_score = score # remove model from the list models.remove(removed) # report results along the way print(‘>%.3f (removed: %s)’ % (score, removed[0])) return best_score, models |
We can connect all of this together into an instance of ensemble pruning on the synthetic binary classification dataset.
# example of ensemble pruning for classification
from numpy import mean
from numpy import std
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import RepeatedStratifiedKFold
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.svm import SVC
from sklearn.naive_bayes import GaussianNB
from sklearn.ensemble import VotingClassifier
from matplotlib import pyplot
# get the dataset
def get_dataset():
X, y = make_classification(n_samples=5000, n_features=20, n_informative=10, n_redundant=10, random_state=1)
return X, y
# get a list of models to evaluate
def get_models():
models = list()
models.append((‘lr’, LogisticRegression()))
models.append((‘knn’, KNeighborsClassifier()))
models.append((‘tree’, DecisionTreeClassifier()))
models.append((‘nb’, GaussianNB()))
models.append((‘svm’, SVC(probability=True)))
return models
# evaluate a list of models
def evaluate_ensemble(models, X, y):
# check for no models
if len(models) == 0:
return 0.0
# create the ensemble
ensemble = VotingClassifier(estimators=models, voting=’soft’)
# define the evaluation procedure
cv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)
# evaluate the ensemble
scores = cross_val_score(ensemble, X, y, scoring=’accuracy’, cv=cv, n_jobs=-1)
# return mean score
return mean(scores)
# perform a single round of pruning the ensemble
def prune_round(models_in, X, y):
# establish a baseline
baseline = evaluate_ensemble(models_in, X, y)
best_score, removed = baseline, None
# enumerate removing each candidate and see if we can improve performance
for m in models_in:
# copy the list of chosen models
dup = models_in.copy()
# remove this model
dup.remove(m)
# evaluate new ensemble
result = evaluate_ensemble(dup, X, y)
# check for new best
if result > best_score:
# store the new best
best_score, removed = result, m
return best_score, removed
# prune an ensemble from scratch
def prune_ensemble(models, X, y):
best_score = 0.0
# prune ensemble until no further improvement
while True:
# remove one model to the ensemble
score, removed = prune_round(models, X, y)
# check for no improvement
if removed is None:
print(‘>no further improvement’)
break
# keep track of best score
best_score = score
# remove model from the list
models.remove(removed)
# report results along the way
print(‘>%.3f (removed: %s)’ % (score, removed[0]))
return best_score, models
# define dataset
X, y = get_dataset()
# get the models to evaluate
models = get_models()
# prune the ensemble
score, model_list = prune_ensemble(models, X, y)
names = ‘,’.join([n for n,_ in model_list])
print(‘Models: %s’ % names)
print(‘Final Mean Accuracy: %.3f’ % score)
Running the instance performs the ensemble pruning procedure, reporting which model was removed every round and the precision of the model after the model was removed.
Your outcomes may demonstrate variance provided the stochastic nature of the algorithm or evaluation procedure, or variations in numerical accuracy. Consider running the instance a few times and contrast the average outcome.
In this scenario, we can observe that three rounds of pruning were carried out, removing the naïve Bayes, decision tree, and logistic regression algorithms, leaving just the SVM and KNN algorithms that accomplished a mean classification precision of approximately 95.7%. This is better than the 95.3% accomplished by SVM and KNN leveraged in a standalone fashion, and clearly better than bringing all models together.
The final listing of models could then be leveraged in a new final voting ensemble model through the “estimators” argument, fit on the entire dataset and leverage to make a prediction on new data.
1 2 3 4 5 6 | >0.939 (removed: nb) >0.948 (removed: tree) >0.957 (removed: lr) >no further improvement Models: knn,svm Final Mean Accuracy: 0.957 |
Now that we are acquainted with developing and evaluating an ensemble pruning strategy, let’s observe the reverse case of growing the ensemble members.
Ensemble Growing Instance
In this portion of the blog, we will look into how to develop a greedy ensemble growing algorithm from the ground up.
The structure of greedily growing an ensemble is a lot like the greedy pruning of members, even though in reverse. We begin with an ensemble with no models and include a singular model that has the best performance. Models are then included one by one if they have the outcome of a lift in the performance over the ensemble prior to the inclusion of the model.
A lot of the code is the same as the pruning scenario so we can concentrate on the differences.
To start with, we must go about defining a function to carry out one round of growing the ensemble. This consists of enumerating all candidate models that could be included and evaluating the impact of including each in turn to the ensemble. The singular model that has the outcome of the biggest improvement is then returned by the function, combined with its score.
The grow_round() function below implements this behaviour.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 | # perform a single round of growing the ensemble def grow_round(models_in, models_candidate, X, y): # establish a baseline baseline = evaluate_ensemble(models_in, X, y) best_score, addition = baseline, None # enumerate adding each candidate and see if we can improve performance for m in models_candidate: # copy the list of chosen models dup = models_in.copy() # add the candidate dup.append(m) # evaluate new ensemble result = evaluate_ensemble(dup, X, y) # check for new best if result > best_score: # store the new best best_score, addition = result, m return best_score, addition |
Then, we require a function to drive the growing procedure.
This consists of a loop that runs rounds of growing till no further additions can be made having the outcome of an improvement in model performance. For every addition that can be made, the primary listing of models in the ensemble is updated and the listing of models presently in the ensemble is reported along with the performance.
The grow_ensemble() function implements this and returns the listing of models greedily determined to have the outcome of the best performance along with the expected mean precision.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 | # grow an ensemble from scratch def grow_ensemble(models, X, y): best_score, best_list = 0.0, list() # grow ensemble until no further improvement while True: # add one model to the ensemble score, addition = grow_round(best_list, models, X, y) # check for no improvement if addition is None: print(‘>no further improvement’) break # keep track of best score best_score = score # remove new model from the list of candidates models.remove(addition) # add new model to the list of models in the ensemble best_list.append(addition) # report results along the way names = ‘,’.join([n for n,_ in best_list]) print(‘>%.3f (%s)’ % (score, names)) return best_score, best_list |
Connecting this together, the complete instance of greedy ensemble growing on the synthetic binary classification dataset is detailed here.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 | # example of ensemble growing for classification from numpy import mean from numpy import std from sklearn.datasets import make_classification from sklearn.model_selection import cross_val_score from sklearn.model_selection import RepeatedStratifiedKFold from sklearn.linear_model import LogisticRegression from sklearn.neighbors import KNeighborsClassifier from sklearn.tree import DecisionTreeClassifier from sklearn.svm import SVC from sklearn.naive_bayes import GaussianNB from sklearn.ensemble import VotingClassifier from matplotlib import pyplot
# get the dataset def get_dataset(): X, y = make_classification(n_samples=5000, n_features=20, n_informative=10, n_redundant=10, random_state=1) return X, y
# get a list of models to evaluate def get_models(): models = list() models.append((‘lr’, LogisticRegression())) models.append((‘knn’, KNeighborsClassifier())) models.append((‘tree’, DecisionTreeClassifier())) models.append((‘nb’, GaussianNB())) models.append((‘svm’, SVC(probability=True))) return models
# evaluate a list of models def evaluate_ensemble(models, X, y): # check for no models if len(models) == 0: return 0.0 # create the ensemble ensemble = VotingClassifier(estimators=models, voting=’soft’) # define the evaluation procedure cv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1) # evaluate the ensemble scores = cross_val_score(ensemble, X, y, scoring=’accuracy’, cv=cv, n_jobs=-1) # return mean score return mean(scores)
# perform a single round of growing the ensemble def grow_round(models_in, models_candidate, X, y): # establish a baseline baseline = evaluate_ensemble(models_in, X, y) best_score, addition = baseline, None # enumerate adding each candidate and see if we can improve performance for m in models_candidate: # copy the list of chosen models dup = models_in.copy() # add the candidate dup.append(m) # evaluate new ensemble result = evaluate_ensemble(dup, X, y) # check for new best if result > best_score: # store the new best best_score, addition = result, m return best_score, addition
# grow an ensemble from scratch def grow_ensemble(models, X, y): best_score, best_list = 0.0, list() # grow ensemble until no further improvement while True: # add one model to the ensemble score, addition = grow_round(best_list, models, X, y) # check for no improvement if addition is None: print(‘>no further improvement’) break # keep track of best score best_score = score # remove new model from the list of candidates models.remove(addition) # add new model to the list of models in the ensemble best_list.append(addition) # report results along the way names = ‘,’.join([n for n,_ in best_list]) print(‘>%.3f (%s)’ % (score, names)) return best_score, best_list
# define dataset X, y = get_dataset() # get the models to evaluate models = get_models() # grow the ensemble score, model_list = grow_ensemble(models, X, y) names = ‘,’.join([n for n,_ in model_list]) print(‘Models: %s’ % names) print(‘Final Mean Accuracy: %.3f’ % score) |
Running the instance incrementally includes a single model at a time to the ensemble and reports the mean classification precision of the ensemble of the models.
Your outcomes may demonstrate variance provided the stochastic nature of the algorithm or evaluation procedure, or variations in numerical accuracy. Take up running the instance a few times and contrast the average outcome.
In this scenario, we can observe that ensemble growing identified the same answer as greedy ensemble pruning where an ensemble of SVM and KNN accomplished a mean classification precision of approximately 95.6%, an enhancement over any singular standalone model and over combining all models.
1 2 3 4 5 | >0.953 (svm) >0.956 (svm,knn) >no further improvement Models: svm,knn Final Mean Accuracy: 0.956 |
Further Reading
Books
Ensemble Methods, Foundations and Algorithms, 2012
Pattern Classification Using Ensemble Methods, 2010
APIs
sklearn.ensemble.VotingClassifer API
Conclusion
In this guide, you found out how to develop ensemble selection algorithms from the ground up.
Particularly, you learned:
- Ensemble methods consists of selecting a subset of ensemble members that has the outcome of reduced complexity than leveraging all members and at times improved performance.
- How to generate and evaluate a greedy ensemble pruning algorithm for classification.
- How to develop and evaluate an algorithm for greedily growing an ensemble from the ground up.