Dynamic ensemble selection is an ensemble learning strategy that automatically chooses a subset of ensemble members just in-time when making a prediction.

The strategy consists of fitting multiple machine learning models on the training dataset, then choosing the models that are expected to execute best when making a prediction for a particular new instance, on the basis of details of the instance to be forecasted.

This can be accomplished leveraging a k-nearest neighbour model to locate instances in the training dataset that are nearest to the new instance to be predicted, assessing all models in the pool on this neighbourhood and leveraging the models that execute the best on the neighbourhood to make a forecast for the new instance.

As such, the dynamic ensemble selection can often have better performance than any singular model in the pool and better than averaging all members of the pool, so-called static ensemble selection.

In this guide, you will find out how to develop ensemble selection models in Python.

After going through this guide, you will be aware of:

• Dynamic ensemble selection algorithms automatically select ensemble members when making a forecast on fresh data.
• How to develop and assess dynamic ensemble selection models for classification activities leveraging the scikit-learn API
• How to explore the impact of dynamic ensemble selection model hyperparameters on classification precision.

Tutorial Summarization

This tutorial is subdivided into three portions, which are:

1] Dynamic Ensemble Selection

2] k-Nearest Neighbour Oracle (KNORA) with Scikit-Learn

1. KNORA-Eliminate (KNORA-E)
2. KNORA-Union (KNORA-U)

3] Hyperparameter Tuning for KNORA

1. Explore k in k-nearest neighbour
2. Explore algorithms for classifier pool

Dynamic Ensemble Selection

Multiple classifier systems reference a domain of machine learning algorithms that leverage multiple models to address classification predictive modelling issues.

The first categorization of multiple classifier systems to identify success is referenced to as Dynamic Classifier Selection, or DCS for short.

Dynamic Classifier Selection: Algorithms that dynamically select one from amongst several trained models to make a prediction on the basis of the particular details of the input.

Dynamic Classifier Selection algorithms typically involve partitioning the input feature space within some fashion and allocating particular models to be accountable for making predictions for every partition. There are an array of differing DCS algorithms and research efforts are primarily concentrated on how to assess and allocate classifiers to particular regions of the input space.

Upon training multiple individual learners, DCS dynamically chooses one learner for every test instance […] DCS makes predictions by leveraging one individual learner.

A natural extension to DCS is algorithms that choose one or more models dynamically in order to make a prediction. That is, choosing a subset or ensemble of classifiers dynamically. These strategies are referenced to as dynamic ensemble selection, or DES.

Dynamic Ensemble Selection: Algorithms that dynamically select a subset of trained models to make a forecast on the basis of the particular details of the input.

Dynamic Ensemble Selection algorithms function much like DCS algorithms, except forecasts are made leveraging votes from multiple classifier models rather than a single best model. In essence, every region of the input feature space is owned by a subset of models that have the best performance in that region.

Provided the fact that choosing just one classifier can be highly error-prone, some researchers decided to choose a subset of the pool of classifier instead of just a singular base classifier. All base classifiers that obtained a specific competence level are leveraged to compose the EoC, and their outputs are aggregated to forecast the label.

Probably the canonical approach to dynamic ensemble selection is the k-Nearest Neighbour Oracle, or KNORA, algorithm as it is a natural extension of the canonical dynamic classifier selection algorithm “Dynamic Classifier Selection Local Accuracy”, or DCS-LA.

DCS-LA consists of choosing the k-nearest neighbours from the training or validation dataset for a provided new input pattern, then choosing the singular best classifier on the basis of its performance in that neighbourhood of k instances to make a forecast on the new instance.

KNORA was detailed by Albert Ko, et al. in their 2008 paper titled “From Dynamic Classifier Selection to Dynamic Ensemble Selection”. It is an extension of DCS-LA that chooses several models that have good performance on the neighbourhood and whose predictions are then brought together leveraging majority voting to make a final output prediction.

For any test data point, KNORA merely identifies its nearest K neighbours in the validation set, finds out which classifiers accurately classify those neighbours in the validation set and leverages them as the ensemble for classifying the provided pattern in that test set.

The chosen classifier models are referenced to as “oracles”, therefore the leveraging of oracle in the name of the method.

The ensemble is viewed as dynamic as the members are selected just in-time conditional on the particular input pattern needing a prediction. This is in opposition to static, where ensemble members are selected once, like averaging predictions from all classifiers in the model.

This is performed through a dynamic fashion, as differing patterns might need differing ensembles of classifiers. Thus, we call our method a dynamic ensemble selection.

Two variants of KNORA are detailed, which includes KNORA-Eliminate and KNORA-Union.

• KNORA-Eliminate (KNORA-E): Ensemble of classifiers that accomplishes perfect precision on the neighbourhood of the new instance, with a reducing neighbourhood size till at least one perfect classifier is located.
• KNORA-Union (KNORA-U): Ensemble of all classifiers that make at least one right prediction on the neighbourhood with weighted voting and votes in proportion to precision on the neighbourhood.

KNORA-Eliminate, or KNORA-E in short, consists of choosing all classifiers that accomplish perfect predictions on the neighbourhood of k examples in the neighbourhood. If no classifier accomplishes 100% precision, the neighbourhood size is reduced by one and the models are re-evaluated. This procedure is repeated till one or more models are discovered that has ideal performance, and then leveraged to make a forecast for the new example.

In the scenario where no classifier can rightly classify all the K-nearest neighbours of the test pattern, then we merely decrease the value of K till at least one classifier rightly classifies its neighbours.

KNORA-Union, or KNORA-U for short, consists of choosing all classifiers that make at least one right prediction in the neighbourhood. The forecast from every classifier are then brought together leveraging a weighted average, where the number of right predictions in the neighbourhood signifies the number of votes allocated to every classifier.

The more neighbours a classifier categorizes rightly, the more votes this classifier will have for a test pattern.

Now that we are acquainted with DES and the KNORA algorithm, let’s look at how we can leverage it on our own classification predictive modelling projects.

k-Nearest Neighbour Oracle (KNORA) with Scikit-Learn

The Dynamic Ensemble Library or DESlib for short, is a Python machine learning library that furnishes an implementation of several differing dynamic classifiers and dynamic ensemble selection algorithms.

DESlib is a simple-to-use ensemble learning library concentrated on the implementation of the state-of-the-art strategies for dynamic classifier and ensemble selection.

To start with, we can setup the DESlib library leveraging pip package manager, if it is not already setup.

sudo pip install deslib

Once setup, we can then check that the library was setup correctly and is ready to be leveraged by loading the library and printing the setup version.

Executing the script will print your version of the DESlib library you have setup.

Your version should be identical or higher. If not, you must update your version of the DESlib library.

0.3

The DESlib furnishes an implementation of the KNORA algorithm with every dynamic ensemble selection strategy through the KNORAE and KNORAU classes respectively.

Every class can be leveraged as a scikit-learn model directly, facilitating the full suite of scikit-learn data preparation, modelling pipelines, and model evaluation strategies to be leveraged directly.

Both classes leverage a k-nearest neighbour algorithm to choose the neighbour with a default value of k=7.

A bootstrap aggregation (bagging) ensemble of decision trees is leveraged as the pool of classifier models considered for every classification that is made by default, even though this can be altered by setting “pool_classifiers” to a list of models.

We can leverage the make_classification() function to develop a synthetic binary classification problem with 10,000 instances and 20 input features.

Running the instance creates the dataset and summarizes the shape of the input and output components.

(10000, 20) (10000,)

Now that we are acquainted with the DESlib API, let’s look at how to leverage every KNORA algorithm on our synthetic classification dataset.

KNORA-Eliminate (KNORA-E)

We can assess a KNORA-Eliminate dynamic ensemble selection algorithm on the synthetic dataset.

In this scenario, we will leverage default model hyperparameters, which includes bagged decision trees as the pool of classifier models and a k=7 for the selection of the local neighbourhood when making a forecast.

We will assess the model leveraged repeated stratified k-fold cross-validation with three repeats and 10 folds. We will report the mean and standard deviation of the precision of the model throughout all repeats and folds.

The full example is detailed below.

Running the instance reports the mean and standard deviation precision of the model.

Your outcomes may demonstrate variance provided the stochastic nature of the algorithm or evaluation procedure, or variations in numerical accuracy. Consider running the instance a few times and contrast the average outcome.

In this scenario, we can see the KNORA-E ensemble and default hyperparameters accomplish a classification precision of about 91.5%.

Mean Accuracy: 0.915 (0.009)

We can also leverage the KNORA-E ensemble as a final model and make forecasts for classification.

First, the model is fitted on all available data, then the predict() function can be called to make forecasts on new data.

The instance below demonstrates this on our binary classification dataset.

Running the instance fits the KNORA-E dynamic ensemble selection algorithm on the entire dataset and is then leveraged to make a forecast on a new row of data, as we might when leveraging the model in an application.

Predicted Class: 0

Now that we are acquainted with leveraging KNORA-E, let’s observe the KNORA-Union method.

KNORA-Union (KNORA-U)

We can assess a KNORA-Union model on the synthetic dataset.

In this scenario, we will leverage default model hyperparameters, which includes bagged decision trees as the pool of classifier models and a k=7 for the selection of the local neighbourhood when making a forecast.

We will assess the model leveraged repeated stratified k-fold cross-validation with three repeats and ten folds. We will report the mean and standard deviation of the precision of the model throughout all folds and repeats.

The full instance is detailed below.

Running the instance reports the mean and standard deviation precision of the model.

Your outcomes may demonstrate variance provided the stochastic nature of the algorithm or evaluation procedure, or variations in numerical accuracy. Take up running the instance a few times and contrast the average outcome.

In this scenario, we can see the KNORA-U dynamic ensemble selection model and default hyperparameters accomplish a classification precision of approximately 93.3%.

Mean Accuracy: 0.933 (0.009)

We can also leverage the KNORA-U model as a final model and make forecasts for classification.

First, the model is fit on all available data, then the predict() function can be called to make forecasts on new data.

The instance below illustrates this on our binary classification dataset.

Running the instance fits the KNORA-U model on the total dataset and is then leveraged to make a forecast on a new row of data, as we might when leveraging the model in an application.

Predicted Class: 0

Now that we are acquainted with leveraging the scikit-learn API to assess and leverage KNORA models, let’s look at setting up the model.

Hyperparamter Tuning for KNORA

In this portion of the blog, we will take a deeper look at some of the hyperparameters you should consider tuning for the KNORA model and their impact on model performance.

There are several hyperparameters we can look at for KNORA, even though in this scenario, we will observe the value of k in the k-nearest neighbour model leveraged in the local assessment of the models, and how to leverage a custom pool of classifiers.

We will leverage the KNORA-Union as the foundation for these experiments, even though the choice of the particular method is random.

Explore k in k-Nearest Neighbours

The configuration of the k-nearest neighbours algorithm is vital to the KNORA model as it defines the scope of the neighbourhood in which every ensemble is taken up for selection.

The k-value manages the size of the neighbourhood and it is critical to set it to a value that is relevant for your dataset, particularly the density of samples within the feature space. A value too small will imply that appropriate instances in the training set might be excluded from the neighbourhood, while values too big may imply that the signal is being washed out by too many instances.

The code instance below explores the classification precision of the KNORA-U algorithm with k values from 2 to 21.

Running the instance first reports the mean precision for every configured neighbourhood size.

Your outcomes may demonstrate variance provided the stochastic nature of the algorithm or evaluation process, or variations in numerical accuracy. Take up executing the instance a few times and contrast the average outcome.

In this scenario, we can observe that the precision increases with the neighbourhood size, probably to k=10, where it seems to level off.

A box and whisker plot is developed for the distribution of precision scores for every configured neighbourhood size.

We can observe the general trend of increasing model performance and k value prior to reaching a plateau.

Explore Algorithms for Classifier Pool

The selection of algorithms leveraged in the pool for the KNORA is another critical hyperparameter.

By default, bagged decision trees are leveraged, as it has proven to be an efficient strategy on an array of classification tasks. Nonetheless, a custom pool of classifiers can be considered.

In the majority of DS publications, the pool of classifiers is produced leveraging either well known ensemble generation methods like Bagging, or by leveraging heterogenous classifiers.

This needs first defining a listing of classifier models to leverage and fitting every one on the training dataset. Unluckily, this implies that the automatic k-fold cross-validation model assessment methods in scikit-learn can’t be leveraged in this scenario. Rather, we will leverage a train-test split so that we can go about fitting the classifier pool manually on the training dataset.

The listing of fit classifiers can then be mentioned to the KNORA-Union (or KNORA-Eliminate) class through the “pool_classifiers” argument. In this scenario, we will leverage a pool that includes logistic regression, a decision tree, and a naïve Bayes classifier.

The full instance of assessing the KNORA ensemble and a custom set of classifiers on the synthetic dataset is detailed below.

Running the instance first reports the mean precision for the model with the custom pool of classifiers.

Your outcomes may demonstrate variance provided the stochastic nature of the algorithm or evaluation procedure, or variations in numerical accuracy. Take up running the instance a few times and contrast the average outcome.

In this scenario, we can observe that the model accomplished a precision of about 91.3%.

Accuracy: 0.913

In order to take up the KNORA model, it must have better performance than any other contributing model. Otherwise, we would merely leverage the contributing model that has better performance.

We can check this by assessing the performance of every contributing classifier on the test set.

The updated instance of KNORA with a customized pool of classifiers that are also assessed independently is detailed below:

# evaluate KNORA-U dynamic ensemble selection with a custom pool of algorithms

from sklearn.datasets import make_classification

from sklearn.model_selection import train_test_split

from sklearn.metrics import accuracy_score

from deslib.des.knora_u import KNORAU

from sklearn.linear_model import LogisticRegression

from sklearn.tree import DecisionTreeClassifier

from sklearn.naive_bayes import GaussianNB

X, y = make_classification(n_samples=10000, n_features=20, n_informative=15, n_redundant=5, random_state=7)

# split the dataset into train and test sets

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5, random_state=1)

# define classifiers to use in the pool

classifiers = [

LogisticRegression(),

DecisionTreeClassifier(),

GaussianNB()]

# fit each classifier on the training set

for c in classifiers:

c.fit(X_train, y_train)

# define the KNORA-U model

model = KNORAU(pool_classifiers=classifiers)

# fit the model

model.fit(X_train, y_train)

# make predictions on the test set

yhat = model.predict(X_test)

# evaluate predictions

score = accuracy_score(y_test, yhat)

print(‘Accuracy: %.3f’ % (score))

# evaluate contributing models

for c in classifiers:

yhat = c.predict(X_test)

score = accuracy_score(y_test, yhat)

print(‘>%s: %.3f’ % (c.__class__.__name__, score))

Running the instance first reports the mean precision for the model with the custom pool of classifiers and the precision of every contributing model.

Your outcomes may demonstrate variance provided the stochastic nature of the algorithm or evaluation procedure, or variations in numerical accuracy. Take up running the instance a few times and contrast the average outcome.

In this scenario, we can observe that again the KNORAU accomplishes a precision of about 91.3%, which is better than any contributing model.

Rather than mentioning a pool of classifiers, it is also doable to mention a singular ensemble algorithm from the sciit-learn library and the KNORA algorithm will immediately leverage the internal ensemble members as classifiers.

For instance, we can leverage a random forest ensemble with 1,000 members as the base classifiers to consider within KNORA as follows:

Connecting this together, the full instance of KNORA-U with random forest ensemble members as classifiers is detailed below.

# evaluate KNORA-U with a random forest ensemble as the classifier pool

from sklearn.datasets import make_classification

from sklearn.model_selection import train_test_split

from sklearn.metrics import accuracy_score

from deslib.des.knora_u import KNORAU

from sklearn.ensemble import RandomForestClassifier

X, y = make_classification(n_samples=10000, n_features=20, n_informative=15, n_redundant=5, random_state=7)

# split the dataset into train and test sets

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5, random_state=1)

# define classifiers to use in the pool

pool = RandomForestClassifier(n_estimators=1000)

# fit the classifiers on the training set

pool.fit(X_train, y_train)

# define the KNORA-U model

model = KNORAU(pool_classifiers=pool)

# fit the model

model.fit(X_train, y_train)

# make predictions on the test set

yhat = model.predict(X_test)

# evaluate predictions

score = accuracy_score(y_test, yhat)

print(‘Accuracy: %.3f’ % (score))

# evaluate the standalone model

yhat = pool.predict(X_test)

score = accuracy_score(y_test, yhat)

print(‘>%s: %.3f’ % (pool.__class__.__name__, score))

Running the instance first reports the mean precision for the model with the custom pool of classifiers and the precision of the random forest model.

Your outcomes may demonstrate variance provided the stochastic nature of the algorithm or evaluation procedure, or variations in numerical accuracy. Take up running the instance a few times and contrast the average outcome.

In this scenario, we can observe that the KNORA model with dynamically chosen ensemble members out-performs the random forest with the statistically chosen (full set) ensemble members.

Accuracy: 0.968

>RandomForestClassifier: 0.967

Further Reading

This part of the blog furnishes additional resources on the topic if you are seeking to delve deeper.

Papers

• From Dynamic Classifier Selection to Dynamic Ensemble Selection, 2008
• Dynamic Selection of Classifiers – a comprehensive review, 2014
• Dynamic classifier selection: Recent advances and perspectives, 2018

Books

• Ensemble methods: Foundations and algorithms, 2012

APIs

• Dynamic selection library project, GitHub
• DESlib API Documentation

Conclusion

In this guide, you found out about how to develop dynamic ensemble selection models in Python.

Particularly, you learned:

• Dynamic ensemble selection algorithms automatically select ensemble members when making a forecast on new data
• How to develop and assess dynamic ensemble selection models for classification activities leveraging the scikit-learn API
• How to explore the impact of dynamic ensemble selection model hyperparameters on classification precision.