A problem with training neural networks is in the selection of the number of training epochs to use.

A lot of epochs can cause overfitting of the training dataset, while too few might have the outcome of an underfit problem. Early stopping is a strategy that facilitates you to mention an arbitrary large number of training epochs and stop training after the model performance ceases improving on a hold out validation dataset.

In this guide, you will find out the Keras API for including early stopping to overfit deep learning neural network models.

After going through this guide, you will be aware of:

• How to survey/monitor the performance of a model during training leveraging the Keras API
• How to develop and configure early stopping and model checkpoint callbacks leveraging the Keras API
• How to minimize overfitting by including an early stopping to a current model.

Tutorial Summarization

This tutorial is subdivided into six portions, which are:

1. Leveraging callbacks in Keras
2. Evaluating a Validation Dataset
3. Monitoring Model Performance
4. Early stopping in Keras
5. Checkpointing in Keras
6. Early stopping case study

Using Callbacks in Keras

Callbacks furnish a method to execute code and interact with the training model procedures automatically.

Callbacks can be furnished to the fit() function through the “callbacks” argument.

To start with, callbacks must be instantiated.

…

cb = Callback(…)

Then one, or more callbacks that you intend to leverage must be included to a Python list.

…

cb_list = [cb, …]

Lastly, the list of callbacks is furnished to the callback argument when fitting the model.

Evaluating a Validation Dataset in Keras

Early stopping needs that a validation dataset is assessed during training.

This can be accomplished by specifying the validation dataset to the fit() function through the validation_data argument. For instance:

Alternatively, the fit() function can automatically split your training dataset into train and validation sets on the basis of a percentage split mentioned through the validation_split argument.

The validation_split is a value between 0 and 1 and defines the percentage amount of the training dataset to leverage for the validation dataset. For instance,

…

model.fit(train_X, train_y, validation_split=0.3)

In both scenarios, the model is not trained on the validation dataset. Rather, the model is assessed on the validation dataset at the conclusion of every training epoch.

Monitoring Model Performance

The loss function selected to be optimized for your model is calculated at the conclusion of every epoch.

To callbacks, this is made available through the name “loss”.

If a validation dataset is specified to the fit() function through the validation_data or validation_split arguments, then the loss on the validation dataset will be made available through the name “val_loss”

Extra metrics can be surveyed during the training of the model.

They can be specified when compiling the model through the “metrics” argument to the compile function. This argument takes a Python list for known metric functions, like ‘mse’ for mean squared error and ‘accuracy’ for accuracy. For instance:

…

model.compile(…, metrics=[‘accuracy’])

If extra metrics are monitored during the course of training, which are also available to the callbacks through the same name, like ‘accuracy’ for accuracy on the training dataset and ‘val_accuracy’ for the accuracy on the validation dataset. Or, ‘mse’ for mean squared error on the training dataset and ‘val_mse’ on the validation dataset.

Early Stopping in Keras

Keras assists the early stopping of training through a callback referred to as EarlyStopping.

This callback facilitates you to specify the performance measure to monitor, the trigger, and upon triggering, it will cease the training procedure.

The EarlyStopping callback is configured when instantiated through arguments.

The “monitor” enables you to specify the performance measure to monitor in order to end training. Recall from the prior section that the calculation of measures on the validation dataset will have the ‘val_’ prefix, like ‘val_loss’ for the loss on the validation dataset.

es = EarlyStopping(monitor=’val_loss’)

On the basis of the selection of performance measure, the “mode” argument will require to be specified as whether the objective of the selected metric is to increase (maximize or ‘max’) or to decrease (minimize or ‘min’)

For instance, we would look for a minimum of validation loss and a minimum for validation mean squared error, while we would seek a maximum for validation precision/accuracy.

es = EarlyStopping(monitor=’val_loss’, mode=’min’)

By default, mode is set to ‘auto’ and knows that you wish to minimize loss or maximize accuracy.

That is all that is required for the simplest variant of early stopping. Training will cease when the selected performance measure ceases improving. To find out the training epoch on which training was ceased, the “verbose” argument can be set to 1. After ceasing, the callback will print the epoch number.

es = EarlyStopping(monitor=’val_loss’, mode=’min’, verbose=1)

Usually, the first indicator of no subsequent improvement might not be the ideal time to cease training. This is due to the fact that the model may coast into a plateau of no improvement or even get a little bit worse prior to getting much better.

We can account for this by including a delay to the trigger in terms of the number of epochs on which we would wish to observe no improvement. This can be performed by setting the “patience” argument.

es = EarlyStopping(monitor=’val_loss’, mode=’min’, verbose=1, patience=50)

The precise amount of patience will vary amongst models and problems. Reviewing plots of your performance measure can be very useful to obtain a notion of how noisy the optimization procedure for your model on your data might be.

By default, any alteration in the performance measure, no difference how fractional, will be considered an enhancement. You might wish to consider an enhancement that is a particular increment, like 1 unit for mean squared error  or 1% for accuracy. This can be specified via the “min_delta” argument.

es = EarlyStopping(monitor=’val_accuracy’, mode=’max’, min_delta=1)

Lastly, it might be desirable to only cease training if performance stays above or below a provided threshold or baseline. For instance, if you are acquainted with the training of the model (e.g. learning curves) and know that after a validation loss of a provided value is accomplished that there is no point in persisting with training. This can be specified by setting the “baseline” argument.

This might be more beneficial when fine tuning a model, after the initial wild fluctuations in the performance measure observed in the preliminary stages of training a new model are past.

es = EarlyStopping(monitor=’val_loss’, mode=’min’, baseline=0.4)

Checkpointing in Keras

The EarlyStopping callback will cease training after being triggered, but the model at the conclusion of training might not be the model with ideal performance on the validation dataset.

An extra callback is needed that will save the ideal model observed during the course of training for later use. This is the ModelCheckpoint callback.

The ModelCheckpoint callback is flexible in the fashion that it can be leveraged, but in this scenario we will leverage it only to save the ideal model witnessed during training, as defined by a selected performance measure on the validation dataset.

Saving and loading models needs that HDF5 support has been setup on your workstation. For instance, leveraging the pip Python installer, this can be accomplished as follows:

sudo pip install h5py

The callback will save the model to file, which needs that a pathway and a filename be mentioned through the first argument.

mc = ModelCheckpoint(‘best_model.h5’)

The loss function of preference to be monitored can be mentioned through the monitor argument, in the same fashion as the EarlyStopping callback. For instance, loss on the validation dataset (the default)

mc = ModelCheckpoint(‘best_model.h5′, monitor=’val_loss’)

Also, as with the EarlyStopping callback, we must mention the “mode” as either reducing or maximizing the performance measure. Again, the default is ‘auto’, which is aware of the traditional performance measures.

mc = ModelCheckpoint(‘best_model.h5′, monitor=’val_loss’, mode=’min’)

Lastly, we are concerned in only the very ideal model witnessed during training, instead of the best contrasted to the prior epoch, which might not be the ideal overall if training is noisy. This can be accomplished by setting the “save_best_only” argument to True.

mc = ModelCheckpoint(‘best_model.h5′, monitor=’val_loss’, mode=’min’, save_best_only=True)

This is all that is required to make sure that the model with the ideal performance is saved when leveraging early stopping, or in general.

It might be fascinating to be aware of the value of the performance measure and at which epoch the model was saved. This can be printed by the callback by setting the “verbose” argument to “1”

mc = ModelCheckpoint(‘best_model.h5′, monitor=’val_loss’, mode=’min’, verbose=1)

The saved model can then be loaded and assessed each time by calling the load_model() function.

Now that we are aware of how to leverage the early stopping and model checkpoint APIs, let’s observe a worked instance.

Early Stopping Case Study

In this portion of the blog, we will illustrate how to leverage early stopping to minimize overfitting of an MLP on a simplistic binary classification problem.

This instance furnishes a template for application of early stopping to your own neural network for classification and regression problems.

Binary Classification Problems

We will leverage a conventional binary classification problem that defines dual semi-circles of observations, one semi-circle for every class.

Every observation has dual input variables with the same scale and a class output value of either 0 or 1. This dataset is referred to as the “moons” dataset as the shape of the observations in every class when plotted.

We can leverage the make_moons() function to produce observations from this problem. We will include noise to the data and seed the random number generator so that the same samples are produced every time the code is run.

# generate 2d classification dataset

X, y = make_moons(n_samples=100, noise=0.2, random_state=1)

We can plot the dataset where the dual variables are taken as x and y coordinates on a graph and the class value is taken as the colour of the observation.

The full instance of producing the dataset and plotting it is detailed below.

Running the instance develops a scatter plot displaying the semi-circle or moon shape of the observations in every class. We can observe the noise in the dispersal of the points making the moons less obvious.

This is a good evaluation problem as the classes cannot be separated by a line, e.g., are not linearly separable, needing a nonlinear strategy like a neural network to address.

We have only produced 100 samples, which is minimal for a neural network, furnishing the opportunity to overfit the training dataset and possess higher error on the evaluation dataset. A good case for leveraging regularization. Further, the samples possess noise, providing the model an opportunity to learn aspects of the samples that don’t generalize.

Overfit Multilayer Perceptron

We can produce an MLP model to tackle this binary classification problem.

The model will have a single hidden layer with additional nodes than may be needed to solve this problem, furnishing an opportunity to overfit. We will also undertake training of the model for longer than is needed to make sure the model overfits.

Prior to defining the model, we will split the dataset into train and evaluation sets, leveraging 30 instances to train the model and 70 to assess the fit model’s performance.

Then, we can define the model.

The hidden layer leverages 500 nodes and the rectified linear activation function. A sigmoid activation function is leveraged in the output layer in order to forecast class values of 0 or 1. The model receives optimization leveraging the binary cross entropy loss function, appropriate for binary classification problems and the effective Adam version of gradient descent.

The defined model is then fitted on the training data for 4,000 epochs and the default batch size of 32.

We will also leverage the evaluation dataset as a validation dataset. This is merely a simplification for this instance. Practically, you would split the training set into train and validation and also hold back a test set for final model evaluation.

We can assess the performance of the model on the evaluation dataset and report the outcome.

Lastly, we will plot the loss of the model on both the train and test set every epoch.

If the model does as a matter of fact, overfit the training dataset, we would predict/expect the line plot of loss (and accuracy) on the training set to continue to increase and the test set to rise and then fall again as the model goes about learning statistical noise in the training dataset.

We can connect all of these pieces together, the full example is detailed below.

Running the instance reports the model performance on the train and evaluation datasets.

We can observe that the model has improved performance on the training dataset than the evaluation dataset, one potential indicator of overfitting.

Your outcomes might demonstrate variance provided the stochastic nature of the algorithm or evaluation process, or variations in numerical accuracy. Consider running the instance a few times and contrast the average outcome.

As the model is drastically overfitted, we typically would not expect much, if any, variation in the accuracy across repetitive runs of the model on the same dataset.

Train: 1.000, Test: 0.914

A figure is developed displaying line plots of the model loss on the train and test sets.

We can observe that expected/predicted shape of an overfitted model where test precision/accuracy increases to a point and then starts to reduce again.

Reviewing the figure, we can additionally observe flat spots in the ups and downs in the validation loss. Any early stopping will have to account for these behaviours. We would also expect/predict that a good time to cease training might be around epoch 800.

Overfit MLP with early stopping

We can update the instance and include very simple early stopping.

As soon as the loss of the model starts to increase on the evaluation dataset, we will cease training.

To start with, we can define the early stopping callback.

# simple early stopping

es = EarlyStopping(monitor=’val_loss’, mode=’min’, verbose=1)

We can then update the call to the fit() function and mention a listing of callback through the “callback” argument.

# fit model

history = model.fit(trainX, trainy, validation_data=(testX, testy), epochs=4000, verbose=0, callbacks=[es])

The total instance with the addition of simple early stopping is detailed below.

# mlp overfit on the moons dataset with simple early stopping

from sklearn.datasets import make_moons

from keras.models import Sequential

from keras.layers import Dense

from keras.callbacks import EarlyStopping

from matplotlib import pyplot

# generate 2d classification dataset

X, y = make_moons(n_samples=100, noise=0.2, random_state=1)

# split into train and test

n_train = 30

trainX, testX = X[:n_train, :], X[n_train:, :]

trainy, testy = y[:n_train], y[n_train:]

# define model

model = Sequential()

model.add(Dense(500, input_dim=2, activation=’relu’))

model.add(Dense(1, activation=’sigmoid’))

model.compile(loss=’binary_crossentropy’, optimizer=’adam’, metrics=[‘accuracy’])

# simple early stopping

es = EarlyStopping(monitor=’val_loss’, mode=’min’, verbose=1)

# fit model

history = model.fit(trainX, trainy, validation_data=(testX, testy), epochs=4000, verbose=0, callbacks=[es])

# evaluate the model

_, train_acc = model.evaluate(trainX, trainy, verbose=0)

_, test_acc = model.evaluate(testX, testy, verbose=0)

print(‘Train: %.3f, Test: %.3f’ % (train_acc, test_acc))

# plot training history

pyplot.plot(history.history[‘loss’], label=’train’)

pyplot.plot(history.history[‘val_loss’], label=’test’)

pyplot.legend()

pyplot.show()

Running the instance reports the model performance on the train and evaluation datasets.

Your outcomes might demonstrate variance provided the stochastic nature of the algorithm or assessment procedure, or variations in numerical accuracy. Consider running the instance a few times and contrast the average outcome.

We can also observe that the callback ceased training at epoch 200. This is too early as we would expect an early stop to be around epoch 800. This is additionally highlighted by the classification precision on both the train and test sets, which is worse than no early stopping.

Reviewing the line plot of train and evaluation loss, we can indeed observe that training was ceased at the point when validation loss started to plateau for the first time.

We can enhance the trigger for early stopping by waiting a while prior to stopping.

This can be accomplished by setting the “patience” argument.

In this scenario, we will wait 200 epochs prior to training being ceased. Particularly, this means that we will enable training to continue for up to an extra 200 epochs after the point that validation loss began to degrade, provided the training process an opportunity to get across flat spots or identify some extra improvement.

# patient early stopping

es = EarlyStopping(monitor=’val_loss’, mode=’min’, verbose=1, patience=200)

The complete instance with this modification is detailed below.

Running the instance, we can observe that training was ceased much later, in this scenario after epoch 1,000.

Your outcomes might demonstrate variance provided the stochastic nature of the algorithm or evaluation process, or variations in numerical accuracy. Consider running the instance a few times and contrast the average outcome.

We can additionally observe that the performance on the evaluation dataset is better than not leveraging any early stopping.

Epoch 01033: early stopping

Train: 1.000, Test: 0.943

Reviewing the line plot of loss during the course of training, we can observe that the patience allowed the training to progress beyond some small flat and bad spots.

We can also observe that the test loss began to increase again in the last approximately 100 epochs.

This implies that even though the performance of the model has enhanced, we might not have the ideal performing or most stable model at the conclusion of training. We can tackle this by leveraging a ModelCheckpoint callback.

In this scenario, we are concerned with saving the model with the ideal accuracy on the evaluation dataset. We could also seek the model with the ideal loss on the evaluation dataset, but this might or might not correlate to the model with the ideal accuracy.

This highlights a critical notion in model choice. The notion of the “best” model during training might conflict when assessed leveraging differing performance measures. Attempt to select models on the basis of the metric by which they will be assessed and put forth in the domain. In a balanced binary classification problem, this will most probably be classification accuracy. Thus, we will leverage accuracy on the validation in the ModelCheckpoint callback to save the ideal model observed during training.

mc = ModelCheckpoint(‘best_model.h5′, monitor=’val_accuracy’, mode=’max’, verbose=1, save_best_only=True)

During the course of training, the total model will be saved to the file “best_model.h5” only when accuracy on the validation dataset enhances overall across the total training process. A verbose output will also provide information to us as to the epoch and accuracy value every time the model is saved to the same file (e.g. overwritten)

This new extra callback can be included to the listing of callbacks when calling the fit() function.

history = model.fit(trainX, trainy, validation_data=(testX, testy), epochs=4000, verbose=0, callbacks=[es,mc])

We are no longer concerned in the line plot of loss during the course of training, it will be much the same as the prior run.

Rather, we wish to load the saved model from file and evaluate its performance on the test dataset.

The full instance with these modifications is detailed below.

Running the instance, we can observe that the verbose output from the ModelCheckpoint callback for both when a fresh best model is saved and from when no enhancement/improvements were observed.

We can observe that the ideal model was observed at epoch 879 during this run.

Your outcomes may demonstrate variance provided the stochastic nature of the algorithm or evaluation process, or variations in numerical accuracy. Consider executing the instance a few times and contrast the average outcome.

Again, we can observe that early stopping continued patiently until after epoch 1,000. Observe that epoch 880+ a patience of 200 is not epoch 1044. Remember that early stopping is monitoring loss on the validation dataset and that the model checkpoint is saving models on the basis of precision. As such, the patience of early stopping began at an epoch other than 800.

In this scenario, we don’t observe any subsequent enhancement in model precision/accuracy on the evaluation dataset. Nonetheless, we have followed a good practice.

Why not monitor validation precision/accuracy for early stopping?

This is a brilliant question. The primary reason is that accuracy is a coarse measure of model performance during training and that loss furnishes more nuance when leveraging early stopping with classification problems. The same measure might be leveraged for early stopping and model checkpointing in the scenario of regression, like mean squared error.

Extensions

This section details some ideas for extension of the guide that you might desire to explore.

• Use Accuracy: Update the instance to survey accuracy on the evaluation dataset instead of loss, and plot learning curves displaying accuracy.
• Use True Validation Set: Update the instance to split the training set into train and validation sets, then assess the model on the evaluation dataset.
• Regression instance: Develop a new instance of leveraging early stopping to address overfitting on a simplistic regression problem and monitoring mean squared error.

Further Reading

This section furnishes additional resources on the subject if you are seeking to delve deeper.

API

H5Py Installation Documentation

Keras Regularizers API

Keras Core Layers API

Keras Convolutional Layers API

Keras Recurrent Layers API

Keras Callbacks API

sklearn.datasets.make_moons API

Conclusion

In this guide, you found out about the Keras API for including early stopping to overfit deep learning neural network models.

Particularly, you learned:

• How to monitor the performance of a model during the course of training leveraging the Keras API.
• How to develop and configure early stopping and model checkpoints callbacks leveraging the Keras API.
• How to minimize overfitting by including a early stopping to a current model.