It can be a challenge to produce a neural network predictive model for a new dataset.

One strategy is to initially inspect the dataset and generate ideas for what models might function, then look into the learning dynamics of simple models on the dataset, then ultimately develop and tune a model for the dataset with a solid test harness.

This process can be leveraged to develop efficient neural network models for classification and regression predictive modelling problems.

In this guide, you will find out how to produce a multilayer perceptron neural network model for the banknote binary classification dataset.

After going through this guide, you will be aware of:

• How to load and summarize the banknote dataset and leverage the outcomes to indicate data preparations and model configurations to leverage.
• How to look into the learning dynamics of simplistic MLP models on the dataset.
• How to develop solid estimates of model performance, tune model performance and make forecasts on fresh data.

Tutorial Summarization

This tutorial is subdivided into four portions, which are:

1] Banknote classification dataset

2] Neural network learning dynamics

3] Robust model evaluation

4] Final model and make forecasts

Banknote Classification Dataset

The first stage is to go about defining and exploring the dataset.

We will be operating with the “Banknote” standard binary classification dataset.

The banknote dataset consists of forecasting whether a provided banknote is genuine provided a number of measures taken from a photograph.

The dataset consists of 1,372 rows with five numeric variables. It is a classification issue with two classes (binary classification).

Below is furnished a list of the five variables within the dataset.

• Variance of Wavelet Transformed image (continuous)
• Skewness of Wavelet Transformed image (continuous)
• kurtosis of Wavelet Transformed image (continuous)
• entropy of image (continuous)
• class (integer)

Below is a sample of the first five rows of the dataset.

We can go about loading the dataset directly from the URL and goes about reporting the shape of the dataset.

In this scenario, we can confirm the dataset possesses five variables (4 input and a single output) and that the dataset possesses has 1,372 rows of data.

This is not several rows of data for a neural network and indicates that a small network, probably with regularization, would be appropriate.

It also indicates that leveraging k-fold cross-validation would be a good idea provided that it will give a more reliable estimate of model performance than a train/test split and as a single model will fit in seconds rather than hours or days with the biggest datasets.

(1372, 5)

Then, we can learn more about the dataset by observing summary statistics and a plot of the data.

Running the instance first loads the data prior and then prints summary stats for every variable.

We can observe that values demonstrate variance with differing means and standard deviations, probably some normalization or standardization would be needed before modelling.

A histogram plot is then developed for every variable.

We can observe that probably the first two variables possess a Gaussian-like distribution and the next two input variables might possess a skewed Gaussian distribution or an exponential distribution.

We might possess some advantages in leveraging a power transform on every variable in order to make the probability distribution less skewed which will probably enhance model performance.

Now that we are acquainted with the dataset, let’s look into how we might develop a neural network model.

Neural Network Learning Dynamics

We will produce a Multilayer Perceptron (MLP) model for the dataset leveraging TensorFlow.

We cannot know what model architecture of learning hyperparameters would be adequate or best for this dataset, so we must go about experimenting and find out what functions well.

Provided that the dataset is small, a small batch size is likely a solid idea, for example, 16 or 32 rows. Leveraging the Adam version of stochastic gradient descent is a good idea when beginning as it will automatically adapt the learning pace and operates well on most datasets.

Prior to evaluating the models in earnest, it is a good idea to review the learning dynamics and tune the model architecture and learning configuration till we possess stable learning dynamics, then observe obtaining the most out of the model.

We can perform this by leveraging a simple train/test split of the data and review plots of the learning curves. This will assist us in observing if we are over-learning or under-learning, then we can adapt the configuration accordingly.

To start with, we must make sure that all input variables are floating-point values and encode the target label as integer values 0 and 1.

Then, we can split the dataset into input and output variables, then into 67/33 train and test sets.

We can define a minimal MLP model. In this scenario, we will leverage one hidden layer with ten nodes and one output layer (selected randomly). We will leverage the ReLU activation function within the hidden layer and the “he_normal” weight initialization, as combined, they are a good practice.

The output of the model is a sigmoid activation for binary classification and we will minimize binary cross-entropy loss.

We will fit the model for 50 training epochs (selected randomly) with a batch size of 32 as it is a small dataset.

We are fitting the model on raw data, which we think may be a good idea, but it is a critical beginning point.

At the conclusion of training, we will assess the model’s performance on the test dataset and report performance as the classification precision.

Lastly, we will plot learning curves of the cross-entropy loss on the train and test sets during training.

Connecting all of this together, the complete instance of assessing our first MLP on the banknote dataset is detailed below.

Running the instance first fits the model on the training dataset, then reports the classification precision on the test dataset.

Your outcomes may demonstrate variance provided the stochastic nature of the algorithm or evaluation procedures, or variations in numerical accuracy. Take up running the instance a few times and contrast the average outcome.

In this scenario, we can observe that the model accomplished great or perfect precision of 100%. This might indicate that the forecast problem is simple and/or that neural networks are a good fir for the problem.

Accuracy: 1.000

Line plots of the loss on the train and test sets are then developed.

We can observe that the model appears to converge well and does not display any indicators of overfitting or underfitting.

We did very well on our first try.

Now that we possess some notion of the learning dynamics for a simplistic MLP model on the dataset, we can look into developing a more solid evaluation of model performance on the dataset.

Robust Model Evalutation

The k-fold cross-validation process can furnish a more reliant estimate of MLP performance, even though it can be really slow.

This is due to the fact that k models must be fitted and assessed. This is not a issue when the dataset size is small, like the banknote dataset.

We can leverage the StratifiedKFold class and enumerate every fold manually, fit the model, assess it, and then report the mean of the evaluation scores at the conclusion of the procedure.

We can leverage this framework to produce a reliable estimate of MLP model performance with our base configuration, and even with an array of differing data preparations, model architectures, and learning configurations.

It is critical that we initially developed a comprehension of the learning dynamics of the models on the dataset in the prior section prior to leveraging a k-fold cross-validation to estimate the performance. If we began to tune the model directly, we might get good outcomes, but if not, we may have no notion of why, for example, that the model was over or under fitting.

If we make major changes to the model again, it is a good idea to go back and confirm that the model is converging in the correct fashion.

The full example of this framework to assess the base MLP model from the prior section is detailed below.

Running the instance reports the model performance every iteration of the evaluation process and reports the mean and standard deviation of classification precision at the conclusion of the run.

Your outcomes might demonstrate variance provided the stochastic nature of the algorithm or evaluation procedure, or variations in numerical accuracy. Take up running the instance a few times and contrast the average outcome.

In this scenario, we can observe that the MLP model accomplished a mean precision of approximately 99.9%.

This provides confirmation of our expectation that the base model config functions really well for this dataset, and indeed the model is a good fit for the problem and probably the problem is quite trivial to solve.

This is a surprise as we would have expected some data scaling and probably  power transform to be needed.

Then, let’s look at how we might fit a final model and leverage it to make forecasts.

Final Model and Make Predictions

After we select a model config, we can train a final model on all available data and leverage it to make predictions on fresh data.

In this scenario, we will leverage the model with dropout and a minimal batch size as our final model.

We can prep the data and fit the model as prior, even though on the entire dataset rather than a training subset of the dataset.

We can then leverage this model to make forecasts on new data.

To start with, we can define a new row of data.

We took this row from the first row of the dataset and the expected label is a ‘0’

We can then make a forecast.

Then invert the transform on the prediction, so we can leverage or interpret the outcome in the right label (which is just an integer for this dataset)

And in this scenario, we will merely report the prediction.

Connecting all of this together, the full instance of fitting a final model for the banknote dataset and leveraging it to make a forecast on new data is detailed below.

Running the instance fits the model on the entire dataset and makes a forecast for a singular row of new data.

Your outcomes might demonstrate variance provided the stochastic nature of the algorithm or evaluation process, or variations in numerical accuracy. Take up running the instance a few times and contrast the average outcome.

In this scenario, we can observe that the model forecasted a ‘0’ label for the input row.

Predicted: 0.0

Conclusion

In this guide, you found out how to develop a multilayer perceptron neural network model for the banknote binary classification dataset.

Particularly, you learned:

• How to load and summarize the banknote dataset and leverage the outcomes to indicate data preparations and model configurations to leverage.
• How to look into the learning dynamics of simple MLP models on the dataset.
• How to develop robust estimates of model performance, tune model performance and make forecasts on fresh data.