Neural networks are the workhorse of several of the algorithms produced by DeepMind. For instance, AlphaGo leverages convolutional networks to assess board positions in the game of Go and DQN and Deep Reinforcement Learning algorithms leverage neural networks to select actions to play at super-human level on video gaming titles.

This blog post by AICoreSpot puts forth some of the research in advancing the capacities and training protocol of neural networks referred to as “Decoupled Neural Interfaces Leveraging Synthetic Gradients”. This research provides a way to enable neural networks to interact, to learn and to transmit messages amongst themselves, in a decoupled scalable fashion paving the path for several neural networks to interact with one another or enhancing the longer-term temporal dependency of recurrent networks. This is accomplished by leveraging a model to approximate error gradients, instead of by computing error gradients explicitly with backpropagation. The remainder of this post goes by the assumption that you have some familiarity with neural networks and how to go about training them.

Neural networks and the issue of locking

If you look at any layer or module within a neural network, it can just be updated after all the subsequent modules of the network have been executed, and gradients have been backpropagated to it. For instance, observe this simple feed forward network.

In this image, following Layer 1’s process of the input, it can just be updated following the output activations (black lines) have been propagated through the remainder of the network, produced a loss, and the error gradients (green lines) backpropagated through every layer until Layer 1 is reached. This sequence of operations implies that Layer 1 has to wait for the backwards and forwards computation of Layers 2 and 3 prior to updating. Layer 1 is locked, coupled, to the remainder of the network.

Why is this an issue? Obviously, for a simple feed-forward network as illustrated we don’t need to be concerned about this problem. But think of a complicated system of several networks, acting in several environments at asynchronous and irregular timescales.

Or a big distributed network spread out over several machines. At times needing all modules in a network to wait for all other modules to execute and backpropagate gradients is really time intensive or even intractable. If we go about decoupling the interfaces, the connections – amongst modules, each module can be updated independently, and is not restricted to the remainder of the network.

So how can one go about decoupling interfaces, which means decouple the connections amongst network modules – and still enable the modules to learn to interact? They remove the dependence on backpropagation to obtain error gradients, and rather learn a parametric model which forecasts what the gradients will be based upon only local data. We refer to these forecasted gradients as synthetic gradients.

The synthetic gradient model takes in activations from a module and generates what it forecasts will be the error gradients – the gradient of the loss of the network with regard to the activations.

Reverting back to the simple feed-forward network example, if we possess a synthetic gradient model we can perform the following:

…and leverage the blue synthetic gradients to update Layer 1 prior to the remainder of the network even being executed.

The synthetic gradient model itself has received training to regress target gradients – these target gradients could be the actual gradients backpropagated from the loss or other synthetic gradients which have been backpropagated from a further downstream synthetic gradient model.

This mechanism is generic for a connection amongst any two modules, not only in a feed-forward network. The play-by-play functioning of this mechanism is demonstrated below, where the modification of colour of a module signifies an update to the weights of that particular module.

Leveraging decoupled neural interfaces (DNI) thus eradicates the locking of prior modules to subsequent modules within a network. In experiments from the research, it is illustrated that they can train convolutional neural networks for CIFAR-10 image classification where each layer is decoupled leveraging synthetic gradients to the same precision as leveraging backpropagation. It’s critical to identify that DNI does not magically enable networks to train with no actual gradient data. The real gradient data does percolate backwards through the network, but just with reduced speed and across several training iterations, through the losses of synthetic gradient models. The synthetic gradient models approximate and smooth over the absence of real gradients.

A worthwhile question at this juncture would be to ask how much computational intricacy do these synthetic gradient models add – probably you would require a synthetic gradient model architecture that is as complicated as the network itself. Quite shockingly, the synthetic gradient models can be very simplistic. With regards to feed-forward nets, it was actually discovered that even a singular linear layer functions well as a synthetic gradient model. Consequently it is both very simple to train and therefore generates synthetic gradients swiftly.

DNI can be applied to any generic neural network architecture, not only feed-forward networks. A fascinating application is to recurrent neural networks (RNNs). An RNN possesses a recurrent core which is unrolled, repetitively applied – in processing sequential information. Ideally to go about training an RNN we would unroll the core over the entire sequence (which could be limitlessly long), and leverage backpropagation through time (BPTT) to propagate error gradients backwards through the graph.

Although, practically, we can just afford to unroll for a set number of steps owing to memory restrictions and the requirement to actually compute an update to the core model consistently. This is referred to as truncated backpropagation through time, and demonstrated below for a truncation of three stages:

The alteration in colour of the core demonstrates an update to the core, that the weights have received updation. In this instance, truncated BPTT appears to tackle some problems with training – we can now go about updating our core weights each three steps and just require three cores in memory. But, the fact that there is no backpropagation of error gradients over more than three steps implies that the update to the core will not be directly impacted by errors committed more than 2 steps into the future. This restricts the temporal dependency that the RNN can learn to model.

What if rather than doing no backpropagation amongst the boundary of BPTT we leveraged DNI and generate synthetic gradients, which model what the error gradients of the future are bounds to be? We can integrate a synthetic gradient framework into the core so that at every time step, the RNN core generates not just the output, but also the synthetic gradients. In this scenario, the synthetic gradients would be the forecasted gradients of all future losses with regard to the hidden state activation of the prior timestep. The synthetic gradients are only leveraged at the boundaries of truncated BPTT where we would have had no gradients prior.

This can be executed over the course of training very efficiently, it just needs us to keep an extra core in memory as demonstrated below. Here a green dotted border signifies just computing gradients with regards to the input state, whereas a solid green border additionally computes gradients with regards to the core’s parameters.

By leveraging DNI and synthetic gradients with an RNN, we are approximating performing backpropagation across an limitlessly unrolled RNN. Practically, this has the outcome of RNNs which can model longer temporal dependencies. Here’s an instance outcome demonstrating this from the research.

Penn Treebank Test Error During Training (lower is better):

This graph demonstrates the application of an RNN trained on next character prediction on Penn Treebank, a language modelling issue. On the y-axis the bits-per-character (BTC) is provided, where smaller means better. The x-axis is the number of characters observed by the model as training goes forth. The dotted blue, red and grey lines are RNNs which have received training with truncated BPTT, unrolled for 8 steps, 20 steps, and 40 steps – the more the number of steps the RNN is unrolled prior to performing backpropagation through time, the more improved the model is, the slower it goes about training. When DNI is leveraged on the RNN unrolled 8 steps (solid blue line) the RNN is capable to capture the longer term dependency on the 40-step model, but is trained doubly as quick (both with regards to data and wall clock time on a conventional desktop machine with a singular graphics processing unit.)

To drive the point home once more, adding synthetic gradient models enables us to go about decoupling the updates amongst two portions of a network. DNI can additionally be applied on hierarchal RNN models – system of two (or more) RNNs operating at differing timescales. DNI considerably enhances the training speed of these models by facilitating the update rate of higher level modules.

The hope is that from the descriptions, details, and explanations in this post, and a short look at few of the experiments reported in the research, it is obvious that it is feasible to develop decoupled neural interfaces. This is performed by developing a synthetic gradient model which takes in local data and forecasts what the error gradient could be. At a high level, this can be viewed of as a communication protocol amongst two modules. One module transmits a message (present activations) another one is on the receiving end, and assesses it leveraging a model of utility (the synthetic gradient model). The model of utility enables the receiver to furnish instant feedback (synthetic gradient) to the transmitter, instead of having to wait for the assessment of the actual utility of the message (through backpropagation). This framework can also be viewed through an error critic perspective and is similar in kind to leveraging a critic in reinforcement learning.

These decoupled neural interfaces enable distributed training of networks, improve the temporal dependency learned with RNNs, and hasten hierarchal RNN systems. It is thrilling to look forward to the future and see what it holds for DNI, as the belief is that this is going be a critical basis for opening up increasingly modular, decoupled, and asynchronous model architectures.