In the literature, word Jacobian is often interchangeably leveraged to reference to both the Jacobian matrix or its determinant.

Both the matrix and the determinant possess useful and critical applications, within machine learning, the Jacobean matrix goes about aggregating the partial derivatives that are required within backpropagation; the determinant is good in the procedure of altering between variables.

In this guide, you will find out about the Jacobian.

After going through this guide, you will be aware of:

• That the Jacobian matrix gathers all first-order partial derivatives of a multivariate function that can be leveraged for backpropagation.
• The Jacobian determinant is good in altering amongst variables, where it function as a scaling factor between one coordinate space and another.

Tutorial Summarization

This guide is subdivided into three portions, which are:

• Partial derivatives within machine learning
• The Jacobian Matrix
• Other uses of the Jacobian

Partial Derivatives within Machine Learning

We have so far observed gradients and partial derivatives as being critical for an optimization algorithm to update, say, the model weights of a neural network to attain an optimum set of weights. The leveraging of partial derivatives allows every weight to be updated autonomously of the others, by quantifying the gradient of the error curve with regard to one weight in turn.

Several of the functions that we typically work with within machine learning are multivariate, vector-valued functions, which implies that they map several real inputs, n to multiple real outputs, m:

For instance, take up a neural network that categorizes grayscale imagery into various classes. The function being implemented by such a classifier would the map the n pixel values of every single-channel input image, to m output probabilities of belonging to every one of the differing classes.

When undertaking training of a neural network, the backpropagation algorithm is accountable for sharing back the error quantified at the output layer, amongst the neurons consisting of the differing hidden layers of the neural network, till it attains the input.

The basic principle of the backpropagation algorithm in modifying the weights within a network is that every weight within a network should be updated proportionally to the sensitivity of the overall error of the network to modifications in that weight.

This sensitivity of the cumulative error of the network to alterations in any one specific weight is quantified in terms of the rate of change, which, in turn, is quantified by taking the partial derivative of the error with regards to the same weight.

For the sake of simplicity, let’s assume that one of the hidden layers of some specific network is made up of only a singular neuron, k. We can indicate this in terms of a singular computational graph.

Again, for the sake of simplicity, let’s assume that a weight, w_k is applied to an input of this neuron to generate an output, z_k, according to the function that this neuron is implementing (which includes the nonlinearity). Then, the weight of this neuron can be linked to the error at the output of the network as follows (the upcoming formula is referred to as the chain rule of calculus, but more on this subject later.

Here, the derivative dz_k / dw_k  first connects the weight, w_k to the output, z_k while the derivative, derror / dz_k, subsequently connects the output, z_k, to the network error.

It is more typically the scenario that we’d have several connected neurons making up the network, every one attributed a different weight. As we posses more interest in such a scenario, then we can do a generalization beyond the scalar scenario to take up several inputs and several outputs:

This sum of terms can be indicated more concisely as follows:

Or, equivalently, in vector notation leveraging the del operator ∇, to indicate the gradient of the error with regard to either the weights w_k, or the outputs, z_k

The back-propagation algorithm is made up of executing such a Jacobian-gradient product for every operation within the graph.

This implies that the backpropagation algorithm can go about relating the sensitivity of the network error to alterations in the weights, via a multiplication by the Jacobian matrix (∂z_k / ∂w_k)^T.

Therefore, what does this Jacobian matrix consist of?

The Jacobian Matrix gathers all first-order partial derivatives of a multivariate function

Particularly, take up first a function that maps u real inputs, to a singular real output:

Then, for an input vector, x of length u, the Jacobian vector of size, u x 1 can be defined as follows:

Now, take up another function that maps u real inputs, to v real outputs:

Then, for the same input vector, x of length u, the Jacobian is now a u x v matrix J ∈ ℝ^u×v that is defined as follows

Reframing the Jacobian matrix into the machine learning issue taken up prior, while retention of the same number of u real inputs and v real outputs, we identify that this matrix would consist of the following partial derivatives:

Other uses of the Jacobian

A critical strategy when operating with integrals consists of the change of variables (also called as, integration by substitution or u-substitution, where an integral is simplified into another integral that is simpler to compute.

In the single variable case, replacing some variable x, with another variable u, can transform the original function into an easier one for which it is simpler to identify an antiderivative. In the two variable scenario, an extra reason might be that we would also desire to transform the region of terms over which we are integrating into a differing shape.

In the single variable scenario, there’s usually only one reason to wish to modify the variable: to make the function “nice” so we can identify an antiderivative. In the two variable scenario, there is a second possible reason, the two-dimensional region over which we require to integrate is somehow unpleasant, and we wish the region in terms of u and v to be nicer – to be a rectangle, for instance.

When executing a substitution amongst two (or potentially more) variables, the procedure begins with a definition of the variables between which the substitution is going to take place. For instance, x = f(u,v) and y = g(u,v). This is then followed by a conversion of the integral limits dependent on how the functions f, and g will transform the u-v plane into the x-y plane. Lastly, the absolute value of the Jacobian determinant is computed and included, to function as a scaling factor amongst one coordinate space and another.