For instance, deep learning neural networks are fit leveraging stochastic gradient descent, and many conventional optimization algorithms leveraged to fit machine learning algorithms leverage gradient data.

To comprehend what a gradient is, you require to comprehend what a derivative is from the domain of calculus. This consists of how to calculate a derivative and go about interpreting the value. A comprehension of the derivative is directly relevant to comprehending how to calculate and interpret gradients as leveraged in optimization and machine learning.

In this guide by AICoreSpot, you will receive an introduction to the gradient and the derivative in machine learning.

Upon finishing this guide, you will know:

• The derivative of a function is the change of the function for a provided input.
• The gradient is merely a derivative vector for a multivariate function.
• How to calculate and go about interpreting derivatives of a simplistic function.

Tutorial overview

This tutorial is subdivided into five portions, they are:

1. What is a derivative?
2. What is a gradient?
3. Worked instance of calculating derivatives
4. How to go about interpreting the derivative
5. How to calculate the derivative of a function

What is a derivative?

In calculus, a derivative is the rate of modification at a provided point in a real-valued function.

For instance, the derivative f’(x) of function f() for variable x is the rate that the function f() modifies at that point x.

It might alter a lot, for example, be really curved, or might alter a bit, for example, a slight curve, or it might not alter at all, for example, flat or stationary.

A function is differentiable if we can quantify the derivative at all points of input for the function variables. Not every function is differentiable.

After we calculate the derivative, we can leverage it in various ways.

For instance, provided an input value x and the derivative at the point f’(x), we can forecast the value of the function f(x) at a close by point delta_x (change in x) leveraging the derivative, as follows:

• f(x + delta_x) = f(x) + f’(x) * delta_x

Here, we can observe that f’(x) is a line and we are estimating the value of the function at a close by point by shifting along the line by delta_x.

We can leverage derivatives in optimization issues as they inform us how to modify inputs to the target function in a fashion that enhances or reduces the output of the function, so we can get nearer to the minimum or maximum of the function.

Identifying the line that can be leveraged to approximate close by values was the primary reason for the initial development of differentiation. This line is called the tangent line or the slope of a function at a provided point.

An instance of the tangent line of a point for a specific function is furnished below, obtained from page 19 of “Algorithms for optimization”

Technically, the derivative detailed so far is referred to as the first derivative or first-order derivative.

The second derivative (or second-order derivative) is the derivative of the derivative function. That is, the speed of change of the speed of change or how much the modification in the function changes.

• First derivative: Rate of change of the target function.
• Second derivative: Rate of change of the first derivative function.

A natural utilization of the second derivative is to approximate the starting derivative at a close by point, just as we can leverage the first derivative to estimate the value of the target function at a close by point.

Now that we are aware of what a derivative is, let’s observe the gradient.

What is a gradient?

A gradient is a derivative of a function that has in excess of a singular input variable.

It is a term leveraged to refer to the derivative of a function from the viewpoint of the field of linear algebra. Particularly, when linear algebra meets calculus, referred to as vector calculus.

The gradient is the generalization of the derivative to multivariate functions. It captures the local slope of the function, enabling us to forecast the impact of taking a small step from a point in any direction.

Several input variables combined define a vector of values, for example, a point in the input space that can be furnished to the target function.

The derivative with regards to a target function with a vector of input variables is likewise a vector. This vector of derivatives for every input variable is the gradient.

Gradient (vector calculus): A vector of derivatives with regards to a function that takes a vector of input variables.

You might remember from school algebra or pre-calculus, the gradient is also in reference in a general sense to the slope of a line on a two-dimensional plot.

It is calculated as the rise (modification on the y-axis) of the function divided by the run (change in x-axis) of the function, simplified to the rule: “rise over run”

Gradient (algebra): Slope of a line, calculated as rise over run.

We can observe that this is a simplistic and rough approximation of the derivative with regards to a function with a singular variable. The derivative function from calculus is more accurate as it leverages limits to identify the precise slope of the function at a point. This concept of gradient from algebra is connected, but not directly useful to the concept of a gradient as leveraged in optimization and machine learning.

A function that takes several input variables, for example, a vector of input variables, may be called as a multivariate function.

The partial derivative of a function with regard to a variable is the derivative going by the assumption that all other input variables are held constant.

Every component in the gradient (vector of derivatives) is referred to as a partial derivative of the targeted function.

A partial derivative goes by the assumption that all other variables of the function remain constant.

Partial derivative: A derivative for one of the variables for a multivariate function.

It is good to work with square matrices in linear algebra, and the square matrix of the second-order derivatives is called the Hessian matrix.

The Hessian of a multivariate function is a matrix consisting of all of the second derivatives with regards to the input.

We can leverage gradient and derivative interchangeably, however in the domains of optimization and machine learning, we usually leverage “gradient” as we are usually concerned with multivariate functions.

Now that we are accustomed to the idea of a derivative and of a gradient, let’s observe a worked instance of calculating derivatives.

Worked instance of calculating derivatives

Let’s make the derivative concrete with a worked instance.

To start with, let us define a simplistic 1D function that squares the input and defines the range of valid inputs from -1.0 to 1.0

f(x) = x^2

The instance below samples inputs from this function in 0.1 increments, calculates the function value for every input, and goes about plotting the outcome.

Executing the instance develops a line plot of the inputs to the function (x-axis) and the calculated output of the function (y-axis) 

We can observe the familiar U-shaped referred to as a parabola. 

We can observe a major change or steep curve on the sides of the shape where we would go about expecting a large derivative and a flat region in the middle of the function where we would expect a small derivative.

Let’s confirm these by quantifying the derivative at -0.5 and 0.5 (steep) and 0.0 (flat)

The derivative for the function is quantified as follows:

f’(x) = x * 2

The instance below calculates the derivatives for the particular input points for our goal function.

Executing the instance prints the derivative value for particular input values.

We can observe that the derivative at the steep points of the function is -1 and 1 and the derivative for the flat part of the function is 0.0

Now that we are aware how to go about calculating derivatives of a function, let’s observe at how we might go about interpreting the derivative values.

How to go about interpreting the derivative

The value of the derivative can be understood as the speed of change (magnitude) and the direction (sign)

Magnitude of derivative: how much change

Sign of derivative: Direction of change.

A derivative of 0.1 signifies no modification in the target function, which is called the stationary point.

A function might possess one or more stationary points and a local or global minimum (bottom of a valley) or maximum (peak of a mountain) of the function are instances of stationary points.

The gradient indicates in the direction of steepest ascent of the tangent hyperplane.

The sign of the derivative informs you if the target function is increasing or reducing at that point.

Positive derivative: Function is increasing at that point.

Negative derivative: Function is reducing at that point.

This might be a source of confusion as, observing the plot from the prior section, the values of the function f(x) are increasing on the y-axis for -0.5 and 0.5.

The trick here is to always interpret the plot of the function from left to right, for example, adhere to the values on the y-axis from left to right for input x-values.

Indeed, the values in the range of x=0.5 are reducing if interpreted from left to right, therefore the negative derivative, and the values around x=0.5 are increasing, therefore the positive derivative.

We can visualize that if we desired to identify the minima of the function in the prior section leveraging just the gradient data, we would increase the x input value if the gradient was negative to go downhill, or reduce the value of x input if the gradient was positive to go downhill.

This is the foundation for the gradient descent (and gradient ascent) class of optimization algorithms that have access to function gradient data.

Now that we are aware how to go about interpreting derivative values, let’s observe how we might identify the derivative of a function.

How to calculate the derivative of a function

Identifying the derivative function f’() that outputs the speed of change of a target function f() is referred to as differentiation.

There are several approaches (algorithms) for quantifying the derivative of a function.

In a few scenarios, we can calculate the derivative of a function leveraging the tools of calculus, either manually, or leveraging an automatic solver.

General classes of strategies for calculating the derivative of a function consist of:

• Finite difference method
• Symbolic differentiation
• Automatic differentiation

The SymPy Python library can be leveraged for symbolic differentiation.

Computational libraries like Theano and TensorFlow can be leveraged for automatic differentiation.

There are also online services you can leverage if your function is simple to mention in plain text.

One instance is the Wolfram Alpha website that will calculate the derivative of the function for you, for instance:

Calculate the derivative of x^2

Not every function is differentiable, and a few functions that are differentiable might make it tough to identify the derivative with some strategies.

Calculating the derivative of a function surpasses the scope of this guide.