### An intro to the method of Lagrange Multipliers

The method of Lagrange multipliers is a simple and sophisticated strategy of identifying the local minima or local maxima of a function subject to equality or inequality constraints. Lagrange multipliers are also referred to as undetermined multipliers. In this guide, we’ll speak about this method when provided equality constraints.

In this guide, you will find out about the method of Lagrange multipliers and how to identify the local minimum or maximum of a function when equality constraints exist.

After going through this guide, you will be aware of:

- How to identify points of local maximum or minimum of a function with equality constraints
- Method of Lagrange multipliers with equality constraints

**Tutorial Summarization**

This tutorial is subdivided into two portions, which are:

- Method of Lagrange multipliers with equality constraints
- Two solved instances

**Prerequisites**

For this guide, knowledge of the following are assumed:

- Derivative of functions
- Function of several variables, partial derivatives and gradient vectors
- An intro to optimization
- Gradient Descent

**What is the method of Lagrange Multipliers with Equality Constraints**

Let’s assume we possess the following optimization issue:

Minimize f(x)

Subject to:

g_1(x) = 0

g_2(x) = 0

g_n(x) = 0

The method of Lagrange multipliers first builds a function referred to as the Lagrange function as provided by the following expression:

L(x, **?**) = f(x) + ?_1 g_1(x) + ?_2 g_2(x) + … + ?_n g_n(x)

Here, **? **indicates a vector of Lagrange multipliers, i.e.,

**?**** **= [ ?_1, ?_2, …, ?_n]^T

To identify the points of local minimum of f(x) subjected to the equality constraints, we identify the stationary points of the Lagrange function L(x, **?**) that is, we solve the subsequent equations:

∇xL = 0

∂L/∂?_i = 0 (for i = 1..n)

Therefore, we obtain a total of m+n equations to solve, where:

m = number of variables in domain of f

n = number of equality constraints

In summary, the points of local minimum would be the answer of the following equations:

∂L/∂x_j = 0 (for j = 1..m)

g_i(x) = 0 (for i = 1..n)

**Solved Instances**

This section consists of two solved instances. If you solve both of them, you’ll obtain a very good idea on how to apply the strategy of Lagrange multipliers to functions of more than two variables, and a higher number of equality constraints.

**Example 1: One Equality Constraint**

Let’s find a solution to the following minimization issue:

Minimize: f(x) = x^2 + y^2

Subject to: x + 2y – 1 = 0

The first step is to build the Lagrange function:

L(x, y, ?) = x^2 + y^2 + ?(x + 2y – 1)

We have the following three equations to find solutions to:

∂L/∂x = 0

2x + ? = 0 (1)

∂L/∂y = 0

2y + 2? = 0 (2)

∂L/∂? = 0

x + 2y -1 = 0 (3)

Leveraging (1) and (2), we obtain:

? = -2x = -y

Plugging this in (3) provides us:

x = 1/5

y = 2/5

Therefore, the local minimum point lies at (1/5, 2/5) as displayed in the right figure. The left figure displays the graph of the function.

__Instance 2: Two Equality Constraints__

Assume we wish to identify the minimum of the following function subject to the provided constraints:

Minimize g(x, y) = x^2 + 4y^2

Subject to:

x + y = 0

x^2 + y^2 – 1 = 0

The solution of this problem can be identified by first building the Lagrange function:

L(x, y, ?_1, ?_2) = x^2 + 4y^2 + ?_1(x + y) + ?_2(x^2 + y^2 – 1)

We have four equations to find solutions to:

∂L/∂x = 0

2x + ?_1 + 2x ?_2 = 0 (1)

∂L/∂y = 0

8y + ?_1 + 2y ?_2 = 0 (2)

∂L/∂?_1 = 0

x + y = 0 (3)

∂L/∂?_2 = 0

x^2 + y^2 – 1 = 0 (4)

Solving the above system of equation provides us two solutions for (x,y), that is, we obtain the two points:

(1/sqrt(2), -1/sqrt(2))

(-1/sqrt(2), 1/sqrt(2))

The function combined with its constraints and local minimum are displayed below:

__Relationship to Maximization Problems__

If you possess a function to maximize, you can find a solution to it in a similar fashion, keeping in mind that maximization and minimization are equivalent issues i.e.,

maximize f(x) is equivalent to minimize -f(x)

__Criticality of the method of Lagrange Multipliers within Machine Learning__

Several popular machine learning algorithms leverage the method of Lagrange multipliers. For instance, the theoretical foundations of principal components analysis (PCA) are constructed leveraging the strategy of Lagrange multipliers with equality constraints. Likewise, the optimization issue in support vector machines SVMs is also solved leveraging this strategy. However, in SVMS, inequality constraints are additionally present.

__Extensions__

This part of the blog lists a few ideas for extension of this guide that you might desire to explore:

- Optimization with inequality constraints
- KKT conditions
- Support vector machines

__Further Reading__

This section furnishes additional resources on the subject if you are seeking to delve deeper.

*Concepts*

Derivatives

Gradient descent for machine learning

What is gradient in machine learning

Partial derivatives and gradient vectors

How to select an optimization algorithm

*Books*

Thomas Calculus, 14^{th} Edition, 2017 (based on the original works of George B. Thomas, revised by Joel Hass, Christopher Heil, Maurice Weir)

Calculus, 3^{rd} edition, 2017 (Gilbert Strang)

Calculus, 8^{th} edition, 2015 (James Stewart)

__Conclusion__

In this guide, you will find out about the method of Lagrange multipliers. Particularly, you learned:

- Lagrange multipliers and the Lagrange function
- How to solve an optimization problem when equality constraints are provided