Function optimisation is a basic sphere of research and study and the strategies are leveraged in nearly every quantitative domain. 

Critically, function optimisation is basic to nearly all machine learning algorithms, and predictive modelling projects. As such, it is crucial to comprehend what function optimisation is, the terminology leveraged in the domain, and the elements that make up a function optimization problem. 

In this guide, you will find out about function optimization. 

After going through this guide, you will be aware of: 

• The three elements of function optimization as candidate solutions, objective functions, and cost. 
• The conceptualization of function optimization as navigation of a search space and response surface. 
• The difference amongst global optima and local optima when finding a solution to a function optimization issue. 

Tutorial Summarization 

This guide is subdivided into four portions, which are: 

1] Function optimisation 

2] Candidate solutions 

3] Objective functions 

4] Evaluation costs
 

Function optimization 

Function optimization is a subfield of mathematics, and in modern times is tackled leveraging numerical computational strategies. 

Continuous function optimization, referred to as function optimization in short, belongs to a wider filed of research referred to as mathematical optimization. 

It is unique from other variants of optimisation as it consists of identifying optimal candidate solutions consisted of numeric input variables, in opposition to candidate solutions consisted of sequences or combinations (for example, combinatorial optimization) 

Function optimisation is a broadly leveraged tool bag of strategies deployed in practically all scientific and engineering disciplines.  

Individuals optimize. Investors look to develop portfolios that avoid excess risk while accomplishing a high rate of return. Optimization is a critical tool in decision science and in the analysis of physical systems. 

It has a central part within machine learning, as nearly all machine learning algorithms leverage function optimization to fit a model to a training dataset.  

For instance, fitting a line to a collection of points needs solving an optimization issue. As does fitting a linear regression or a neural network model on a training dataset. 

In this fashion, optimisation furnishes a tool to adapt a general model to a particular situation. Learning is viewed as an optimisation or search problem. 

Practically, function optimisation details a category of problems for identifying the input to a provided function that has the outcome in the minimum or maximum output from the function. 

The objective is dependent on specific characteristics of the system, referred to as variables or unknowns. Our objective is to identify values of the variables that optimise the objective. 

Function optimisation consists of three aspects: the input to the function (e.g. x), the objective function itself (e.g. f()) and the output from the function (e.g. cost) 

• Input(x): The input to the function to be assessed for instance, a candidate solution. 
• Function (f()): The objective function or target function that assesses inputs. 
• Cost: The outcome of assessing a candidate solution with the objective function, minimized or maximized. 

Candidate Solutions 

A Candidate Solution is a singular input to the objective function.  

The form of a candidate solution is dependent on the particulars of the objective function. It might be a singular floating point number, a vector of numbers, a matrix of numbers, or as complicated as required for the particular problem domain. 

Most typically, vectors of numbers. For a test problem, the vector indicates the particular values of every input variable to the function (x=x1, x2, x3 …, xn) For a machine learning model, the vector may indicate model coefficients or weights. 

From a mathematical standpoint, optimization is the minimization or maximization of a function subject to constraints on its variables. 

There might be constraints put forth by the problematic domain or the objective function on the candidate solutions. This might consist of aspects like: 

• The number of variables (1, 20, 1,000,000, etc.) 
• The data type of variables (integer, binary, real-valued, etc.) 
• The range of accepted values (between 0 and 1, etc.) 

Critically, candidate solutions are discrete and there are several of them. 

The universe of candidate solutions may be massive, too big to enumerate. Rather, the best we can do is sample candidate solutions within the search space. As a practitioner, we look for an optimisation algorithm that makes the best usage of the data available with regards to the problem to effectively sample the search space and locate a good or best candidate solution.  

Search space: Universe of candidate solutions defined by the number, type, and range of accepted inputs to the objective function. 

Lastly, candidate solutions can be classified by rank on the basis of their assessment by the objective function, implying that some are better than others. 

Objective Functions 

The objective function is particular to the problem domain. 

It might be a test function, for instance, a well-known equation with a particular number of input variables, the calculation of which gives back the cost of the input. The optima of test functions are known, enabling algorithms to be contrasted on the basis of their capability to navigate the search space efficiently. 

Within machine learning, the objective function may consist of plugging the candidate solution into a model and assessing it against a portion of the training dataset, and the cost might be an error score, often referred to as the loss of the model. 

The objective function is simple to define, although costly to assess. Efficiency in function optimization is a reference to minimization of the total number of function evaluations. 

Even though the objective function is simple to define, it might be challenging to go about optimizing. The difficulty of an objective function may range from being capable to analytically solve the function directly leveraging calculus or linear algebra, to leveraging a local search algorithm, to leveraging a global search algorithm. 

The difficulty of an objective function is on the basis of how much is knowledge with regards to the function. This usually cannot be determined through simple review of the equation or code for assessing candidate solutions. Rather, it refers to the structure of the response surface. 

The response surface (or search landscape) is the geometrical structure of the cost in relation to the search space of candidate solutions. For instance, a smooth response surface indicates that minimal changes to the input have the outcome of minimal changes to the output (cost) from the objective function. 

Response surface: Geometrical attributes of the cost from the objective function in reaction to changes to the candidate solutions. 

The response surface can be visualized in low dimensions, for example, for candidate solutions with one or two input variables. A one-dimensional input can be plotted as a 2D scatter plot with input values on the x-axis and the cost on the y-axis. A 2D input could be plotted as a 3D surface plot with input variables on the x and y-axis, and the height of the surface indicating the cost. 

In a minimization problem, weak solutions would be indicated as hills in the response surface and good solutions would be indicated by valleys. This would be inverted for maximization of problems. 

The structure and shape of this response surface determines the difficulty an algorithm will have in navigation of the search space to an answer. 

The intricacy of real objective functions implies we cannot undertake analysis of the surface analytically, and the high dimensionality of the inputs and the computational expenditure of function evaluations makes mapping and plotting real objective functions intractable. 

Evaluation costs 

The cost of a candidate solution is nearly always a singular real-value number.  

The scale of the cost values will demonstrate variance dependent on the particulars of the objective function. Generally, the only worthwhile comparison of cost values is to other cost values quantified by the same objective function. 

The minimum or maximum output from the function is referred to as the optima of the function, usually simplified to merely the minimum. Any function we desire to maximize can be converted to minimizing through including a negative sign to the front of the cost returned from the function. 

In global optimization, the true global solution of the optimization problem is identified, the compromise is efficiency. The worst-case intricacy of global optimization strategies grows exponentially with the problem sizes. 

An objective function may posses a singular ideal solution, usually called the global optimum of the objective function. Alternatively, the objective function might have several global optima, in which scenario we might be interested in situating one or all of them.  

Several numerical optimization techniques search for local minima. Local minima are locally optimal, but we are not generally aware whether a local minimum is a global minimum. 

On top of a global optima, a function may possess local optima, which are ideal candidates solutions that might be comparatively simple to locate, but not as good as the global optima. Local optima may seem to be global optima to a search algorithm for example, might be in a valley of the response surface, in which scenario we might refer to them as deceptive as the algorithm will simply locate them and get stuck, failing to locate the global optima. 

Global optima: The candidate solution with the best cost from the objective function.  

Local optima: Candidate solutions are good but not as good as the global optima. 

The relative nature of cost values implies that a baseline in performance on challenging issues can be setup leveraging a naïve search algorithm (for example, random) and “goodness” of optimal solutions discovered by more advanced search algorithms can be contrasted comparative to the baseline. 

Candidate solutions are usually really simple to detail and very simple to go about constructing. The challenging aspect of function optimization is assessing candidate solutions. 

Finding a solution to a function optimization issue or objective function refers to identifying the optima. The entire objective of the project is to situate a particular candidate solution with a good or best cost, provided the time and resources available. In simple and moderate issues, we might be able to situate the optimal candidate solution precisely and possess some confidence that we have done so. 

Several algorithms for nonlinear optimization issues look just for a local solution, a point at which the objective function is smaller than at all other feasible close by points. They don’t always identify the global solution, which is the point with lowest function value amongst all feasible points. Global solutions are required in a few applications, but for several problems, they are tough to recognize and even more tough to situate. 

Further Reading 

This section furnishes more resources on the subject if you’re looking for an in-depth dive. 

Books 

Algorithms for Optimisation, 2019 

Convex Optimisation, 2004 

Numerical Optimisation, 2006 

Articles 

Mathematical optimization, on Wikipedia. 

Conclusion 

In this guide, you found out about function optimization. 

Particularly, you learned: 

• The three aspects of function optimization as candidate solutions, objective functions and cost. 
• The conceptualization of function optimization as navigating a search space and response surface 
• The difference amongst global optima and local optima when finding a solution to a function optimization issue