Optimization Theory Overview

Optimization techniques are used to find a set of design parameters, x = {x₁,x₂,...,x_n}, that can in some way be defined as optimal. In a simple case, this process might be the minimization or maximization of some system characteristic that is dependent on x. In a more advanced formulation, the objective function f(x), to be minimized or maximized, might be subject to constraints in one or more of these forms:

• Equality constraints, G_i(x) = 0 ( i = 1,...,m_e)

• Inequality constraints, G_i( x) ≤ 0 (i = m_e + 1,...,m)

• Parameter bounds, x_l, x_u, where x_l ≤ x ≤ x_u, some x_l can be –∞, and some x_u can be ∞

A General Problem (GP) description is stated as

$$\underset{x}{\min }f(x),$$

(1)

subject to

$$\begin{array}{l}\begin{array}{cc}{G}_i(x)=0& i=1,\mathrm{...},m_e\\ G_i(x)\le 0& i=m_e+1,\mathrm{...},m\end{array}\\ x_l\le x\le x_u,\end{array}$$

where x is the vector of length n design parameters, f(x) is the objective function (which returns a scalar value), and the vector function G(x) returns a vector of length m containing the values of the equality and inequality constraints evaluated at x.

Optimization Toolbox™ solvers apply to smooth problems. This means the objective function and any nonlinear constraint functions should be at least twice continuously differentiable. Sometimes, Optimization Toolbox solvers work with nonsmooth functions; feel free to try them on any problem. The base MATLAB^® optimization solvers fminbnd, fminsearch, and fzero have no smoothness requirements, and most Global Optimization Toolbox solvers have no smoothness requirements.

An efficient and accurate solution to this problem depends not only on the size of the problem in terms of the number of constraints and design variables, but also on characteristics of the objective function and constraints. When both the objective function and the constraints are linear functions of the design variable, the problem is known as a Linear Programming (LP) problem. Quadratic Programming (QP) concerns the minimization or maximization of a quadratic objective function that is linearly constrained. For both the LP and QP problems, reliable solution procedures are readily available. More difficult to solve is the Nonlinear Programming (NP) problem in which the objective function and constraints can be nonlinear functions of the design variables. A solution of the NP problem generally requires an iterative procedure to establish a direction of search at each major iteration. This solution is usually achieved by the solution of an LP, QP, or unconstrained subproblem.

All optimization takes place in real numbers. However, unconstrained least-squares problems and equation solving can be formulated and solved using complex analytic functions. See Complex Numbers in Optimization Toolbox Solvers.