Variants of the Least-Squares Problem
Linearly constrained least-squaresDefinitionAn interesting variant of the ordinary least-squares problem involves equality constraints on the decision variable ![]() where Examples: SolutionWe can express the solution by first computing the nullspace of ![]() where Expressing ![]() where Minimum-norm solution to linear equationsA special case of linearly constrained LS is ![]() in which we implicitly assume that the linear equation in The above problem allows to select a particular solution to a linear equation, in the case when there are possibly many, that is, the linear system As seen here, when ![]() Examples: Control positioning of a mass. Regularized least-squaresIn the case when the matrix The regularized least-squares problem has the form ![]() where The regularized problem can be expressed as an ordinary least-squares problem, where the data matrix is full column rank. Indeed, the above problem can be written as the ordinary LS problem ![]() where ![]() The presence of the identity matrix in the SolutionSince the data matrix in the regularized LS problem has full column rank, the formula seen here applies. The solution is unique, and given by ![]() For The above formula explains one of the motivations for using regularized least-squares in the case of a rank-deficient matrix Weighted regularized least-squaresSometimes, as in kernel methods, we are led to problems of the form ![]() where ![]() |