Least squares

Least squares

Overview
The method of least squares is a standard approach to the approximate solution of overdetermined system
Overdetermined system
In mathematics, a system of linear equations is considered overdetermined if there are more equations than unknowns. The terminology can be described in terms of the concept of counting constraints. Each unknown can be seen as an available degree of freedom...

s, i.e., sets of equations in which there are more equations than unknowns. "Least squares" means that the overall solution minimizes the sum of the squares of the errors made in solving every single equation.

The most important application is in data fitting
Curve fitting
Curve fitting is the process of constructing a curve, or mathematical function, that has the best fit to a series of data points, possibly subject to constraints. Curve fitting can involve either interpolation, where an exact fit to the data is required, or smoothing, in which a "smooth" function...

. The best fit in the least-squares sense minimizes the sum of squared residuals
Errors and residuals in statistics
In statistics and optimization, statistical errors and residuals are two closely related and easily confused measures of the deviation of a sample from its "theoretical value"...

, a residual being the difference between an observed value and the fitted value provided by a model.
Discussion
Ask a question about 'Least squares'
Start a new discussion about 'Least squares'
Answer questions from other users
Full Discussion Forum
 
Unanswered Questions
Encyclopedia
The method of least squares is a standard approach to the approximate solution of overdetermined system
Overdetermined system
In mathematics, a system of linear equations is considered overdetermined if there are more equations than unknowns. The terminology can be described in terms of the concept of counting constraints. Each unknown can be seen as an available degree of freedom...

s, i.e., sets of equations in which there are more equations than unknowns. "Least squares" means that the overall solution minimizes the sum of the squares of the errors made in solving every single equation.

The most important application is in data fitting
Curve fitting
Curve fitting is the process of constructing a curve, or mathematical function, that has the best fit to a series of data points, possibly subject to constraints. Curve fitting can involve either interpolation, where an exact fit to the data is required, or smoothing, in which a "smooth" function...

. The best fit in the least-squares sense minimizes the sum of squared residuals
Errors and residuals in statistics
In statistics and optimization, statistical errors and residuals are two closely related and easily confused measures of the deviation of a sample from its "theoretical value"...

, a residual being the difference between an observed value and the fitted value provided by a model. When the problem has substantial uncertainties in the independent variable
Independent variable
The terms "dependent variable" and "independent variable" are used in similar but subtly different ways in mathematics and statistics as part of the standard terminology in those subjects...

 (the 'x' variable), then simple regression and least squares methods have problems; in such cases, the methodology required for fitting errors-in-variables models
Errors-in-variables models
In statistics and econometrics, errors-in-variables models or measurement errors models are regression models that account for measurement errors in the independent variables...

 may be considered instead of that for least squares.

Least squares problems fall into two categories: linear or ordinary least squares
Ordinary least squares
In statistics, ordinary least squares or linear least squares is a method for estimating the unknown parameters in a linear regression model. This method minimizes the sum of squared vertical distances between the observed responses in the dataset and the responses predicted by the linear...

 and non-linear least squares
Non-linear least squares
Non-linear least squares is the form of least squares analysis which is used to fit a set of m observations with a model that is non-linear in n unknown parameters . It is used in some forms of non-linear regression. The basis of the method is to approximate the model by a linear one and to...

, depending on whether or not the residuals are linear in all unknowns. The linear least-squares problem occurs in statistical regression analysis
Regression analysis
In statistics, regression analysis includes many techniques for modeling and analyzing several variables, when the focus is on the relationship between a dependent variable and one or more independent variables...

; it has a closed-form solution. A closed-form solution (or closed-form expression
Closed-form expression
In mathematics, an expression is said to be a closed-form expression if it can be expressed analytically in terms of a bounded number of certain "well-known" functions...

) is any formula that can be evaluated in a finite number of standard operations. The non-linear problem has no closed-form solution and is usually solved by iterative refinement; at each iteration the system is approximated by a linear one, thus the core calculation is similar in both cases.

The least-squares method was first described by Carl Friedrich Gauss
Carl Friedrich Gauss
Johann Carl Friedrich Gauss was a German mathematician and scientist who contributed significantly to many fields, including number theory, statistics, analysis, differential geometry, geodesy, geophysics, electrostatics, astronomy and optics.Sometimes referred to as the Princeps mathematicorum...

 around 1794. Least squares corresponds to the maximum likelihood
Maximum likelihood
In statistics, maximum-likelihood estimation is a method of estimating the parameters of a statistical model. When applied to a data set and given a statistical model, maximum-likelihood estimation provides estimates for the model's parameters....

 criterion if the experimental errors have a normal distribution and can also be derived as a method of moments estimator.

The following discussion is mostly presented in terms of linear
Linear
In mathematics, a linear map or function f is a function which satisfies the following two properties:* Additivity : f = f + f...

 functions but the use of least-squares is valid and practical for more general families of functions. For example, the Fourier series
Fourier series
In mathematics, a Fourier series decomposes periodic functions or periodic signals into the sum of a set of simple oscillating functions, namely sines and cosines...

 approximation of degree n is optimal in the least-squares sense, amongst all approximations in terms of trigonometric polynomial
Trigonometric polynomial
In the mathematical subfields of numerical analysis and mathematical analysis, a trigonometric polynomial is a finite linear combination of functions sin and cos with n a natural number. The coefficients may be taken as real numbers, for real-valued functions...

s of degree n. Also, by iteratively applying local quadratic approximation to the likelihood (through the Fisher information
Fisher information
In mathematical statistics and information theory, the Fisher information is the variance of the score. In Bayesian statistics, the asymptotic distribution of the posterior mode depends on the Fisher information and not on the prior...

), the least-squares method may be used to fit a generalized linear model
Generalized linear model
In statistics, the generalized linear model is a flexible generalization of ordinary linear regression. The GLM generalizes linear regression by allowing the linear model to be related to the response variable via a link function and by allowing the magnitude of the variance of each measurement to...

.


Context


The method of least squares grew out of the fields of astronomy
Astronomy
Astronomy is a natural science that deals with the study of celestial objects and phenomena that originate outside the atmosphere of Earth...

 and geodesy
Geodesy
Geodesy , also named geodetics, a branch of earth sciences, is the scientific discipline that deals with the measurement and representation of the Earth, including its gravitational field, in a three-dimensional time-varying space. Geodesists also study geodynamical phenomena such as crustal...

 as scientists and mathematicians sought to provide solutions to the challenges of navigating the Earth's oceans during the Age of Exploration. The accurate description of the behavior of celestial bodies was key to enabling ships to sail in open seas where before sailors had relied on land sightings to determine the positions of their ships.

The method was the culmination of several advances that took place during the course of the eighteenth century:
  • The combination of different observations taken under the same conditions contrary to simply trying one's best to observe and record a single observation accurately. This approach was notably used by Tobias Mayer
    Tobias Mayer
    Tobias Mayer was a German astronomer famous for his studies of the Moon.He was born at Marbach, in Württemberg, and brought up at Esslingen in poor circumstances. A self-taught mathematician, he had already published two original geometrical works when, in 1746, he entered J.B. Homann's...

     while studying the libration
    Libration
    In astronomy, libration is an oscillating motion of orbiting bodies relative to each other, notably including the motion of the Moon relative to Earth, or of Trojan asteroids relative to planets.-Lunar libration:...

    s of the moon.
  • The combination of different observations as being the best estimate of the true value; errors decrease with aggregation rather than increase, perhaps first expressed by Roger Cotes
    Roger Cotes
    Roger Cotes FRS was an English mathematician, known for working closely with Isaac Newton by proofreading the second edition of his famous book, the Principia, before publication. He also invented the quadrature formulas known as Newton–Cotes formulas and first introduced what is known today as...

    .
  • The combination of different observations taken under different conditions as notably performed by Roger Joseph Boscovich
    Roger Joseph Boscovich
    Ruđer Josip Bošković was a theologian, physicist, astronomer, mathematician, philosopher, diplomat, poet, Jesuit, and a polymath from the city of Dubrovnik in the Republic of Ragusa , who studied and lived in Italy and France where he also published many of his works.He is famous for...

     in his work on the shape of the earth and Pierre-Simon Laplace
    Pierre-Simon Laplace
    Pierre-Simon, marquis de Laplace was a French mathematician and astronomer whose work was pivotal to the development of mathematical astronomy and statistics. He summarized and extended the work of his predecessors in his five volume Mécanique Céleste...

     in his work in explaining the differences in motion of Jupiter
    Jupiter
    Jupiter is the fifth planet from the Sun and the largest planet within the Solar System. It is a gas giant with mass one-thousandth that of the Sun but is two and a half times the mass of all the other planets in our Solar System combined. Jupiter is classified as a gas giant along with Saturn,...

     and Saturn
    Saturn
    Saturn is the sixth planet from the Sun and the second largest planet in the Solar System, after Jupiter. Saturn is named after the Roman god Saturn, equated to the Greek Cronus , the Babylonian Ninurta and the Hindu Shani. Saturn's astronomical symbol represents the Roman god's sickle.Saturn,...

    .
  • The development of a criterion that can be evaluated to determine when the solution with the minimum error has been achieved, developed by Laplace in his Method of Least Squares.

The method


Carl Friedrich Gauss
Carl Friedrich Gauss
Johann Carl Friedrich Gauss was a German mathematician and scientist who contributed significantly to many fields, including number theory, statistics, analysis, differential geometry, geodesy, geophysics, electrostatics, astronomy and optics.Sometimes referred to as the Princeps mathematicorum...

 is credited with developing the fundamentals of the basis for least-squares analysis in 1795 at the age of eighteen. Legendre
Adrien-Marie Legendre
Adrien-Marie Legendre was a French mathematician.The Moon crater Legendre is named after him.- Life :...

 was the first to publish the method, however.

An early demonstration of the strength of Gauss's method came when it was used to predict the future location of the newly discovered asteroid Ceres. On January 1, 1801, the Italian astronomer Giuseppe Piazzi
Giuseppe Piazzi
Giuseppe Piazzi was an Italian Catholic priest of the Theatine order, mathematician, and astronomer. He was born in Ponte in Valtellina, and died in Naples. He established an observatory at Palermo, now the Osservatorio Astronomico di Palermo – Giuseppe S...

 discovered Ceres and was able to track its path for 40 days before it was lost in the glare of the sun. Based on this data, astronomers desired to determine the location of Ceres after it emerged from behind the sun without solving the complicated Kepler's nonlinear equations
Kepler's laws of planetary motion
In astronomy, Kepler's laws give a description of the motion of planets around the Sun.Kepler's laws are:#The orbit of every planet is an ellipse with the Sun at one of the two foci....

 of planetary motion. The only predictions that successfully allowed Hungarian astronomer Franz Xaver von Zach to relocate Ceres were those performed by the 24-year-old Gauss using least-squares analysis.

Gauss did not publish the method until 1809, when it appeared in volume two of his work on celestial mechanics, Theoria Motus Corporum Coelestium in sectionibus conicis solem ambientium.
In 1822, Gauss was able to state that the least-squares approach to regression analysis is optimal in the sense that in a linear model where the errors have a mean of zero, are uncorrelated, and have equal variances, the best linear unbiased estimator of the coefficients is the least-squares estimator. This result is known as the Gauss–Markov theorem.

The idea of least-squares analysis was also independently formulated by the Frenchman Adrien-Marie Legendre
Adrien-Marie Legendre
Adrien-Marie Legendre was a French mathematician.The Moon crater Legendre is named after him.- Life :...

 in 1805 and the American Robert Adrain
Robert Adrain
Robert Adrain was a scientist and mathematician, considered one of the most brilliant mathematical minds of the time in America....

 in 1808. In the next two centuries workers in the theory of errors and in statistics found many different ways of implementing least squares.

Problem statement


The objective consists of adjusting the parameters of a model function to best fit a data set. A simple data set consists of n points (data pairs) , i = 1, ..., n, where is an independent variable
Independent variable
The terms "dependent variable" and "independent variable" are used in similar but subtly different ways in mathematics and statistics as part of the standard terminology in those subjects...

 and is a dependent variable whose value is found by observation. The model function has the form , where the m adjustable parameters are held in the vector . The goal is to find the parameter values for the model which "best" fits the data. The least squares method finds its optimum when the sum, S, of squared residuals
is a minimum. A residual
Errors and residuals in statistics
In statistics and optimization, statistical errors and residuals are two closely related and easily confused measures of the deviation of a sample from its "theoretical value"...

 is defined as the difference between the actual value of the dependent variable and the value predicted by the model.
.

An example of a model is that of the straight line. Denoting the intercept as and the slope as , the model function is given by . See linear least squares for a fully worked out example of this model.

A data point may consist of more than one independent variable. For an example, when fitting a plane to a set of height measurements, the plane is a function of two independent variables, x and z, say. In the most general case there may be one or more independent variables and one or more dependent variables at each data point.

Limitations


This regression formulation considers only residuals in the dependent variable. There is an implicit assumption that errors in the independent variable
Independent variable
The terms "dependent variable" and "independent variable" are used in similar but subtly different ways in mathematics and statistics as part of the standard terminology in those subjects...

 are zero or strictly controlled so as to be negligible. When errors in the independent variable
Independent variable
The terms "dependent variable" and "independent variable" are used in similar but subtly different ways in mathematics and statistics as part of the standard terminology in those subjects...

 are non-negligible, models of measurement error can be used; such methods are more robust
Robust statistics
Robust statistics provides an alternative approach to classical statistical methods. The motivation is to produce estimators that are not unduly affected by small departures from model assumptions.- Introduction :...

 for parameter estimation than for hypothesis testing or for computing confidence interval
Confidence interval
In statistics, a confidence interval is a particular kind of interval estimate of a population parameter and is used to indicate the reliability of an estimate. It is an observed interval , in principle different from sample to sample, that frequently includes the parameter of interest, if the...

s.

Solving the least squares problem


The minimum
Maxima and minima
In mathematics, the maximum and minimum of a function, known collectively as extrema , are the largest and smallest value that the function takes at a point either within a given neighborhood or on the function domain in its entirety .More generally, the...

 of the sum of squares is found by setting the gradient
Gradient
In vector calculus, the gradient of a scalar field is a vector field that points in the direction of the greatest rate of increase of the scalar field, and whose magnitude is the greatest rate of change....

 to zero. Since the model contains m parameters there are m gradient equations.


and since the gradient equations become
.

The gradient equations apply to all least squares problems. Each particular problem requires particular expressions for the model and its partial derivatives.

Linear least squares


A regression model is a linear one when the model comprises a linear combination
Linear combination
In mathematics, a linear combination is an expression constructed from a set of terms by multiplying each term by a constant and adding the results...

 of the parameters, i.e.,


where the coefficients, , are functions of .

Letting


we can then see that in that case the least square estimate (or estimator, in the context of a random sample), is given by


For a derivation of this estimate see Linear least squares
Linear least squares
In statistics and mathematics, linear least squares is an approach to fitting a mathematical or statistical model to data in cases where the idealized value provided by the model for any data point is expressed linearly in terms of the unknown parameters of the model...

.

Functional analysis


A generalization to approximation of a data set is the approximation of a function by a sum of other functions, usually an orthogonal set
Orthogonal functions
In mathematics, two functions f and g are called orthogonal if their inner product \langle f,g\rangle is zero for f ≠ g. Whether or not two particular functions are orthogonal depends on how their inner product has been defined. A typical definition of an inner product for functions is...

:


with the set of functions {} an orthonormal set over the interval of interest, : see also Fejér's theorem
Fejér's theorem
In mathematics, Fejér's theorem, named for Hungarian mathematician Lipót Fejér, states that if f:R → C is a continuous function with period 2π, then the sequence of Cesàro means of the sequence of partial sums of the Fourier series of f converges uniformly to f on...

. The coefficients {} are selected to make the magnitude of the difference ||||2 as small as possible. For example, the magnitude, or norm, of a function over the can be defined by:


where the ‘*’ denotes complex conjugate in the case of complex functions. The extension of Pythagoras' theorem in this manner leads to function space
Function space
In mathematics, a function space is a set of functions of a given kind from a set X to a set Y. It is called a space because in many applications it is a topological space, a vector space, or both.-Examples:...

s and the notion of Lebesgue measure
Lebesgue measure
In measure theory, the Lebesgue measure, named after French mathematician Henri Lebesgue, is the standard way of assigning a measure to subsets of n-dimensional Euclidean space. For n = 1, 2, or 3, it coincides with the standard measure of length, area, or volume. In general, it is also called...

, an idea of “space” more general than the original basis of Euclidean geometry. The satisfy orthonormality relations:


where δij is the Kronecker delta. Substituting function into these equations then leads to
the n-dimensional Pythagorean theorem
Pythagorean theorem
In mathematics, the Pythagorean theorem or Pythagoras' theorem is a relation in Euclidean geometry among the three sides of a right triangle...

:


The coefficients {aj} making ||f − fn||2 as small as possible are found to be:


The generalization of the n-dimensional Pythagorean theorem to infinite-dimensional  real
Real number
In mathematics, a real number is a value that represents a quantity along a continuum, such as -5 , 4/3 , 8.6 , √2 and π...

 inner product spaces is known as Parseval's identity
Parseval's identity
In mathematical analysis, Parseval's identity is a fundamental result on the summability of the Fourier series of a function. Geometrically, it is thePythagorean theorem for inner-product spaces....

 or Parseval's equation. Particular examples of such a representation of a function are the Fourier series
Fourier series
In mathematics, a Fourier series decomposes periodic functions or periodic signals into the sum of a set of simple oscillating functions, namely sines and cosines...

 and the generalized Fourier series
Generalized Fourier series
In mathematical analysis, many generalizations of Fourier series have proved to be useful.They are all special cases of decompositions over an orthonormal basis of an inner product space....

.

Non-linear least squares


There is no closed-form solution to a non-linear least squares problem. Instead, numerical algorithms are used to find the value of the parameters which minimize the objective. Most algorithms involve choosing initial values for the parameters. Then, the parameters are refined iteratively, that is, the values are obtained by successive approximation.
k is an iteration number and the vector of increments, is known as the shift vector. In some commonly used algorithms, at each iteration the model may be linearized by approximation to a first-order Taylor series
Taylor series
In mathematics, a Taylor series is a representation of a function as an infinite sum of terms that are calculated from the values of the function's derivatives at a single point....

 expansion about


The Jacobian, J, is a function of constants, the independent variable and the parameters, so it changes from one iteration to the next. The residuals are given by
.

To minimize the sum of squares of , the gradient equation is set to zero and solved for


which, on rearrangement, become m simultaneous linear equations, the normal equations.


The normal equations are written in matrix notation as

These are the defining equations of the Gauss–Newton algorithm.

Differences between linear and non-linear least squares

  • The model function, f, in LLSQ (linear least squares) is a linear combination of parameters of the form The model may represent a straight line, a parabola or any other linear combination of functions. In NLLSQ (non-linear least squares) the parameters appear as functions, such as and so forth. If the derivatives are either constant or depend only on the values of the independent variable, the model is linear in the parameters. Otherwise the model is non-linear.
  • Algorithms for finding the solution to a NLLSQ problem require initial values for the parameters, LLSQ does not.
  • Like LLSQ, solution algorithms for NLLSQ often require that the Jacobian be calculated. Analytical expressions for the partial derivatives can be complicated. If analytical expressions are impossible to obtain either the partial derivatives must be calculated by numerical approximation or an estimate must be made of the Jacobian.
  • In NLLSQ non-convergence (failure of the algorithm to find a minimum) is a common phenomenon whereas the LLSQ is globally concave so non-convergence is not an issue.
  • NLLSQ is usually an iterative process. The iterative process has to be terminated when a convergence criterion is satisfied. LLSQ solutions can be computed using direct methods, although problems with large numbers of parameters are typically solved with iterative methods, such as the Gauss–Seidel method.
  • In LLSQ the solution is unique, but in NLLSQ there may be multiple minima in the sum of squares.
  • Under the condition that the errors are uncorrelated with the predictor variables, LLSQ yields unbiased estimates, but even under that condition NLLSQ estimates are generally biased.

These differences must be considered whenever the solution to a non-linear least squares problem is being sought.

Least squares, regression analysis and statistics


The methods of least squares and regression analysis
Regression analysis
In statistics, regression analysis includes many techniques for modeling and analyzing several variables, when the focus is on the relationship between a dependent variable and one or more independent variables...

 are conceptually different. However, the method of least squares is often used to generate estimators and other statistics in regression analysis.

Consider a simple example drawn from physics. A spring should obey Hooke's law
Hooke's law
In mechanics, and physics, Hooke's law of elasticity is an approximation that states that the extension of a spring is in direct proportion with the load applied to it. Many materials obey this law as long as the load does not exceed the material's elastic limit. Materials for which Hooke's law...

 which states that the extension of a spring is proportional to the force, F, applied to it.
constitutes the model, where F is the independent variable. To estimate the force constant, k, a series of n measurements with different forces will produce a set of data, , where yi is a measured spring extension. Each experimental observation will contain some error. If we denote this error , we may specify an empirical model for our observations,


There are many methods we might use to estimate the unknown parameter k. Noting that the n equations in the m variables in our data comprise an overdetermined system
Overdetermined system
In mathematics, a system of linear equations is considered overdetermined if there are more equations than unknowns. The terminology can be described in terms of the concept of counting constraints. Each unknown can be seen as an available degree of freedom...

 with one unknown and n equations, we may choose to estimate k using least squares. The sum of squares to be minimized is


The least squares estimate of the force constant, k, is given by


Here it is assumed that application of the force causes the spring to expand and, having derived the force constant by least squares fitting, the extension can be predicted from Hooke's law.

In regression analysis the researcher specifies an empirical model. For example, a very common model is the straight line model which is used to test if there is a linear relationship between dependent and independent variable. If a linear relationship is found to exist, the variables are said to be correlated. However, correlation does not prove causation
Correlation does not imply causation
"Correlation does not imply causation" is a phrase used in science and statistics to emphasize that correlation between two variables does not automatically imply that one causes the other "Correlation does not imply causation" (related to "ignoring a common cause" and questionable cause) is a...

, as both variables may be correlated with other, hidden, variables, or the dependent variable may "reverse" cause the independent variables, or the variables may be otherwise spuriously correlated. For example, suppose there is a correlation between deaths by drowning and the volume of ice cream sales at a particular beach. Yet, both the number of people going swimming and the volume of ice cream sales increase as the weather gets hotter, and presumably the number of deaths by drowning is correlated with the number of people going swimming. Perhaps an increase in swimmers causes both the other variables to increase.

In order to make statistical tests on the results it is necessary to make assumptions about the nature of the experimental errors. A common (but not necessary) assumption is that the errors belong to a Normal distribution. The central limit theorem
Central limit theorem
In probability theory, the central limit theorem states conditions under which the mean of a sufficiently large number of independent random variables, each with finite mean and variance, will be approximately normally distributed. The central limit theorem has a number of variants. In its common...

 supports the idea that this is a good approximation in many cases.
  • The Gauss–Markov theorem. In a linear model in which the errors have expectation
    Expected value
    In probability theory, the expected value of a random variable is the weighted average of all possible values that this random variable can take on...

     zero conditional on the independent variables, are uncorrelated
    Uncorrelated
    In probability theory and statistics, two real-valued random variables are said to be uncorrelated if their covariance is zero. Uncorrelatedness is by definition pairwise; i.e...

     and have equal variance
    Variance
    In probability theory and statistics, the variance is a measure of how far a set of numbers is spread out. It is one of several descriptors of a probability distribution, describing how far the numbers lie from the mean . In particular, the variance is one of the moments of a distribution...

    s, the best linear unbiased estimator of any linear combination of the observations, is its least-squares estimator. "Best" means that the least squares estimators of the parameters have minimum variance. The assumption of equal variance is valid when the errors all belong to the same distribution.
  • In a linear model, if the errors belong to a Normal distribution the least squares estimators are also the maximum likelihood estimators.


However, if the errors are not normally distributed, a central limit theorem
Central limit theorem
In probability theory, the central limit theorem states conditions under which the mean of a sufficiently large number of independent random variables, each with finite mean and variance, will be approximately normally distributed. The central limit theorem has a number of variants. In its common...

 often nonetheless implies that the parameter estimates will be approximately normally distributed so long as the sample is reasonably large. For this reason, given the important property that the error mean is independent of the independent variables, the distribution of the error term is not an important issue in regression analysis. Specifically, it is not typically important whether the error term follows a normal distribution.

In a least squares calculation with unit weights, or in linear regression, the variance on the jth parameter,
denoted , is usually estimated with

where the true residual variance σ2 is replaced by an estimate based on the minimised value of the sum of squares objective function S. The denominator, n-m, is the statistical degrees of freedom
Degrees of freedom (statistics)
In statistics, the number of degrees of freedom is the number of values in the final calculation of a statistic that are free to vary.Estimates of statistical parameters can be based upon different amounts of information or data. The number of independent pieces of information that go into the...

; see effective degrees of freedom for generalizations.

Confidence limits can be found if the probability distribution
Probability distribution
In probability theory, a probability mass, probability density, or probability distribution is a function that describes the probability of a random variable taking certain values....

 of the parameters is known, or an asymptotic approximation is made, or assumed. Likewise statistical tests on the residuals can be made if the probability distribution of the residuals is known or assumed. The probability distribution of any linear combination of the dependent variables can be derived if the probability distribution of experimental errors is known or assumed. Inference is particularly straightforward if the errors are assumed to follow a normal distribution, which implies that the parameter estimates and residuals will also be normally distributed conditional on the values of the independent variables.

Weighted least squares


The expressions given above are based on the implicit assumption that the errors are uncorrelated with each other and with the independent variables and have equal variance. The Gauss–Markov theorem shows that, when this is so, is a best linear unbiased estimator (BLUE). If, however, the measurements are uncorrelated but have different uncertainties, a modified approach might be adopted. Aitken
Alexander Aitken
Alexander Craig Aitken was one of New Zealand's greatest mathematicians. He studied for a PhD at the University of Edinburgh, where his dissertation, "Smoothing of Data", was considered so impressive that he was awarded a DSc in 1926, and was elected a fellow of the Royal Society of Edinburgh...

 showed that when a weighted sum of squared residuals is minimized, is BLUE if each weight is equal to the reciprocal of the variance of the measurement.
The gradient equations for this sum of squares are


which, in a linear least squares system give the modified normal equations,


When the observational errors are uncorrelated and the weight matrix, W, is diagonal, these may be written as


If the errors are correlated, the resulting estimator is BLUE if the weight matrix is equal to the inverse of the variance-covariance matrix of the observations.

When the errors are uncorrelated, it is convenient to simplify the calculations to factor the weight matrix as . The normal equations can then be written as


where


For non-linear least squares systems a similar argument shows that the normal equations should be modified as follows.


Note that for empirical tests, the appropriate W is not known for sure and must be
estimated. For this Feasible Generalized Least Squares (FGLS) techniques may be used.

Relationship to principal components


The first principal component about the mean of a set of points can be represented by that line which most closely approaches the data points (as measured by squared distance of closest approach, i.e. perpendicular to the line). In contrast, linear least squares tries to minimize the distance in the direction only. Thus, although the two use a similar error metric, linear least squares is a method that treats one dimension of the data preferentially, while PCA treats all dimensions equally.

LASSO method


In some contexts a regularized
Regularization (machine learning)
In statistics and machine learning, regularization is any method of preventing overfitting of data by a model. It is used for solving ill-conditioned parameter-estimation problems...

 version of the least squares solution may be preferable. The LASSO (least absolute shrinkage and selection operator) algorithm, for example, finds a least-squares solution with the constraint that , the L1-norm of the parameter vector, is no greater than a given value. Equivalently, it may solve an unconstrained minimization of the least-squares penalty with added, where is a constant (this is the Lagrangian
Lagrange multipliers
In mathematical optimization, the method of Lagrange multipliers provides a strategy for finding the maxima and minima of a function subject to constraints.For instance , consider the optimization problem...

 form of the constrained problem.) This problem may be solved using quadratic programming
Quadratic programming
Quadratic programming is a special type of mathematical optimization problem. It is the problem of optimizing a quadratic function of several variables subject to linear constraints on these variables....

 or more general convex optimization methods, as well as by specific algorithms such as the least angle regression algorithm. The L1-regularized formulation is useful in some contexts due to its tendency to prefer solutions with fewer nonzero parameter values, effectively reducing the number of variables upon which the given solution is dependent. For this reason, the LASSO and its variants are fundamental to the field of compressed sensing
Compressed sensing
Compressed sensing, also known as compressive sensing, compressive sampling and sparse sampling, is a technique for finding sparse solutions to underdetermined linear systems...

.

See also

  • Best linear unbiased prediction
    Best linear unbiased prediction
    In statistics, best linear unbiased prediction is used in linear mixed models for the estimation of random effects. BLUP was derived by Charles Roy Henderson in 1950 but the term "best linear unbiased predictor" seems not to have been used until 1962...

     (BLUP)
  • L2 norm
  • Least absolute deviation
  • Measurement uncertainty
    Measurement uncertainty
    In metrology, measurement uncertainty is a non-negative parameter characterizing the dispersion of the values attributed to a measured quantity. The uncertainty has a probabilistic basis and reflects incomplete knowledge of the quantity. All measurements are subject to uncertainty and a measured...

  • Root mean square
    Root mean square
    In mathematics, the root mean square , also known as the quadratic mean, is a statistical measure of the magnitude of a varying quantity. It is especially useful when variates are positive and negative, e.g., sinusoids...

  • Squared deviations
    Squared deviations
    In probability theory and statistics, the definition of variance is either the expected value , or average value , of squared deviations from the mean. Computations for analysis of variance involve the partitioning of a sum of squared deviations...

  • Quadratic loss function

External links