Partial least squares regression
Encyclopedia
Partial least squares regression (PLS regression) is a statistical
method that bears some relation to principal components regression
; instead of finding hyperplane
s of maximum variance
between the response and independent variables, it finds a linear regression
model by projecting the predicted variables and the observable variable
s to a new space. Because both the X and Y data are projected to new spaces, the PLS family of methods are known as bilinear factor models. Partial least squares Discriminant Analysis (PLS-DA) is a variant used when the Y is binary.
PLS is used to find the fundamental relations between two matrices
(X and Y), i.e. a latent variable
approach to modeling the covariance
structures in these two spaces. A PLS model will try to find the multidimensional direction in the X space that explains the maximum multidimensional variance direction in the Y space. PLS regression is particularly suited when the matrix of predictors has more variables than observations, and when there is multicollinearity
among X values. By contrast, standard regression will fail in these cases.
The PLS algorithm is employed in PLS path modelling, a method of modeling a causal network of latent variable
s. This technique is a form of structural equation modeling
, distinguished from the classical method by being component-based rather than covariance-based.
Partial least squares was introduced by the Swedish statistician Herman Wold
, who then developed it with his son, Svante Wold, a professor of chemometrics
at Umeå University
. An alternative term for PLS (and more correct according to Svante Wold) is projection to latent structures, but the term partial least squares is still dominant in many areas. Although the original applications were in the social sciences, PLS regression is today most widely used in chemometrics
and related areas. It is also used in bioinformatics, sensometrics, neuroscience and anthropology. In contrast, PLS path modeling is most often used in social sciences, econometrics, marketing and strategic management.
Statistics
Statistics is the study of the collection, organization, analysis, and interpretation of data. It deals with all aspects of this, including the planning of data collection in terms of the design of surveys and experiments....
method that bears some relation to principal components regression
Principal component regression
In statistics, principal component regression is a regression analysis that uses principal component analysis when estimating regression coefficients...
; instead of finding hyperplane
Hyperplane
A hyperplane is a concept in geometry. It is a generalization of the plane into a different number of dimensions.A hyperplane of an n-dimensional space is a flat subset with dimension n − 1...
s of maximum variance
Variance
In probability theory and statistics, the variance is a measure of how far a set of numbers is spread out. It is one of several descriptors of a probability distribution, describing how far the numbers lie from the mean . In particular, the variance is one of the moments of a distribution...
between the response and independent variables, it finds a linear regression
Linear regression
In statistics, linear regression is an approach to modeling the relationship between a scalar variable y and one or more explanatory variables denoted X. The case of one explanatory variable is called simple regression...
model by projecting the predicted variables and the observable variable
Observable variable
In statistics, observable variables or manifest variables, as opposed to latent variables, are those variables that can be observed and directly measured.- See also :* Observables in physics* Observability in control theory* Latent variable model...
s to a new space. Because both the X and Y data are projected to new spaces, the PLS family of methods are known as bilinear factor models. Partial least squares Discriminant Analysis (PLS-DA) is a variant used when the Y is binary.
PLS is used to find the fundamental relations between two matrices
Matrix (mathematics)
In mathematics, a matrix is a rectangular array of numbers, symbols, or expressions. The individual items in a matrix are called its elements or entries. An example of a matrix with six elements isMatrices of the same size can be added or subtracted element by element...
(X and Y), i.e. a latent variable
Latent variable
In statistics, latent variables , are variables that are not directly observed but are rather inferred from other variables that are observed . Mathematical models that aim to explain observed variables in terms of latent variables are called latent variable models...
approach to modeling the covariance
Covariance
In probability theory and statistics, covariance is a measure of how much two variables change together. Variance is a special case of the covariance when the two variables are identical.- Definition :...
structures in these two spaces. A PLS model will try to find the multidimensional direction in the X space that explains the maximum multidimensional variance direction in the Y space. PLS regression is particularly suited when the matrix of predictors has more variables than observations, and when there is multicollinearity
Multicollinearity
Multicollinearity is a statistical phenomenon in which two or more predictor variables in a multiple regression model are highly correlated. In this situation the coefficient estimates may change erratically in response to small changes in the model or the data...
among X values. By contrast, standard regression will fail in these cases.
The PLS algorithm is employed in PLS path modelling, a method of modeling a causal network of latent variable
Latent variable
In statistics, latent variables , are variables that are not directly observed but are rather inferred from other variables that are observed . Mathematical models that aim to explain observed variables in terms of latent variables are called latent variable models...
s. This technique is a form of structural equation modeling
Structural equation modeling
Structural equation modeling is a statistical technique for testing and estimating causal relations using a combination of statistical data and qualitative causal assumptions...
, distinguished from the classical method by being component-based rather than covariance-based.
Partial least squares was introduced by the Swedish statistician Herman Wold
Herman Wold
Herman Ole Andreas Wold was a Norwegian-born econometrician and statistician who had a long career in Sweden...
, who then developed it with his son, Svante Wold, a professor of chemometrics
Chemometrics
Chemometrics is the science of extracting information from chemical systems by data-driven means. It is a highly interfacial discipline, using methods frequently employed in core data-analytic disciplines such as multivariate statistics, applied mathematics, and computer science, in order to...
at Umeå University
Umeå University
Umeå University is a university in Umeå in the mid-northern region of Sweden. The university was founded in 1965 and is the fifth oldest within Sweden's present borders....
. An alternative term for PLS (and more correct according to Svante Wold) is projection to latent structures, but the term partial least squares is still dominant in many areas. Although the original applications were in the social sciences, PLS regression is today most widely used in chemometrics
Chemometrics
Chemometrics is the science of extracting information from chemical systems by data-driven means. It is a highly interfacial discipline, using methods frequently employed in core data-analytic disciplines such as multivariate statistics, applied mathematics, and computer science, in order to...
and related areas. It is also used in bioinformatics, sensometrics, neuroscience and anthropology. In contrast, PLS path modeling is most often used in social sciences, econometrics, marketing and strategic management.
Underlying model
The general underlying model of multivariate PLS is-
where
is an 
matrix of predictors, 
is an 
matrix of responses, 
is an 
matrix (the score, component or factor matrix), 
and 
are, respectively, 
and 
loading matrices, and matrices 
and 
are the error terms, assumed to be i.i.d. normal.
Algorithms
A number of variants of PLS exist for estimating the factor and loading matrices
and 
. Most of them construct estimates of the linear regression between 
and 
as 
. Some PLS algorithms are only appropriate for the case where 
is a column vector, while others deal with the general case of a matrix 
. Algorithms also differ on whether they estimate the factor matrix 
as an orthogonal, an orthonormal matrix or not.
The final prediction will be the same for all these varieties of PLS, but the components will differ.
PLS1
PLS1 is a widely used algorithm appropriate for the vector
case. It estimates 
as an orthonormal matrix. In pseudocode it may be expressed as:
1 function PLS1(
)
2
3
, an initial estimate of 
.
4
5 for
= 0 to 
6
7
8
9
10 if
= 0
11
, break the for loop
12 if
13
14
15
16 end for
17 define
to be the matrix with columns 
. Similarly define 
18
19
20 return
This form of the algorithm does not require centering of the input
and 
, as this is performed implicitly by the algorithm.
This algorithm features 'deflation' of the matrix
(subtraction of 
), but deflation of the vector 
is not performed, as it is not necessary (it can be proved that deflating 
yields the same results as not deflating.). The user-supplied variable 
is the limit on the number of latent factors in the regression; if it equals the rank of the matrix 
, the algorithm will yield the least squares regression estimates for 
and 
Extensions
In 2002 a new preprocessing method was published called orthogonal projections to latent structures (OPLS). In OPLS systematic variation in the X matrix not correlated to Y is removed. This improves the interpretability (but not the predictivity) of the PLS models. L-PLS extends PLS regression to 3 connected data blocks.
Software implementation
Most major statistical software packages offer PLS regression, including SASSAS- Special forces :* Special Air Service, a special forces unit of the British Army* Australian Special Air Service Regiment * New Zealand Special Air Service * Rhodesian Special Air Service...
, JMPJMP (statistical software)JMP is a computer program that was first developed by John Sall and others to perform simple and complex statistical analyses.It dynamically links statistics with graphics to interactively explore, understand, and visualize data...
, SPSSSPSSSPSS is a computer program used for survey authoring and deployment , data mining , text analytics, statistical analysis, and collaboration and deployment ....
and StatisticaSTATISTICASTATISTICA is a statistics and analytics software package developed by StatSoft. STATISTICA provides data analysis, data management, data mining, and data visualization procedures...
. The following PLS path modeling software tools implement PLS regression or variations of it: PLS-Graph, SmartPLS, and WarpPLS. There is a PLS regression module in XLSTATXLSTATXLSTAT is a commercial statistical and multivariate analysis software. The software has been developed by Addinsoft and was introduced by Thierry Fahmy, the founder of Addinsoft, in 1993. It is a Microsoft Excel add-in...
, an add-in for Microsoft ExcelMicrosoft ExcelMicrosoft Excel is a proprietary commercial spreadsheet application written and distributed by Microsoft for Microsoft Windows and Mac OS X. It features calculation, graphing tools, pivot tables, and a macro programming language called Visual Basic for Applications...
. SIMCA-P+ from Umetrics and The UnscramblerThe UnscramblerThe Unscrambler is a commercial software product for multivariate data analysis, used primarily for calibration in the application of near infrared spectroscopy and development of predictive models for use in real-time spectroscopic analysis of materials. The software was originally developed in...
are popular amongst chemometricians and sensometricians. PLS regression routines are available for MATLABMATLABMATLAB is a numerical computing environment and fourth-generation programming language. Developed by MathWorks, MATLAB allows matrix manipulations, plotting of functions and data, implementation of algorithms, creation of user interfaces, and interfacing with programs written in other languages,...
and the popular open source language RR (programming language)R is a programming language and software environment for statistical computing and graphics. The R language is widely used among statisticians for developing statistical software, and R is widely used for statistical software development and data analysis....
.
Further reading
- R. Kramer, Chemometric Techniques for Quantitative Analysis, (1998) Marcel-Dekker, ISBN 0-8247-0198-4.
- Helland Inge S. (1990). PLS regression and statisticalmodels. Scandivian Journal of Statistics, 17, 97–114.
- Wold, HermanHerman WoldHerman Ole Andreas Wold was a Norwegian-born econometrician and statistician who had a long career in Sweden...
. (1966). Estimation of principal components and related models by iterative least squares. In P.R. Krishnaiaah (Ed.). Multivariate Analysis. (pp. 391–420) New York: Academic Press. - Wold, HermanHerman WoldHerman Ole Andreas Wold was a Norwegian-born econometrician and statistician who had a long career in Sweden...
. (1981). The fix-point approach to interdependent systems. Amsterdam: North Holland. - Wold, HermanHerman WoldHerman Ole Andreas Wold was a Norwegian-born econometrician and statistician who had a long career in Sweden...
. (1985). Partial least squares, pp. 581–591 in Samuel Kotz and Norman L. Johnson, eds., Encyclopedia of statistical sciences, Vol. 6, New York: Wiley, 1985. - Svante Wold, Axel Ruhe, Herman WoldHerman WoldHerman Ole Andreas Wold was a Norwegian-born econometrician and statistician who had a long career in Sweden...
, and W.J. Dunn. "The collinearity problem in linear regression. the partial least squares (PLS) approach to generalized inverses." SIAM J. Sci. Stat. Comp., 5:735-743, 1984. - Garthwaite, Paul H. (1994) "An Interpretation of Partial Least Squares", Journal of the American Statistical AssociationJournal of the American Statistical AssociationThe Journal of the American Statistical Association is the most prestigious journal published by the American Statistical Association, the main professional body for statisticians in the United States...
, 89, 122–127 - Vinzi, V., Chin, W.W., Henseler, J., Wang, H. (eds) (2010). Handbook of Partial Least Squares. ISBN 978-3-540-32825-4
- Stone, M. and Brooks, R. J. (1990) "Continuum Regression: Cross-Validated Sequentially Constructed Prediction embracing Ordinary Least Squares, Partial Least Squares and Principal Components Regression", Journal of the Royal Statistical SocietyJournal of the Royal Statistical SocietyThe Journal of the Royal Statistical Society is a series of three peer-reviewed statistics journals published by Blackwell Publishing for the London-based Royal Statistical Society.- History :...
, Series B, 52 (2), 237–269
External links
- Algorithm description
- PLS and regression tutorial
- PLS in Brain Imaging
- on-line PLS regression (PLSR) at Virtual Computational Chemistry Laboratory
- Uncertainty estimation for PLS
- A short introduction to PLS regression and its history