Factor analysis is a
statisticalStatistics is a branch of mathematics concerned with collecting and interpreting data. According to other definitions, it is a mathematical science pertaining to the collection, analysis, interpretation or explanation, and presentation of data. Statisticians improve the quality of data with the...
method used to describe
variabilityIn probability theory and statistics, the variance of a random variable or distribution is the expected square deviation of that variable from its expected value or mean, or to put it another way: variance is the measure of the amount of variation of all the scores for a variable...
among observed
variablesA variable is a symbol that stands for a value that may vary; the term usually occurs in opposition to constant, which is a symbol for a non-varying value, i.e. completely fixed or fixed in the context of use...
in terms of fewer unobserved variables called
factors. The observed variables are modeled as
linear combinationIn mathematics, linear combinations are a concept central to linear algebra and related fields of mathematics.Most of this article deals with linear combinations in the context of a vector space over a field, with some generalizations given at the end of the article.- Definition:Suppose that K is a...
s of the factors, plus "
errorIn statistics and optimization, statistical errors and residuals are two closely related and easily confused measures of "deviation of a sample from the mean": the error of a sample is the deviation of the sample from the population mean or actual function, while the residual of a sample is the...
" terms. The information gained about the interdependencies can be used later to reduce the set of variables in a dataset. Factor analysis originated in
psychometricsPsychometrics is the field of study concerned with the theory and technique of educational and psychological measurement, which includes the measurement of knowledge, abilities, attitudes, and personality traits. The field is primarily concerned with the study of measurement instruments such as...
, and is used in behavioral sciences,
social sciencesThe social sciences are the fields of scientific knowledge and academic scholarship that study social groups and, more generally, human society. The social sciences initially were constituted of five fields: Jurisprudence and Amendment of the Law; Education; Health; Economy and Trade; Art...
,
marketingMarketing is an integrated communications-based process through which individuals and communities are informed or persuaded that existing and newly-identified needs and wants may be satisfied by the products and services of others....
,
product managementProduct management is an organizational lifecycle function within a company dealing with the planning or marketing of a product or products at all stages of the product lifecycle....
,
operations researchOperations research or Quantitative management, as termed in the USA, Canada, South Africa and Australia, and operational research, as termed in Europe, is an interdisciplinary branch of applied mathematics that uses methods such as mathematical modeling, statistics, and algorithms to arrive at...
, and other applied sciences that deal with large quantities of data.
Factor analysis is related to principal component analysis (PCA) but not identical.
Factor analysis is a
statisticalStatistics is a branch of mathematics concerned with collecting and interpreting data. According to other definitions, it is a mathematical science pertaining to the collection, analysis, interpretation or explanation, and presentation of data. Statisticians improve the quality of data with the...
method used to describe
variabilityIn probability theory and statistics, the variance of a random variable or distribution is the expected square deviation of that variable from its expected value or mean, or to put it another way: variance is the measure of the amount of variation of all the scores for a variable...
among observed
variablesA variable is a symbol that stands for a value that may vary; the term usually occurs in opposition to constant, which is a symbol for a non-varying value, i.e. completely fixed or fixed in the context of use...
in terms of fewer unobserved variables called
factors. The observed variables are modeled as
linear combinationIn mathematics, linear combinations are a concept central to linear algebra and related fields of mathematics.Most of this article deals with linear combinations in the context of a vector space over a field, with some generalizations given at the end of the article.- Definition:Suppose that K is a...
s of the factors, plus "
errorIn statistics and optimization, statistical errors and residuals are two closely related and easily confused measures of "deviation of a sample from the mean": the error of a sample is the deviation of the sample from the population mean or actual function, while the residual of a sample is the...
" terms. The information gained about the interdependencies can be used later to reduce the set of variables in a dataset. Factor analysis originated in
psychometricsPsychometrics is the field of study concerned with the theory and technique of educational and psychological measurement, which includes the measurement of knowledge, abilities, attitudes, and personality traits. The field is primarily concerned with the study of measurement instruments such as...
, and is used in behavioral sciences,
social sciencesThe social sciences are the fields of scientific knowledge and academic scholarship that study social groups and, more generally, human society. The social sciences initially were constituted of five fields: Jurisprudence and Amendment of the Law; Education; Health; Economy and Trade; Art...
,
marketingMarketing is an integrated communications-based process through which individuals and communities are informed or persuaded that existing and newly-identified needs and wants may be satisfied by the products and services of others....
,
product managementProduct management is an organizational lifecycle function within a company dealing with the planning or marketing of a product or products at all stages of the product lifecycle....
,
operations researchOperations research or Quantitative management, as termed in the USA, Canada, South Africa and Australia, and operational research, as termed in Europe, is an interdisciplinary branch of applied mathematics that uses methods such as mathematical modeling, statistics, and algorithms to arrive at...
, and other applied sciences that deal with large quantities of data.
Factor analysis is related to principal component analysis (PCA) but not identical. Because PCA performs a variance-maximizing rotation of the variable space, it takes into account all variability in the variables. In contrast, factor analysis estimates how much of the variability is due to common factors ("communality"). The two methods become essentially equivalent if the error terms in the factor analysis model (the variability not explained by common factors, see below) can be assumed to all have the same variance.
Definition
Suppose we have a set of observable random variables, with means .
Suppose for some unknown constants and unobserved random variables , where and , where , we have
Here is independently distributed error terms with zero mean and finite variance - which may not be the same for all of them. Let , so that we have
- and .
In matrix terms, we have
Also we will impose the following assumptions on .
- and are independent.
Any solution for the above set of equations following the constraints for is defined as the
factors, and as the
loading matrix.
Suppose . Then note that from the conditions just imposed on , we have
- , or
- , or
Note that for any
orthogonal matrixIn linear algebra, an orthogonal matrix is a square matrix with real entries whose columns are orthogonal unit vectors .Equivalently, a matrix Q is orthogonal if its transpose is equal to its inverse:alternatively,...
if we set and , the criteria for being factors and factor loadings still hold. Hence a set of factors and factor loadings is identical only up to orthogonal transformations.
Example
The following example is a simplification for expository purposes, and should not be taken to be realistic. Suppose a psychologist proposes a theory that there are two kinds of intelligence, "verbal intelligence" and "mathematical intelligence", neither of which is directly observed.
EvidenceEvidence in its broadest sense includes everything that is used to determine or demonstrate the truth of an assertion. Giving or procuring evidence is the process of using those things that are either a) presumed to be true, or b) were themselves proven via evidence, to demonstrate an assertion's...
for the theory is sought in the examination scores from each of 10 different academic fields of 1000 students. If each student is chosen randomly from a large population, then each student's 10 scores are random variables. The psychologist's theory may say that for each of the 10 academic fields, the score averaged over the group of all students who share some common pair of values for verbal and mathematical "intelligences" is some
constantIn mathematics, a constant is a non-varying value, i.e. completely fixed or fixed in the context of use. The term usually occurs in opposition to variable, which is a symbol that stands for a value that may vary....
times their level of verbal intelligence plus another constant times their level of mathematical intelligence, i.e., it is a
linear combinationIn mathematics, linear combinations are a concept central to linear algebra and related fields of mathematics.Most of this article deals with linear combinations in the context of a vector space over a field, with some generalizations given at the end of the article.- Definition:Suppose that K is a...
of those two "factors". The numbers for a particular subject, by which the two kinds of intelligence are multiplied to obtain the expected score, are posited by the theory to be the same for all intelligence level pairs, and are called
"factor loadings" for this subject. For example, the theory may hold that the average student's aptitude in the field of amphibiology is
- {10 × the student's verbal intelligence} + {6 × the student's mathematical intelligence}.
The numbers 10 and 6 are the factor loadings associated with amphibiology. Other academic subjects may have different factor loadings.
Two students having identical degrees of verbal intelligence and identical degrees of mathematical intelligence may have different aptitudes in amphibiology because individual aptitudes differ from average aptitudes. That difference is called the "error" — a statistical term that means the amount by which an individual differs from what is average for his or her levels of intelligence (see
errors and residuals in statisticsIn statistics and optimization, statistical errors and residuals are two closely related and easily confused measures of "deviation of a sample from the mean": the error of a sample is the deviation of the sample from the population mean or actual function, while the residual of a sample is the...
).
The observable data that go into factor analysis would be 10 scores of each of the 1000 students, a total of 10,000 numbers. The factor loadings and levels of the two kinds of intelligence of each student must be inferred from the data.
Mathematical model of the same example
In the example above, for
i = 1, ..., 1,000 the
ith student's scores are
where
- xk,i is the ith student's score for the kth subject
- is the mean of the students' scores for the kth subject (assumed to be zero, for simplicity, in the example as described above, which would amount to a simple shift of the scale used)
- vi is the ith student's "verbal intelligence",
- mi is the ith student's "mathematical intelligence",
- are the factor loadings for the kth subject, for j = 1, 2.
- εk,i is the difference between the ith student's score in the kth subject and the average score in the kth subject of all students whose levels of verbal and mathematical intelligence are the same as those of the ith student,
In
matrixIn mathematics, a matrix is a rectangular array of numbers, such asEntries of a matrix are often denoted by a variable with two subscripts, as shown on the right. Matrices of the same size can be added and subtracted entrywise and matrices of compatible size can be multiplied...
notation, we have
where
- X is a 10 × 1,000 matrix of observable random variables,
- μ is a 10 × 1 column vector of unobservable constants (in this case "constants" are quantities not differing from one individual student to the next; and "random variables" are those assigned to individual students; the randomness arises from the random way in which the students are chosen),
- L is a 10 × 2 matrix of factor loadings (unobservable constants, ten academic topics, each with two intelligence parameters that determine success in that topic),
- F is a 2 × 1,000 matrix of unobservable random variables (two intelligence parameters for each of 1000 students),
- ε is a 10 × 1,000 matrix of unobservable random variables.
Observe that by doubling the scale on which "verbal intelligence"—the first component in each column of
F—is measured, and simultaneously halving the factor loadings for verbal intelligence makes no difference to the model. Thus, no generality is lost by assuming that the standard deviation of verbal intelligence is 1. Likewise for mathematical intelligence. Moreover, for similar reasons, no generality is lost by assuming the two factors are
uncorrelatedIn probability theory and statistics, two real-valued random variables are said to be uncorrelated if their covariance is zero.Uncorrelated random variables have a correlation coefficient of zero, except in the trivial case when both variables have variance zero...
with each other. The "errors" ε are taken to be independent of each other. The variances of the "errors" associated with the 10 different subjects are not assumed to be equal.
Note that, since any rotation of a solution is also a solution, this makes interpreting the factors difficult. See disadvantages below. In this particular example, if we do not know beforehand that the two types of intelligence are uncorrelated, then we cannot interpret the two factors as the two different types of intelligence. Even if they are uncorrelated, we cannot tell which factor corresponds to verbal intelligence and which corresponds to mathematical intelligence without an outside argument.
The values of the loadings
L, the averages μ, and the
varianceIn probability theory and statistics, the variance of a random variable or distribution is the expected square deviation of that variable from its expected value or mean, or to put it another way: variance is the measure of the amount of variation of all the scores for a variable...
s of the "errors" ε must be estimated given the observed data
X.
History
Charles SpearmanCharles Edward Spearman, FRS was an English psychologist known for work in statistics, as a pioneer of factor analysis, and for Spearman's rank correlation coefficient...
spearheaded the use of factor analysis in the field of psychology and is sometimes credited with the invention of factor analysis. He discovered that school children's scores on a wide variety of seemingly unrelated subjects were positively correlated, which led him to postulate that a general mental ability, or
g, underlies and shapes human cognitive performance. His postulate now enjoys broad support in the field of intelligence research, where it is known as the
g theoryThe general intelligence factor is a controversial construct used in the field of psychology to quantify what is common to the scores of all intelligence tests....
.
Raymond CattellRaymond Bernard Cattell was a British and American psychologist known for his exploration of a wide variety of substantive areas in psychology...
expanded on Spearman’s idea of a two-factor theory of intelligence after performing his own tests and factor analysis. He used a multi-factor theory to explain intelligence. Cattell’s theory addressed alternate factors in intellectual development, including motivation and psychology. Cattell also developed several mathematical methods for adjusting psychometric graphs, such as his "scree" test and similarity coefficients. His research led to the development of his theory of
fluid and crystallized intelligenceIn psychology, fluid and crystallized intelligence are factors of general intelligence originally identified by Raymond Cattell. Fluid intelligence is the ability to find meaning in confusion and solve new problems. It is the ability to draw inferences and understand the relationships of various...
, as well as his
16 Personality FactorsThe 16 Personality Factors, measured by the 16PF Questionnaire, were multivariately-derived by psychologist Raymond Cattell.Below is a table outlining this model.- Raymond Cattell's 16 Personality Factors :...
theory of personality. Cattell was a strong advocate of factor analysis and
psychometricsPsychometrics is the field of study concerned with the theory and technique of educational and psychological measurement, which includes the measurement of knowledge, abilities, attitudes, and personality traits. The field is primarily concerned with the study of measurement instruments such as...
. He believed that all theory should be derived from research, which supports the continued use of empirical observation and objective testing to study human intelligence.
Applications in psychology
Factor analysis is used to identify "factors" that explain a variety of results on different tests. For example, intelligence research found that people who get a high score on a test of verbal ability are also good on other tests that require verbal abilities. Researchers explained this by using factor analysis to isolate one factor, often called crystallized intelligence or verbal intelligence, that represents the degree to which someone is able to solve problems involving verbal skills.
Factor analysis in psychology is most often associated with intelligence research. However, it also has been used to find factors in a broad range of domains such as personality, attitudes, beliefs, etc. It is linked to
psychometricsPsychometrics is the field of study concerned with the theory and technique of educational and psychological measurement, which includes the measurement of knowledge, abilities, attitudes, and personality traits. The field is primarily concerned with the study of measurement instruments such as...
, as it can assess the validity of an instrument by finding if the instrument indeed measures the postulated factors.
Advantages
- Reduction of number of variables, by combining two or more variables into a single factor. For example, performance at running, ball throwing, batting, jumping and weight lifting could be combined into a single factor such as general athletic ability. Usually, in an item by people matrix, factors are selected by grouping related items. In the Q factor analysis technique, the matrix is transposed and factors are created by grouping related people: For example, liberals, libertarians, conservatives and socialists, could form separate groups.
- Identification of groups of inter-related variables, to see how they are related to each other. For example, Carroll used factor analysis to build his Three Stratum Theory
In 1993 John Carroll published "Human cognitive abilities: A survey of factor-analytic studies", which outlined his hierarchical, Three-Stratum Theory of cognitive abilities. The theory is based on a factor analytic study of correlation of individual differences variables from measures including...
. He found that a factor called "broad visual perception" relates to how good an individual is at visual tasks. He also found a "broad auditory perception" factor, relating to auditory task capability. Furthermore, he found a global factor, called "g" or general intelligence, that relates to both "broad visual perception" and "broad auditory perception". This means someone with a high "g" is likely to have both a high "visual perception" capability and a high "auditory perception" capability, and that "g" therefore explains a good part of why someone is good or bad in both of those domains.
Disadvantages
- "...each orientation is equally acceptable mathematically. But different factorial theories proved to differ as much in terms of the orientations of factorial axes for a given solution as in terms of anything else, so that model fitting did not prove to be useful in distinguishing among theories." (Sternberg, 1977). This means all rotations represent different underlying processes, but all rotations are equally valid outcomes of standard factor analysis optimization. Therefore, it is impossible to pick the proper rotation using factor analysis alone.
- Factor analysis can be only as good as the data allows. In psychology, where researchers have to rely on more or less valid and reliable measures such as self-reports, this can be problematic.
- Interpreting factor analysis is based on using a “heuristic”, which is a solution that is "convenient even if not absolutely true" (Richard B. Darlington). More than one interpretation can be made of the same data factored the same way, and factor analysis cannot identify causality.
Factor analysis in marketing
The basic steps are:
- Identify the salient attributes consumers use to evaluate products
The noun product is defined as a "thing produced by labor or effort" or the "result of an act or a process", and stems from the verb produce, from the Latin prōdūce ' lead or bring forth'. Since 1575, the word "product" has referred to anything produced. Since 1695, the word has referred to "thing...
in this category.
- Use quantitative marketing research
Quantitative marketing research is the application of quantitative research techniques to the field of marketing. It has roots in both the positivist view of the world, and the modern marketing viewpoint that marketing is an interactive process in which both the buyer and seller reach a satisfying...
techniques (such as surveysStatistical surveys are used to collect quantitative information about items in a population. Surveys of human populations and institutions are common in political polling and government, health, social science and marketing research. A survey may focus on opinions or factual information depending...
) to collect data from a sample of potential customerA customer, also called client, buyer, or purchaser, is usually used to refer to a current or potential buyer or user of the products of an individual or organization, called the supplier, seller, or vendor. This is typically through purchasing or renting goods or services...
s concerning their ratings of all the product attributes.
- Input the data into a statistical program and run the factor analysis procedure. The computer will yield a set of underlying attributes (or factors).
- Use these factors to construct perceptual maps
Perceptual mapping is a graphics technique used by asset marketers that attempts to visually display the perceptions of customers or potential customers. Typically the position of a product, product line, brand, or company is displayed relative to their competition.Perceptual maps can have any...
and other product positioningIn marketing, positioning has come to mean the process by which marketers try to create an image or identity in the minds of their target market for its product, brand, or organization...
devices.
Information collection
The data collection stage is usually done by marketing research professionals. Survey questions ask the respondent to rate a product sample or descriptions of product concepts on a range of attributes. Anywhere from five to twenty attributes are chosen. They could include things like: ease of use, weight, accuracy, durability, colourfulness, price, or size. The attributes chosen will vary depending on the product being studied. The same question is asked about all the products in the study. The data for multiple products is coded and input into a statistical program such as
RIn computing, R is a programming language and software environment for statistical computing and graphics. It is an implementation of the S programming language with lexical scoping semantics inspired by Scheme. R was created by Ross Ihaka and Robert Gentleman at the University of Auckland, New...
,
SPSSPASW is a computer program used for statistical analysis. Before 2009 it was called SPSS, but in 2009 it was re-branded as PASW...
,
SASSAS is an integrated system of software products provided by SAS Institute that enables the programmer to perform:*data entry, retrieval, management, and mining*report writing and graphics...
,
StataStata is a general-purpose statistical software package created in 1985 by StataCorp. It is used by many businesses and academic institutions around the world...
, and SYSTAT.
Analysis
The analysis will isolate the underlying factors that explain the data. Factor analysis is an interdependence technique. The complete set of interdependent relationships are examined. There is no specification of either dependent variables, independent variables, or causality. Factor analysis assumes that all the rating data on different attributes can be reduced down to a few important dimensions. This reduction is possible because the attributes are related. The rating given to any one attribute is partially the result of the influence of other attributes. The statistical algorithm deconstructs the rating (called a raw score) into its various components, and reconstructs the partial scores into underlying factor scores. The degree of correlation between the initial raw score and the final factor score is called a
factor loading. There are two approaches to factor analysis: "principal component analysis" (the total
varianceIn probability theory and statistics, the variance of a random variable or distribution is the expected square deviation of that variable from its expected value or mean, or to put it another way: variance is the measure of the amount of variation of all the scores for a variable...
in the data is considered); and "common factor analysis" (the common variance is considered).
Note that principal component analysis and common factor analysis differ in terms of their conceptual underpinnings. The factors produced by principal component analysis are conceptualized as being linear combinations of the variables whereas the factors produced by common factor analysis are conceptualized as being latent variables. Computationally, the only difference is that the diagonal of the relationships matrix is replaced with communalities (the variance accounted for by more than one variable) in common factor analysis. This has the result of making the factor scores indeterminate and thus differ depending on the method used to compute them whereas those produced by principal component analysis are not dependent on the method of computation. Although there have been heated debates over the merits of the two methods, a number of leading statisticians have concluded that in practice there is little difference (Velicer and Jackson, 1990) which makes sense since the computations are quite similar despite the differing conceptual bases, especially for datasets where communalities are high and/or there are many variables, reducing the influence of the diagonal of the relationship matrix on the final result (Gorsuch, 1983).
The use of principal components in a semantic space can vary somewhat because the components may only "predict" but not "map" to the vector space. This produces a statistical principal component use where the most salient words or themes represent the preferred
basisIn linear algebra, a basis is a set of vectors that, in a linear combination, can represent every vector in a given vector space or free module, and such that no element of the set can be represented as a linear combination of the others...
. [ok]
Advantages
- Both objective and subjective attributes can be used
- Factor Analysis can be used to identify the hidden dimensions or constructs which may or may not be apparent from direct analysis.
- It is not extremely difficult to do, inexpensive, and accurate
- There is flexibility in naming and using dimensions
Disadvantages
- Usefulness depends on the researchers' ability to develop a complete and accurate set of product attributes - If important attributes are missed the value of the procedure is reduced accordingly.
- Naming of the factors can be difficult - multiple attributes can be highly correlated with no apparent reason.
- If the observed variables are completely unrelated, factor analysis is unable to produce a meaningful pattern (though the eigenvalues will highlight this: suggesting that each variable should be given a factor in its own right).
- If sets of observed variables are highly similar to each other but distinct from other items, Factor analysis will assign a factor to them, even though this factor will essentially capture true variance of a single item. In other words, it is not possible to know what the 'factors' actually represent; only theory can help inform the researcher on this.
Factor analysis in physical sciences
Factor analysis has also been widely used in physical sciences such as
geochemistryThe field of geochemistry involves study of the chemical composition of the Earth and other planets, chemical processes and reactions that govern the composition of rocks and soils, and the cycles of matter and energy that transport the Earth's chemical components in time and space, and their...
,
ecologyEcology is the interdisciplinary scientific study of the interactions between organisms and the interactions of these organisms with their environment....
, and hydrochemistry .
In groundwater quality management, it is important to relate the spatial distribution of different chemical
parameters to different possible sources, which have different chemical signatures. For example, a sulfide mine is likely to be associated with high levels of acidity, dissolved sulfates and transition metals. These signatures can be identified as factors through R-mode factor analysis, and the location of possible sources can be suggested by contouring the factor scores.
In
geochemistryThe field of geochemistry involves study of the chemical composition of the Earth and other planets, chemical processes and reactions that govern the composition of rocks and soils, and the cycles of matter and energy that transport the Earth's chemical components in time and space, and their...
, different factors can correspond to different mineral associations, and thus to mineralisation.
Factor analysis in economics
Economists might use factor analysis to see whether productivity, profits and workforce can be reduced down to an underlying dimension of company growth.
External links