Infomax
Encyclopedia
Infomax is an optimization principle for neural networks
and other information processing systems. It prescribes that a function that maps a set of input values I to a set of output values O should be chosen or learned so as to maximize the average Shannon mutual information
between I and O, subject to a set of specified constraints and/or noise processes. Infomax algorithms are learning algorithms that perform this optimization process. The principle was described by Linsker in 1987.
Infomax, in its zero-noise limit, is related to the principle of redundancy reduction proposed for biological sensory processing by Horace Barlow
in 1961, and applied quantitatively to retinal processing by Atick and Redlich.
One of the applications of infomax has been to an Independent component analysis
algorithm
that finds independent signals by maximising entropy
. Infomax-based ICA was described by Bell and Sejnowski in 1995.
Neural Networks
Neural Networks is the official journal of the three oldest societies dedicated to research in neural networks: International Neural Network Society, European Neural Network Society and Japanese Neural Network Society, published by Elsevier...
and other information processing systems. It prescribes that a function that maps a set of input values I to a set of output values O should be chosen or learned so as to maximize the average Shannon mutual information
Mutual information
In probability theory and information theory, the mutual information of two random variables is a quantity that measures the mutual dependence of the two random variables...
between I and O, subject to a set of specified constraints and/or noise processes. Infomax algorithms are learning algorithms that perform this optimization process. The principle was described by Linsker in 1987.
Infomax, in its zero-noise limit, is related to the principle of redundancy reduction proposed for biological sensory processing by Horace Barlow
Horace Barlow
Horace Basil Barlow FRS is a British visual neuroscientist.Barlow is the son of the civil servant Sir Alan Barlow and his wife Lady Nora, , and thus the great-grandson of Charles Darwin . He earned an M.D...
in 1961, and applied quantitatively to retinal processing by Atick and Redlich.
One of the applications of infomax has been to an Independent component analysis
Independent component analysis
Independent component analysis is a computational method for separating a multivariate signal into additive subcomponents supposing the mutual statistical independence of the non-Gaussian source signals...
algorithm
Algorithm
In mathematics and computer science, an algorithm is an effective method expressed as a finite list of well-defined instructions for calculating a function. Algorithms are used for calculation, data processing, and automated reasoning...
that finds independent signals by maximising entropy
Entropy
Entropy is a thermodynamic property that can be used to determine the energy available for useful work in a thermodynamic process, such as in energy conversion devices, engines, or machines. Such devices can only be driven by convertible energy, and have a theoretical maximum efficiency when...
. Infomax-based ICA was described by Bell and Sejnowski in 1995.