Cross entropy
Encyclopedia
In information theory
, the cross entropy between two probability distribution
s measures the average number of bit
s needed to identify an event from a set of possibilities, if a coding scheme is used based on a given probability distribution
, rather than the "true" distribution 
.
The cross entropy for two distributions
and 
over the same probability space
is thus defined as follows:
,
where
is the entropy
of
, and 
is the Kullback-Leibler divergence of 
from 
(also known as the relative entropy).
For discrete
and 
this means
The situation for continuous distributions is analogous:
NB: The notation
is sometimes used for both the cross entropy as well as the joint entropy of 
and 
.
is unknown. An example is language model
ing, where a model is created based on a training set
, and then its cross-entropy is measured on a test set to assess how accurate the model is in predicting the test data. In this example, 
is the true distribution of words in any corpus, and 
is the distribution of words as predicted by the model. Since the true distribution is unknown, cross-entropy cannot be directly calculated. In these cases, an estimate of cross-entropy is calculated using the following formula:
where
is the size of the test set, and 
is the probability of event 
estimated from the training set. It should be noted that the sum is calculated over 
.
.
When comparing a distribution
against a fixed reference distribution 
, cross entropy and KL divergence are identical up to an additive constant (since 
is fixed): both take on their minimal values when 
, which is 
for KL divergence, and 
for cross entropy. In the engineering literature, the principle of minimising KL Divergence (Kullback's "Principle of Minimum Discrimination Information") is often called the Principle of Minimum Cross-Entropy (MCE), or Minxent.
However, as discussed in the article Kullback-Leibler divergence, sometimes the distribution q is the fixed prior reference distribution, and the distribution p is optimised to be as close to q as possible, subject to some constraint. In this case the two minimisations are not equivalent. This has led to some ambiguity in the literature, with some authors attempting to resolve the inconsistency by redefining cross-entropy to be DKL(p||q) , rather than H(p,q).
Information theory
Information theory is a branch of applied mathematics and electrical engineering involving the quantification of information. Information theory was developed by Claude E. Shannon to find fundamental limits on signal processing operations such as compressing data and on reliably storing and...
, the cross entropy between two probability distribution
Probability distribution
In probability theory, a probability mass, probability density, or probability distribution is a function that describes the probability of a random variable taking certain values....
s measures the average number of bit
Bit
A bit is the basic unit of information in computing and telecommunications; it is the amount of information stored by a digital device or other physical system that exists in one of two possible distinct states...
s needed to identify an event from a set of possibilities, if a coding scheme is used based on a given probability distribution
, rather than the "true" distribution .The cross entropy for two distributions
and over the same probability spaceProbability space
In probability theory, a probability space or a probability triple is a mathematical construct that models a real-world process consisting of states that occur randomly. A probability space is constructed with a specific kind of situation or experiment in mind...
is thus defined as follows:
,where
is the entropyInformation entropy
In information theory, entropy is a measure of the uncertainty associated with a random variable. In this context, the term usually refers to the Shannon entropy, which quantifies the expected value of the information contained in a message, usually in units such as bits...
of
, and is the Kullback-Leibler divergence of from (also known as the relative entropy).For discrete
and this meansThe situation for continuous distributions is analogous:
NB: The notation
is sometimes used for both the cross entropy as well as the joint entropy of and .Estimation
There are many situations where cross-entropy needs to be measured but the distribution of is unknown. An example is language modelLanguage model
A statistical language model assigns a probability to a sequence of m words P by means of a probability distribution.Language modeling is used in many natural language processing applications such as speech recognition, machine translation, part-of-speech tagging, parsing and information...
ing, where a model is created based on a training set
, and then its cross-entropy is measured on a test set to assess how accurate the model is in predicting the test data. In this example, is the true distribution of words in any corpus, and is the distribution of words as predicted by the model. Since the true distribution is unknown, cross-entropy cannot be directly calculated. In these cases, an estimate of cross-entropy is calculated using the following formula:where
is the size of the test set, and is the probability of event estimated from the training set. It should be noted that the sum is calculated over .Cross-entropy minimization
Cross-entropy minimization is frequently used in optimization and rare-event probability estimation; see the cross-entropy methodCross-entropy method
The cross-entropy method attributed to Reuven Rubinstein is a general Monte Carlo approach tocombinatorial and continuous multi-extremal optimization and importance sampling.The method originated from the field of rare event simulation, where...
.
When comparing a distribution
against a fixed reference distribution , cross entropy and KL divergence are identical up to an additive constant (since is fixed): both take on their minimal values when , which is for KL divergence, and for cross entropy. In the engineering literature, the principle of minimising KL Divergence (Kullback's "Principle of Minimum Discrimination Information") is often called the Principle of Minimum Cross-Entropy (MCE), or Minxent.However, as discussed in the article Kullback-Leibler divergence, sometimes the distribution q is the fixed prior reference distribution, and the distribution p is optimised to be as close to q as possible, subject to some constraint. In this case the two minimisations are not equivalent. This has led to some ambiguity in the literature, with some authors attempting to resolve the inconsistency by redefining cross-entropy to be DKL(p||q) , rather than H(p,q).