Self-information

	Email		Print
	Bookmark		Link

Self-information

In information theory

Information theory

Information theory is a branch of applied mathematics and electrical engineering involving the quantification of information. Historically, information theory was developed by Claude E....
(elaborated by Claude E. Shannon, 1948), self-information is a measure of the information content associated with the outcome of a random variable

Random variable

In mathematics, random variables are used in the study of Randomness and probability. They were developed to assist in the analysis of Game of chance, stochastic events, and the results of experiment by capturing only the mathematical properties necessary to answer probability questions....
. It is expressed in a unit

Units of measurement

The definition, agreement and practical use of units of measurement have played a crucial role in human endeavour from early ages up to this day....
of information

Information

Information as a Conveyed concept has a diversity of meanings, from everyday usage to technical settings. Generally speaking, the concept of information is closely related to notions of constraint, communication, control system, data, form, instruction, knowledge, Meaning , stimulation, pattern, perception, and knowledge representation....
, for example bit

Bit

A bit is a binary numeral system numerical digit, taking a value of either 0 or 1. Binary digits are a basic unit of information Computer data storage and transmission in digital computing and digital information theory....
s, nat

Nat (information)

A nat is a logarithmic unit of information or information entropy, based on natural logarithms and powers of e , rather than the powers of 2 and binary logarithm which define the bit....
s, or hartley

Ban (information)

A ban, sometimes called a hartley or a dit , is a logarithmic unit which measures information or information entropy, based on base 10 logarithms and powers of 10, rather than the powers of 2 and binary logarithm which define the bit....
s, depending on the base of the logarithm used in its calculation. The term self-information is also sometimes used as a synonym of entropy, i.e. the expected value of self-information in the first sense, because , where is the mutual information

Mutual information

In probability theory and information theory, the mutual information of two random variables is a quantity that measures the mutual dependence of the two variables....
of X with itself.

Discussion

Ask a question about 'Self-information'

Start a new discussion about 'Self-information'

Answer questions from other users

Full Discussion Forum

Encyclopedia

In information theory

Information theory

Random variable

Units of measurement

The definition, agreement and practical use of units of measurement have played a crucial role in human endeavour from early ages up to this day....
of information

Information

Bit

Nat (information)

Ban (information)

Mutual information

In probability theory and information theory, the mutual information of two random variables is a quantity that measures the mutual dependence of the two variables....
of X with itself. These two meanings are not equivalent, and this article covers the first sense only. For the other sense, see entropy.

By definition, the amount of self-information contained in a probabilistic event

Event (probability theory)

In probability theory, an event is a Set of outcomes to which a probability is assigned. Typically, when the sample space is finite, any subset of the sample space is an event ....
depends only on the probability

Probability

Probability, or wikt:chance, is a way of expressing knowledge or belief that an Event will occur or has occurred. In mathematics the concept has been given an exact meaning in probability theory, that is used extensively in such areas of study as mathematics, statistics, finance, gambling, science, and philosophy to draw conclusions about t...
of that event: the smaller its probability, the larger the self-information associated with receiving the information that the event indeed occurred.

Further, by definition, the measure

Measurement

Measurement is the process of assigning a number to an attribute according to a rule or set of rules. The term can also be used to refer to the result obtained after performing the process....
of self-information has the following property. If an event C is composed of two mutually independent

Statistical independence

In probability theory, to say that two event s are independent intuitively means that the occurrence of one event makes it neither more nor less probable that the other occurs....
events A and B, then the amount of information at the proclamation that C has happened, equals the sum of the amounts of information at proclamations of event A and event B respectively.

Taking into account these properties, the self-information associated with outcome with probability is:

This definition complies with the above conditions. In the above definition, the base of the logarithm is not specified: if using base 2, the unit of is in bit

Bit

Nat (information)

Ban (information)

Surprise (emotion)

Surprise is a brief emotional state that is the result of experiencing an expectation relevant event. Surprise can have any Valence ; that is, it can be neutral, pleasant, or unpleasant....
" of seeing the outcome (a highly probable outcome is not surprising). This term was coined by Myron Tribus

Myron Tribus

Myron T. Tribus is perhaps best known as former director of the Center for Advanced Engineering Study at MIT. He headed the center when it published W....
in his 1961 book Thermostatics and Thermodynamics.

The information entropy

Information entropy

In information theory, entropy is a measure of the uncertainty associated with a random variable. The term by itself in this context usually refers to the Shannon entropy, which quantifies, in the sense of an expected value, the self-information contained in a message, usually in units such as bits....
of a random event is the expected value

Expected value

In probability theory and statistics, the expected value of a random variable is the Lebesgue integral of the random variable with respect to its probability measure....
of its self-information.

Self-information is an example of a proper scoring rule

Scoring rule

In decision theory a score function, or scoring rule, is a measure of someone's performance when they are repeatedly making decisions under uncertainty....
.

Examples

On tossing a coin
Coin flipping

Coin flipping or coin tossing is the practice of throwing a coin in the air to resolve a dispute between two parties or otherwise choose between two alternatives....
, the chance of 'tail' is 0.5. When it is proclaimed that indeed 'tail' occurred, this amounts to

I('tail') = log₂ (1/0.5) = log₂ 2 = 1 bits of information.

When throwing a fair dice
Dice

A die is a small polyhedron object, usually cubic, used for generating Statistical randomnesss or other symbols. This makes dice suitable as gambling devices, especially for craps or sic bo, or for use in non-gambling tabletop games....
, the probability of 'four' is 1/6. When it is proclaimed that 'four' has been thrown, the amount of self-information is

I('four') = log₂ (1/(1/6)) = log₂ (6) = 2.585 bits.

When, independently, two dice are thrown, the amount of information associated with equals

I('throw 1 is two & throw 2 is four') = log₂ (1/P(throw 1 = 'two' & throw 2 = 'four')) = log₂ (1/(1/36)) = log₂ (36) = 5.170 bits.
This outcome equals the sum of the individual amounts of self-information associated with and ; namely 2.585 + 2.585 = 5.170 bits.

Suppose that the average probability of finding survivors in a large evolving population is P, then - when a survivor has been found, the amount of self-information will be -log_e(P) nats (-log₂(P) bits).

Self-information

Examples

External links