CCJ PRML Study Note - Chapter 1.6 : Information Theory
Chapter 1.6 : Information Theory
Chapter 1.6 : Information Theory
Christopher M. Bishop, PRML, Chapter 1 Introdcution
1. Information h(x)
Given a random variable
and we ask how much information is received when we observe a specific value for this variable.
- The amount of information can be viewed as the “degree of surprise” on learning the value of
.
- information
:
where the negative sign ensures that information is positive or zero.
- the units of
:
- using logarithms to the base of 2: the units of
are bits (‘binary digits’).
- using logarithms to the base of
, i.e., natural logarithms: the units of
are nats.
- using logarithms to the base of 2: the units of
2. Entropy H(x): average amount of information
2.1 Entropy H(x)
Firstly we interpret the concept of entropy in terms of the average amount of information needed to specify the state of a random variable.
Now suppose that a sender wishes to transmit the value of a random variable to a receiver. The average amount of information that they transmit in the process is obtained by taking the expectation of (1.92) with respect to the distribution and is given as
- discrete entropy for discrete random variable by
- or differential/continuous entropy for continuous random variable by
- Note that
and so we shall take
whenever we encounter a value for
such that
.
- The nonuniform distribution has a smaller entropy than the uniform one.
2.2 Noiseless coding theorem (Shannon, 1948)
The noiseless coding theorem states that the entropy is a lower bound on the number of bits needed to transmit the state of a random variable.
2.3 Alternative view of entropy H(x)
Secondly, let us introduces the concept of entropy in physics in the context of equilibrium thermodynamics and later given a deeper interpretation as a measure of disorder through developments in statistical mechanics.
Consider a set of identical objects that are to be divided amongst a set of bins, such that there are
objects in the
bin. Consider the number of different ways of allocating the objects to the bins.
- There are
ways to choose the first object,
ways to choose the second object, and so on, leading to a total of
ways to allocate all
objects to the bins.
- However, we don’t wish to distinguish between rearrangements of objects within each bin. In the
bin there are
ways of reordering the objects, and so the total number of ways of allocating the
objects to the bins is given by
which is called the multiplicity.
- The entropy is then defined as the logarithm of the multiplicity scaled by an appropriate constant
- We now consider the limit
, in which the fractions
are held fixed, and apply Stirling’s approximation
- which gives
- where we have used
, and
is the probability of an object being assigned to the
bin.
- microstate: In physics terminology, the specific arrangements of objects in the bins is called a microstate,
- macrostate: the overall distribution of occupation numbers, expressed through the ratios
, is called macrostate.
- The multiplicity
is also known as the weight of the macrostate.
- We can interpret the bins as the states
of a discrete random variable
, where
. The entropy of the random variable X is then
2.4 Comparison between discrete entropy and continuous entropy
H(x) | Discrete Distribution X | Continuous Distribution X |
---|---|---|
Maximum | Uniform X | Gaussian X |
+/- | could be negative |
- Maximum entropy H(x) :
- In the case of discrete distributions, the maximum entropy configuration corresponded to an equal distribution of probabilities across the possible states of the variable.
- For a continuous variable, the distribution that maximizes the differential entropy is the Gaussian [see Page 54 in PRML].
- Is Negative or Positive?
- The discrete entropy in (1.93) is always
, because
. It will equal its minimum value of
when one of the
and all other
.
- The differential entropy can be negative, because
in (1.110) for
.
If we evaluate the differential entropy of the Gaussian, we obtain
This result also shows that the differential entropy, unlike the discrete entropy, can be negative, because
in (1.110) for
.
- The discrete entropy in (1.93) is always
2.5 Conditional entropy H(y|x)
- Conditional entropy:
Suppose we have a joint distributionfrom which we draw pairs of values of
and
. If a value of
is already known, then the additional information needed to specify the corresponding value of
is given by
. Thus the average additional information needed to specify y can be written as
which is called the conditional entropy of y given x. - It is easily seen, using the product rule, that the conditional entropy satisfies the relation
where
is the differential entropy (i.e., continuous entropy) of
, and
is the differential entropy of the marginal distribution
.
- From (1.112) we get to know that
the information needed to describe
and
is given by the sum of the information needed to describe
alone plus the additional information required to specify
given
.
3. Relative entropy or KL divergence
3.1 Relative entropy or KL divergence
Problem: How to relate the notion of entropy to pattern recognition?
Description: Consider some unknown distribution, and suppose that we have modeled this using an approximating distribution
.
- Relative entropy or Kullback-Leibler divergence, or KL divergence between the distributions
and
: If we use
to construct a coding scheme for the purpose of transmitting values of
to a receiver, then the average additional amount of information (in nats) required to specify the value of
(assuming we choose an efficient coding scheme) as a result of using
instead of the true distribution
is given by
and
-
It can be rewritten as [see Ref-1]
-
Cross Entropy: where
is called the cross entropy,
- Understanding of Cross Entropy: One can show that the cross entropy is the average number of bits (or nats) needed to encode data coming from a source with distribution
when we use model
to define our codebook.
- Understanding of (“Regular”) Entropy: Hence the “regular” entropy
, is the expected number of bits if we use the true model.
- Understanding of Relative Entropy: So the KL divergence is the difference between these (shown in 2.111). In other words, the KL divergence is the average number of extra bits (or nats) needed to encode the data, due to the fact that we used approximation distribution
to encode the data instead of the true distribution
.
- Understanding of Cross Entropy: One can show that the cross entropy is the average number of bits (or nats) needed to encode data coming from a source with distribution
-
Asymmetric: Note that KL divergence is not a symmetrical quantity, that is to say
.
- KL divergence is a way to measure the dissimilarity of two probability distributions,
and
[see Ref-1].
3.2 Information inequality [see Ref-1]
The “extra number of bits” interpretation should make it clear that , and that the KL is only equal to zero i.f.f.
. We now give a proof of this important result.
Proof:
- 1) Convex functions: To do this we first introduce the concept of convex functions. A function
is said to be convex if it has the property that every chord lies on or above the function, as shown in Figure 1.31.
- Convexity then implies
- Convexity then implies
- 2) Jensen’s inequality:
- Using the technique of proof by induction(数学归纳法), we can show from (1.114) that a convex function
satisfies
where
and
, for any set of points
. The result (1.115) is known as Jensen’s inequality.
- If we interpret the
as the probability distribution over a discrete variable
taking the values
, then (1.115) can be written
For continuous variables, Jensen’s inequality takes the form
- Using the technique of proof by induction(数学归纳法), we can show from (1.114) that a convex function
- 3) Apply Jensen’s inequality in the form (1.117) to the KL divergence (1.113) to give
where we have used the fact that
is a convex function (In fact,
is a strictly convex function, so the equality will hold if, and only if,
for all
), together with the normalization condition
.
- 4) Similarly, Let
be the support of
, and apply Jensen’s inequality in the form (1.115) to the discrete form KL divergence (2.110) to get [see Ref-1]
where the first inequality follows from Jensen’s. Since
is a strictly concave (i.e., the inverse of convex) function, we have equality in Equation (2.115) iff
for some
. We have equality in Equation (2.116) iff
, which implies
.
- 5) Hence
iff
for all
.
3.3 How to use KL divergence
Note that:
- we can interpret the KL divergence as a measure of the dissimilarity of the two distributions
and
.
- If we use a distribution that is different from the true one, then we must necessarily have a less efficient coding, and on average the additional information that must be transmitted is (at least) equal to the Kullback-Leibler divergence between the two distributions.
Problem description:
- Suppose that data is being generated from an unknown distribution
that we wish to model.
- We can try to approximate this distribution using some parametric distribution
, governed by a set of adjustable parameters
, for example a multivariate Gaussian.
- One way to determine
is to minimize the KL divergence between
and
with respect to
.
- We cannot do this directly because we don’t know
. Suppose, however, that we have observed a finite set of training points
, for
, drawn from
. Then the expectation with respect to
can be approximated by a finite sum over these points, using
(1.35), so that (???don’t know how to derive it)
- the first term is the negative log likelihood function for
under the distribution
evaluated using the training set.
- Thus we see that minimizing this KL divergence is equivalent to maximizing the likelihood function.
3.4 Mutual information
Now consider the joint distribution between two sets of variables
and
given by
.
Mutual information between the variables and
:
- If
and
are independent, p(x, y) = p(x)p(y).
- If
and
are not independent, we can gain some idea of whether they are “close” to being independent by considering the KL divergence between the joint distribution and the product of the marginals, given by
which is called the mutual information between the variables
and
.
- Using the sum and product rules of probability, we see that the mutual information is related to the conditional entropy
through
Understanding of Mutual information:
- Thus we can view the mutual information as the reduction in the uncertainty about
by virtue of being told the value of
(or vice versa).
- From a Bayesian perspective, we can view
as the prior distribution for
and
as the posterior distribution after we have observed new data
. The mutual information therefore represents the reduction in uncertainty about
as a consequence of the new observation
.
Reference
[1]: Section 2.8.2, Page 57, Kevin P. Murphy. 2012. Machine Learning: A Probabilistic Perspective. The MIT Press.