1. Home
  2. Information and signal theory
  3. Information theory
  4. Differential entropy

5.4 Differential entropy

<< Chapter < Page
  Information and signal theory   Page 1 / 1
Chapter >> Page >
In this module we consider differential entropy.

Consider the entropy of continuous random variables. Whereas the (normal) entropy is the entropy of a discrete random variable, the differential entropy is the entropy of a continuous random variable.

Differential entropy

Differential entropy
The differential entropy h X of a continuous random variable X with a pdf f x is defined as
h X x ∞ ∞ f x f x
Usually the logarithm is taken to be base 2, so that the unit of the differential entropy is bits/symbol. Note that is the discrete case, h X depends only on the pdf of X . Finally, we note that the differential entropy is the expected value of f x , i.e.,
h X E f x

Now, consider a calculating the differential entropy of some random variables.

Consider a uniformly distributed random variable X from c to c Δ . Then its density is 1 Δ from c to c Δ , and zero otherwise.

We can then find its differential entropy as follows,

h X x c c Δ 1 Δ 1 Δ Δ
Note that by making Δ arbitrarily small, the differential entropy can be made arbitrarily negative, while taking Δ arbitrarily large, the differential entropy becomes arbitrarily positive.

Got questions? Get instant answers now!

Consider a normal distributed random variable X , with mean m and variance σ 2 . Then its density is 1 2 σ 2 e x m 2 2 σ 2 .

We can then find its differential entropy as follows, first calculate f x :

f x 1 2 2 σ 2 e x m 2 2 σ 2
Then since E X m 2 σ 2 , we have
h X 1 2 2 σ 2 1 2 e 1 2 2 e σ 2

Got questions? Get instant answers now!

Properties of the differential entropy

In the section we list some properties of the differential entropy.

  • The differential entropy can be negative
  • h X c h X , that is translation does not change the differential entropy.
  • h a X h X a , that is scaling does change the differential entropy.
The first property is seen from both and . The two latter can be shown by using .
<< Chapter < Page Page > Chapter >>

Read also:

Get Jobilize Job Search Mobile App in your pocket Now!

Get it on Google Play Download on the App Store Now




Source:  OpenStax, Information and signal theory. OpenStax CNX. Aug 03, 2006 Download for free at http://legacy.cnx.org/content/col10211/1.19
Google Play and the Google Play logo are trademarks of Google Inc.

Notification Switch

Would you like to follow the 'Information and signal theory' conversation and receive update notifications?

Ask