6.19 Entropy

Equally likely symbols each have a probability of $\frac{1}{K}$ . Thus, $H(A)=-\sum \frac{1}{K}\log_2\left(\frac{1}{K}\right)=\log_{2}K$ . To prove that this is the maximum-entropy probability assignment, we must explicitly take into accountthat probabilities sum to one. Focus on a particular symbol, say the first. $(a_0)$ appears twice in the entropy formula: the terms $(a_0)\log_2(a_0)$ and $(1-(a_0)+\dots +(a_{K-2}))\log_2(1-(a_0)+\dots +(a_{K-2}))$ . The derivative with respect to this probability (and all the others) must be zero. The derivative equals $\log_2(a_0)-\log_2(1-(a_0)+\dots +(a_{K-2}))$ , and all other derivatives have the same form (just substitute your letter's index). Thus, eachprobability must equal the others, and we are done. For the minimum entropy answer, one term is $1\log_{2}1=0$ , and the others are $0\log_{2}0$ , which we define to be zero also. The minimum value of entropy is zero.