|
Probably a basic question, but I can't seem to wrap my head around it: In wikipedia the entropy of an univariate gaussian is (1/2)log(2 * Pi * e * var). But in the special case of variance of zero, the log will evaluate as -Infinity. Also for small variances (such that the log will return a negative number), the entropy is negative. This doesn't make sense, the entropy cannot be smaller than zero. What did I miss here?
showing 5 of 6
show all
|
|
There is no natural definition of entropy for continuous distributions that preserves all properties of the entropy for discrete distributions. The most common definition, differential entropy is the one which gives the value you refer to for the gaussian. While the differential entropy has often the same intuitive interpretations as normal entropy (in that more "uncertainty" implies more entropy), it differs in a few crucial ways, including that it can be negative, as you noticed. Okay, thanks for your answer. So I guess I will have to live with this and adapt my algorithm accordingly...
(May 15 '13 at 19:12)
r follador
|
|
You can think of it as "relative to a bin size", with a unit bin size. So when the gaussian gets skinnier than one bin, the entropy becomes negative, because you know with greater precision than the bin-size limit. If you think of it this way, then changing the bin size just adds or subtracts a constant. If you make the bin size small enough that the distribution does not have features smaller than a single bin, then the constant you add will make the entropy positive. So what is this constant you should add to make it "relative to bin size b"? Simply log(b). Which comes to zero for b=1. |
If the variance is zero, the gaussian distribution does not make sense at all either does it?
Also, from the definition of empiric variance, the only case when the variance is zero would be if all and every sampling point is equal to the mean.
You would not have a probability distribution, but a single point in space.
And remember that the entropy, per definition measures the uncertainty of a random variable, but if you have zero variance, you don't have a random variable, you have a deterministic variable.
I'm assuming gaussian distribution, I don't know the variance beforehand. In the special case of variance of zero the entropy is supposed to be zero, but the equation gives me negative infinity.
I could of course check for this special case, but I also don't understand why for example a variance of 0.005 has an entropy of -1.23022
I completely agree that the entropy measures the uncertainty of a random variable; but wouldn't the case of a gaussian having a variance of zero be somehow equal to the example in wikipedia (http://en.wikipedia.org/wiki/Entropy_(information_theory)) of a coin toss with two heads, which has an entropy of zero. A coin having two heads will also have a variance of zero.
A coin toss is a Bernoulli process, not a gaussian distribution, In which case the variance would be zero for the case where the probability of a 1 is equal to zero
Yes, you got it! And the entropy is also zero
But in that case, since the probability of the event happening is zero, it means that is no longer a random event, since the coin will always fall on the alternative (x=0)