|
What cannot be said about a Confidence Interval? Is it correct to say:
Or is it correct only under some assumptions? |
|
It's correctly under some unreasonable almost-bayesian assumptions. What a confidence interval says is that, repeating that same interval-calculating process with different distributions infinitely many times, 90% of the intervals will contain the true value. It says nothing about this specific interval, and, more importantly, it does not place a probability distribution on the true value, so it cannot make claims about where it is. Also, when computing a confidence interval, it is not necessarily obvious where to place the endpoints (in assymetric distributions, for example, you can either have the smallest possible confidence interval or the smallest confidence interval centered on the mean, and they are usually very different). A bayesian confidence interval (credible interval), on the other hand, says that, assuming your prior and likelihood are correct, the posterior distribution for the true value has 90% of its mass inside the confidence interval. Of course, it rests of the assumption that your prior is correct. |
|
Confidence intervals are notoriously tricky for many people to interpret, and are often conflated with credible intervals, as Alexandre notes (e.g. in physics). I will add a few notes, mildly in defense of the frequentist use of confidence intervals, and trying to clarify the differing motives for confidence verses credible intervals. The key difference confidence intervals and credible intervals is that of acceptance of differing interpretations of probability: frequentist or Bayesian, respectively. The frequentist accepts confidence intervals as a legitimate construction around an estimator, given the nature of sampling variance and their limiting behavior. This need not be interpreted as the interval containing the true value 90% of the time given infinite repetition of the same experiment/sampling set-up, as Larry Wasserman explains in All of Statistics:
The Bayesian has the license (under their assumptions), to encode prior information into their credible intervals, and thus speak directly to properties of the estimated value. One may generally see frequentism as a more pessimistic approach that tries to derive its methodology from relatively domain independent results, such as sampling convergence, to statistical problems, whereas Bayesians allow expertise and other domain specific knowledge to speak directly to the same problems. 1
I think an ofter overlooked problem about confidence intervals (that fits your answer better) is that they almost always encode parametric assumptions about the data (except for some bootstrapped confidence intervals) that are ignored by many practicioners. We've all seen confidence intervals over probabilities (or error rates) using the t statistic or gaussian distribution, for example, and this makes no sense. In general, whenever the distributions don't look gaussian it is usually a bad idea to just report a confidence interval computed by the t statistic or gaussian distribution.
(Jul 07 '10 at 13:37)
Alexandre Passos ♦
Good points, Alexandre!
(Jul 07 '10 at 13:48)
John L Taylor
|
|
Another approach to confidence intervals that may help is Shafer and Vovk's "Tutorial on conformal prediction" [PDF]. In an online setting they show how one can transform any standard prediction algorithm into a "conformal predictor" – one that takes in an observation and a confidence parameter ε and outputs a set of predictions such that the set contains the correct prediction with probability 1- ε. The book that the tutorial is based on, Algorithmic Learning in a Random World, has some good discussions about confidence intervals and their various interpretations. |
|
An illuminating way to consider the difference between credible intervals and confidence intervals is to considering extreme case reasoning. We can construct a confidence interval by having it enclose the whole real line 95% of the time and a single point 5% of the time; choosing between the two options randomly and ignoring the data. This procedure produces a confidence interval with 95% coverage. Credible intervals are more along the lines of advising a 19:1 bet that the true parameter is inside the interval. The true parameter will be inside the credible interval 95% of the time under draws from the prior. If the true parameter is in a high probability region of the prior you will get better than 95% coverage and worse if the true parameter is in a low probability mass region of the prior. That said, if you interpret a confidence interval as a 19:1 bet you will often perform worse than a credible interval even with prior mis-specification. For example, in the case of estimating the mean of Gaussian the credible interval will have a lower average risk in 19:1 bets than the standard confidence interval even if the standard deviation of the prior on mean is off by ~50x. |
