In Mackay's paper "Good Error-Correcting codes based on Very Sparse Matrices" (available at http://www.inference.phy.cam.ac.uk/mackay/abstracts/mncN.html) Appendix C the author explains how he created the error bars on his simulation plots. I will paraphrase:
The plots are the result of, for each of a set of channel parameters, running n trials and detecting r coding failures. Then the estimate of probability of failure is r/n and if r>0 the confidence interval for p is [p-,p+] where sigma=((n-r)/(nr))^(1/2) and p+/-=(r/n)exp(+/-2sigma). The plots are given as log-linear plots (log(probability of failure) on the vertical axis, channel parameter on horizontal axis).
I don't know where these formulae come from. It seems that the distribution of the probability of failure is binomial and the wikipedia page for Binomial proportion confidence interval has a somewhat similar formula but replacing p by r/n doesn't give me Mackay's formula.
Hopefully this question is straightforward and someone can explain how to derive confidence intervals to me. Let me know if you have any clarifying questions.
