A statistician who works in the pharmaceutical industry writes:
I just read your paper (with Dan Simpson and Mike Betancourt) “The Prior Can Often Only Be Understood in the Context of the Likelihood” and I find it refreshing to read that “the practical utility of a prior distribution within a given analysis then depends critically on both how it interacts with the assumed probability model for the data in the context of the actual data that are observed.” I also welcome your comment about the importance of “data generating mechanism” because, for me, is akin to selecting the “appropriate” distribution for a given response. I always make the point to the people I’m working with that we need to consider the clinical, scientific, physical and engineering principles governing the underlying phenomenon that generates the data; e.g., forces are positive quantities, particles are counts, yield is bounded between 0 and 1. You also talk about the “big data, small signal revolution.” In industry, however, we face the opposite problem, our datasets are usually quite small. We may have a new product, for which we want to make some claims, and we may have only 4 observations. I do not consider myself a Bayesian, but I do believe that Bayesian methods can be very helpful in industrial situations. I also read your Prior Choice Recommendations [see also discussion here — AG] but did not find anything specific about small sample sizes. Do you have any recommendations for useful priors when datasets are small?
My reply:
When datasets are small, and when data are noisier, that’s when priors are more important. When in doubt, I think the way to explore anything in statistics, including priors, is through fake data simulation, which in this case will give you a sense of what is implied, in terms of potential patterns in data, from any particular set of prior assumptions. Typically we set priors to be too weak, and this can be seen in replicated data that include extreme and implausible results.