Donny Williams writes:
I have a question I have been considering asking you for a while. The more I have learned about Bayesian methods, including regularly reading the journal Bayesian Analysis (preparing a submission here, actually!), etc., I have come to not only see that frequency properties are studied of Bayesian models, but it is the norm in more statistical papers that I have read. As I am in psychology, why do you think Bayesian psychologists are so averse to this notion? It would seem that a reasonable expectation for new method (to the field at least) is for an evaluation under repeated use. Part of me thinks it is because they brought to psychology the approach of the mid 1900’s such as Jeffreys’ and, as a result, we are having decades-old debates in psychology. Furthermore, from reading more methods oriented journals, it is not so clear to me that much of the Bayes factor advocacy could have been published elsewhere, as it was more here is what we say Bayes can do but never demonstrated it, even when we know the truth (as in simulation). Do you have any thoughts here?
My reply:
As a non-psychologist, I’m probably not the best person to opine on the psychology of psychologists’ choice of psychological methods. It does seem that some in psychology (and, before that, sociology) have rediscovered a naive enthusiasm for Bayes factors, which I do find frustrating. That said, just about any method can be useful, depending on how it’s used, so at some point you just have to do your best and move on.
Regarding the question of frequency properties of statistical methods: Yes, this is a good idea but you have to do it right. Over the past few decades I’ve seen a few zillion papers (and written a few of my own) featuring mind-numbing tables and graphs of frequency evaluations of statistical methods using simulation studies to show bias, variance, mean squared error, and coverage rates of various procedures. It’s often not clear what to do with such results, as any method should perform best when its assumptions are satisfied. This is not to say that simulation studies and frequency evaluations are useless, just that I think they often should be more focused. This paper for example features a very simple frequency analysis, not a big grid of simulations but I think it’s helpful in context.