1) Stan
In my own work, I have bounced around between various off-the-shelf samplers and needing to write my own for certain special-purpose models. But Stan made it so easy to implement big multivariate distributions and rather complex link functions (the Wiener first passage time distribution) and NUTS explores the posterior in such an efficient manner that I find it hard to imagine I will go back to other programs or custom samplers any time soon. We might have been able to do the work without Stan, but we would have had to either reduce the scope of our project or rest uneasy in the worry that our posterior samples were just a cloud of dust. Stan makes me feel that we can rest easy.
2) Multiverse analysis
Like anyone who analyzes data, particularly in an exploratory setting (see below), we tried a bunch of different things before settling on the stuff in the main text. I admit, we originally were not going to talk about all those other tries in the paper until the editor asked about them, but I’m very glad he (the editor) did! For context, our analyses involved selecting a certain number of principal components to focus on, so that we could reduce the complexity of the data to something we might have a chance of understanding (and communicating to a reader), but how was this choice made? While hardly gripping reading, Appendix D shows the results of different choices, and even the results that would occur if individual tasks were dropped out (as if we had not measured them). The result is that a reader has more information to make a reasoned judgment about our analyses, since they now have a clearer picture of why we thought the main one was most useful and also how much the conclusions might change as a function of different choices or different observations.
3) Open data exploration
The original motivation for this study, for which credit goes to Pernille Hemmer and Amy Criss, was to produce an openly available set of data across these various tasks to aid the development of memory theory, so really this whole project was “born free”. We have tried to adhere to this ideal as best we can, putting out our data and code on the OSF, doing so as soon as we submitted it for publication. We know that there is a lot more detail in these data than we have analyzed so far, but we are finite creatures. By making all of our data/methods available, we don’t have to do all the work ourselves, so this is a benefit to us–someone with a good idea can just do it and contribute immediately without going to the trouble of collecting another 450 participants. And obviously it is a benefit to those folks with good ideas since now they can test and refine those ideas much more easily. As we say in our discussion, “we do not expect this will be the final word with regard to this dataset or the tasks represented therein.”
But while I think our work is an example of a number of good research practices, I am sure that there are things we could do better . . . As a final remark, many psychologists/cognitive scientists like myself feel frustrated that the public face of our field is one that is pocked with a mixture of tabloid silliness, deliberate ignorance, and outright fraud, as discussed often on your blog. Thus I hope that our work, as imperfect as it is, can also serve as a reminder that there have been a lot of us—going back to the birth of scientific psychology with Helmholtz, Wundt, etc.—that have been diligently trying to understand how minds work through careful experiments, rigorous analyses, and well-developed theories. We may not be the loudest voices in the room, but we were always there and always will be.