[image of cat with a checklist]
Paul Cuffe writes:
Your idea of “researcher degrees of freedom” [actually not my idea; the phrase comes from Simmons, Nelson, and Simonsohn] really resonates with me: I’m continually surprised by how many researchers freestyle their way through a statistical analysis, using whatever tests, and presenting whatever results, strikes their fancy. Do you think there’s scope for a “standard recipe,” at least as a starting point for engaging with an arbitrary list of numbers that might pop out of an experiment? I’ll freely admit that I am very ignorant about inferential statistics, but as an outsider it seems that a paper’s methodology often gets critiqued based on how they navigated various pratfalls, e.g. some sage shows up and says “The authors forget to check if their errors are normally distributed, so therefore their use of such-and-such a test is inappropriate.” It’s well known that humans can really only keep 7±2 ideas in their working memory at a time, and it seems that the list of potential statistical missteps goes well beyond this (perhaps your “Handy statistical lexicon” is intended to help a bit with regard to working memory?) I’d just wonder if there’s a way to codify all relevant wisdom into a standard checklist or flowchart? So that inappropriate missteps are forbidden, and the analyst is guided down the proper path without much freedom. How much of the “abstract” knowledge of pure statisticians could be baked into such a procedure for practitioners? Atul Gawande has written persuasively on how the humble checklist can help surgeons overcome the limits of working memory to substantially improve medical outcomes. Is there any scope for the same approach in applied data analysis?
My reply:
This reminds me of the discussion we had a few years ago [ulp! actually almost 10 years ago!] on “interventions” vs. “checklists” as two paradigms for improvement.
It would be tough, though. Just to illustrate on a couple of your points above:
– I think “freestyling your way across a statistical analysis” is not such a bad thing. It’s what I do! I do agree that it’s important to share all your data, though.
– Very few things in statistics depend on the distribution of the errors, and if someone tells you that your test is inappropriate because your error terms are normally distributed, my suggestion is to (a) ignore the criticism because, except for prediction, who cares about the error term, it’s the least important part of a regression model; and (b) stop doing hypothesis tests anyway!
But, OK, I’ll give some general advice:
-
What do practitioners need to know about regression?
-
See the advice on pages 639-640 of this article.
I hope that others can offer their checklist suggestions in the comments.