Someone who wishes to remain anonymous points to a new study of David Yeager et al. on educational mindset interventions (link from Alex Tabarrok) and asks:
On the blog we talk a lot about bad practice and what not to do. Might this be an example of how to do things? Or did they just get lucky? The theory does not seem any stronger than for myriad other too-noisy-to-say-anything studies.
My reply: Hey, I actually was involved in that project a bit! I don’t remember the details but I did help them in some way, I think at the design stage. I haven’t looked at all the details but they seemed to be doing all the right things, including careful measurements, connection to theory, and analysis of intermediate outcomes.
My correspondent also asks:
Also, if we need 65 random schools and 12,000 students to do a study, I fear that most researchers could not do research. Is it pointless to do small studies? I fear they are throwing out the baby with the bathwater.
My reply:
You don’t need such a large sample size if you collect enough data on each individual student and class. Don’t forget, you can learn from N=1 in a good qualitative study—after all, where do you think the ideas for all these interventions came from? I do think, though, that the sloppy-measurement, small-N study is not such a good idea: in that case, it’s all bathwater with no baby inside.
What we really need are bridges between the following three things:
-
Qualitative research, could be N=1 or could be larger N, but the point is to really understand what’s going on in individual cases.
-
Quantitative research with careful measurement, within-person comparisons, and large N.
-
The real world. Whatever people are doing when they’re not doing research.
The gaps between 1, 2, and 3 are just too large.