James Coyne pointed me with distress or annoyance to this new paper, “Tutorial: The Practical Application of Longitudinal Structural Equation Mediation Models in Clinical Trials,” by K. A. Goldsmith, D. P. MacKinnon, T. Chalder, P. D. White, M. Sharpe, and A. Pickles. This is the team behind the PACE trial for systemic exercise intolerance disease.
Some of the words in the abstract seem ok:
Repeated measurements allow the application of various types of longitudinal structural equation mediation models. These provide flexibility in modeling, including the ability to incorporate some types of measurement error and unmeasured confounding that can strengthen the robustness of findings. . . . In this tutorial, we outline how to fit several longitudinal mediation models, including simplex, latent growth and latent change models. . . .
The trouble is that these good intentions don’t do anything for you if the models don’t make sense. It would be as if I tried to build a nuclear reactor at home and wrote a fancy-sounding prospectus with a lot of fine phrases about atomic theory and the forces on the nucleus. It might do the job to get my article published in the Lancet but it wouldn’t actually be producing nuclear power.
What’s striking here is that:
-
They’re giving a tutorial in mediation models, but they don’t seem to understand these methods, or at least be able to explain them clearly. For example, the above figure from this paper just seems like bad news. I can’t imagine it ever being a good idea to do this sort of thing.
-
They still won’t share their data! From the abstract:
We use the Pacing, Graded Activity, and Cognitive Behavioral Therapy: A Randomized Evaluation (PACE) trial of rehabilitative treatments for chronic fatigue syndrome (ISRCTN 54285094) as a motivating example . . . The simulated data set and Mplus code and output are provided.
I asked Coyne if these researchers said there were some legal or confidentiality rules whereby they could not share their data, and he replied: Yes but that was soundly rejected by a tribunal. So I don’t know whassup with that.
Psychological Methods is a respected journal, I thought!
P.S. There’s some discussion in comments about whether I (and, I suppose, Coyne) are being unfair by saying how we don’t believe these methods, but without giving detailed criticism. Maybe Coyne and I are wrong; that’s certainly possible. But the real point is the science. It’s not about me, or Coyne, or even the authors of this paper; it’s about understanding the effects of these medical treatments. And, the thing is, I don’t see any reason to think that the methods of mediation analysis discussed above will answer the applied questions of interest here. Real people are suffering from systemic exercise intolerance disease, and I think the burden is on these researchers to demonstrate that these complicated and controversial procedures make sense. It’s not our job to untangle these methods.