Colleen Flaherty asks:
Do you get asked to peer review a lot? I’m guessing you do… This new very short paper says it’s not a crisis, though, since only the people who publish the most are getting asked to review a lot… The authors pose two solutions: either we need to “democratize” the system of peer review OR we start thinking about it as a credit system, where you should basically expect to review three papers for every one you publish. Where would you fall on this? Or do you think there is a crisis anyway?
The paper in question is called “Are Potential Peer Reviewers Overwhelmed Altruists or Free-Riders?,” by Paul Djupe, Amy Smith, and Anand Sokhey, who write: “where does peer review in the social sciences stand? Are academics overburdened altruists or peer-review free-riders? Our new Professional Activity in the Social Sciences data set suggests the answer is ‘Neither.’ Instead, most academics get few peer review requests and perform most of them. . . . the peer review crisis may be overblown.”
My reply to Flaherty was to point her to this post from a couple years ago, “An efficiency argument for post-publication review,” and continue with this mini-rant:
Peer review is wasteful in that every paper, no matter how crappy, gets reviewed multiple times (for example, consider a useless paper that gets 3 reviews and is rejected from journal A, then gets 3 reviews and is rejected from journal B, then gets 3 reviews and is accepted in journal C). But even the most important papers don’t get traditional peer review after publication. I think post-publication review makes more sense in that the reviewer resources are focused on the most important or talked-about papers. The only argument I can see in favor of the current system of peer review is it makes use of the unpaid labor of thousands of people–and if the structure were re-created from scratch, it might be hard to get most of these people to continue to work for free.
To put it another way: the Djupe et al. paper has some interesting data, and it’s fine for descriptive purposes, but it’s hard for me to even think of the existing pre-publication peer review system without being reminded of how wasteful it is, and how often it is used to shield and justify bad work.
P.S. I get hundreds of peer-review requests a year. I say No to most of these requests—I have to, or I’d have no time for anything else—but I say Yes enough that I still review a lot.