Javier Benitez points us to this article from 2014 by Susan Howitt and Anna Wilson, which has subtitle, “The way textbooks and scientific research articles are being used to teach undergraduate students could convey a misleading image of scientific research,” and begins:
In 1963, Peter Medawar gave a talk, Is the scientific paper a fraud?, in which he argued that scientific journal articles give a false impression of the real process of scientific discovery. In answering his question, he argued that, “The scientific paper in its orthodox form does embody a totally mistaken conception, even a travesty, of the nature of scientific thought.” His main concern was that the highly formalized structure gives only a sanitized version of how scientists come to a conclusion and that it leaves no room for authors to discuss the thought processes that led to the experiments. Medawar explained that papers were presented to appear as if the scientists had no pre-conceived expectations about the outcome and that they followed an inductive process in a logical fashion. In fact, scien- tists do have expectations and their observa- tions and analysis are made in light of those expectations. Although today’s scientific papers are increasingly presented as being hypothesis-driven, the underlying thought processes remain hidden; scientists appear to follow a logical and deductive process to test their idea and the results of these tests lead them to support or reject the hypothesis. However, even the trend toward more explicit framing of a hypothesis is often misleading, as hypotheses may be framed to explain a set of observations post hoc, suggesting a linear process that does not describe the actual discovery.
Howitt and Wilson continue:
There is, of course, a good reason why the scientific paper is highly formalized and structured. Its purpose is to communicate a finding and it is important to do this as clearly as possible. Even if actual process of discovery had been messy, a good paper presents a logical argument, provides supporting evidence, and comes to a conclusion. The reader usually does not need or want to know about false starts, failed experiments, and changes of direction.
Fair enough. There’s a tension between full and accurate description of the scientific process, on one hand, and concise description of scientific findings, on the other. Howitt and Wilson talk about the relevance of this to science teaching: by giving students journal articles to read, they get a misleading impression of what science is actually about.
Here I want to go in a slightly different discussion and talk about the ways in which the form of conventional science paper has been directly damaging science itself.
The trouble comes in when the article contains misrepresentations or flat-out lies, when the authors falsely or incompletely describe the processes of design, data collection, data processing, and data analysis. We’ve seen lots of examples of this in recent years.
Three related problems arise:
-
A scientific paper can mislead. People can read a paper, or see later popularizations of the work, and think that “science shows” something that science didn’t show. Within science, overconfidence in published claims can distort future research: lots of people can waste their time trying to track down an effect that was never there in the first place, or is too variable to measure using existing techniques.
-
Without a culture of transparency, there is an incentive to cheat. OK, a short-term incentive. Long-term, if your goal is scientific progress, cheating can just send you down the wrong track, or at best a random track. Cheating can get you publication and fame. There’s also an incentive for a sort of soft cheating, in which researchers pursue a strategy of active incompetence. Recall Clarke’s Law.
-
Scientific papers are typically written as triumphant success stories, and this can fool real-life adult scientists—not just students—leading them to expect the unrealistic, and to make it hard for them to learn from the data they do end up collecting.
P.S. One problem is that some people who present themselves as promoters or defenders of science will go around defending incompetent science, I assume because they think that science in general is a good thing, and so even the bad stuff deserves defending. What I think is that the people who do the bad science are exploiting this attitude.
To all the science promoters and defenders out there—yes, I’m looking at you, NPR, PNAS, Freakonomics, Gladwell, and the Cornell University Media Relations Office: Junk scientists are not your friends. By defending bad work, or dodging criticism of bad work, you’re doing science no favors. Criticism is an essential part of science. The purveyors of junk science are taking advantage of your boosterish attitude.
To put it another way: If you want to defend every scientific paper ever published—or every paper ever published in Science, Nature, Psychological Science, and PNAS, or whatever—fine. But then, on the same principle, you should be defending every Arxiv paper ever written, and for that matter every scrawling on every lab notebook ever prepared by anyone with a Ph.D. or with some Ivy League connection or whatever other rule you’re using to decide what’s worth giving the presumption of correctness. If you really think the replication rate in your field is “statistically indistinguishable from 100%”—or if you don’t really believe this, but you think you have to say it to defend your field from outsiders—then you’re the one being conned.