Teppo Felin sends along this article with Mia Felin, Joachim Krueger, and Jan Koenderink on “surprise-hacking,” and writes:
We essentially see surprise-hacking as the upstream, theoretical cousin of p-hacking. Though, surprise-hacking can’t be resolved with replication, more data or preregistration. We use perception and priming research to make these points (linking to Kahneman and priming, Simons and Chabris’s famous gorilla study and its interpretation, etc). We think surprise-hacking implicates theoretical issues that haven’t meaningfully been touched on – at least in the limited literatures that we are aware of (mostly in cog sci, econ, psych). Though, there are probably related literatures out there (which you are very likely to know) – so I’m curious if you are aware of papers in other domains that deal with this or related issues?
I think the point that Felin et al. are making is that results obtained under conditions of surprise might not generalize to normal conditions. The surprise in the experiment is typically thought of as a mechanism for isolating some phenomenon—part of the design of the experiment—but arguably is it one of the conditions of the experiment as well. Thus, the conclusion of a study conducted under surprise should not be, “People show behavior X,” but rather, “People show behavior X under a condition of surprise.”
Regarding Felin’s question to me: I am not aware of any discussion of this issue in the political science literature, but maybe there’s something out there, or perhaps something related? All I can think of right now is experiments on public opinion and voting, where there is some discussion of relevance of isolated experiments to real-world behavior when people are subject to many influences.
I’ll conclude with a line from Felin et al.’s paper:
The narrative of blindness and illusion sells, and therefore continues to be the central thesis of popular books written by psychologists and cognitive scientists.
I’m reminded of the two modes of reasoning in pop-microeconomics: (1) People are rational and respond to incentives. Behavior that looks irrational is actually completely rational once you think like an economist, or (2) People are irrational and they need economists, with their open minds, to show them how to be rational and efficient.
They get you coming and going, and the common thread is that they know best. The message is that we are all foolish fools and we need the experts’ expertise for life-hacks that will change our lives.
If we step back a bit further, we can associate this with a general approach to social science, or science in general, which is to focus on “puzzles” or anomalies to our existing theories. From a Popperian/Lakatosian perspective, it makes sense to gnaw on puzzles and to study the counterintuitive. The point, though, is that the blindness and illusion is a property of researchers—after all, the point is to investigate phenomena that don’t fit with our scientific models of the world—as of the people being studied. It’s not so much that people are predictably irrational, but that existing scientific theories are wrong in some predictable ways.