Dale Lehman sends along a paper, “The ballot order effect is huge: Evidence from Texas,” by Darren Grant, which begins:
Texas primary and runoff elections provide an ideal test of the ballot order hypothesis, because ballot order is randomized within each county and there are many counties and contests to analyze. Doing so for all statewide offices contested in the 2014 Democratic and Republican primaries and runoffs yields precise estimates of the ballot order effect across twenty-four different contests. Except for a few high-profile, high-information races, the ballot order effect is large, especially in down-ballot races and judicial positions. In these, going from last to first on the ballot raises a candidate’s vote share by nearly ten percentage points.
Lehman writes:
This is related to a number of themes you have repeatedly blogged about. While I have not checked the methodology in detail, among the points I think are relevant are: – If the order listed on the ballot can have such a large impact, how does that jibe with statements you have made concerning unbelievably large claims of influences on voting behavior (shark attacks, etc.)? – If a Bayesian analysis were being conducted, what would the prior look like? It seems that the issue has been researched before, but this study seems to find substantially larger effects – but based on an arguably better sample and better methodology. Should we raise the burden of proof for this study based on prior results? – The study uses null hypothesis significance testing throughout. While I generally agree with your criticisms of that methodology, it is not clear to me why/how it is inappropriate in the attached study. In other words, it seems to me that the null of no ballot order effect might be a reasonable way to proceed.
My reply:
It definitely makes sense that ballot-order effects are larger in minor races. Indeed, last year I assessed a claim by political scientist Jon Krosnick that Trump’s 2016 election was determined by ballot order effects in Michigan, Wisconsin, and Florida. For ballot order to have made the difference, it would’ve had to have caused a 1.2% swing in Florida. My best guess is that the effect would not have been that large. However, in the absence of ballot order effects, I think the election would’ve been much closer, so it’s fair enough to include ballot order among the several effects that, put together, made the difference in that close race.
To return to the Grant paper under discussion: Just at a technical level, there are some problems (for example: “Because ballot order is randomly determined, the most natural approach for analyzing the data would appear to be analysis of variance, with ballot order being the “treatment.” However, because vote shares are proportions, the assumptions for an analysis of variance are not met.”). But Grant ended up running a regression, which would be the standard analysis anyway, so we can set aside that digression. In addition, I think null hypothesis testing is irrelevant here, because nobody doubts there is some nonzero ballot order effect. The only question is how large is it in different elections; there’s no interest in testing its pure existence.
What I really want to see are some scatterplots. Without any sense of the data, I have to put more trust in the analysis. And I was starting to get confused about some of the details here. With a randomized treatment effect, I’d think you’d start by just regressing vote share for each candidate on ballot order, displaying those coefficients, then going from there. Maybe a hierarchical model or whatever. Actually what they did was “A separate regression is estimated for each candidate but one; we adopt the convention of omitting the candidate who received the fewest votes.”—I can’t figure out why they didn’t do it with all the candidates. Also this: “This system is estimated with seemingly unrelated regression, to account for those inter-candidate correlations in popularity that are not captured by our controls”—maybe this is a good idea, I have no idea. And this: “the weight equals one half the base-10 logarithm of the number of voters”: this might work, but why the “one half”? That won’t affect the weights.
Anyway, my point here is not to bash the paper. I don’t really know enough to say whether a ballot order effect of 10 percentage points is plausible or not. As I said, it’s hard for me to evaluate a claim when the analysis is complicated and I can’t see the data. I expect that others who are interested in pursuing the topic will be able to try out different model specifications and explore further.