The effects of political disappointment

Testing how we feel about each other when democracy doesn't go our way, a work in progress.

On the night of the 27th of October 2024 I woke up with a start at 2am. It had to be now! Not many things in research have this kind of urgency, but I realised at that point that I needed to act. Not in the morning, but NOW, in the middle of the night.

I was staying with a friend in Brussels, sharing the spare room with his extensive collection of classic sci-fi. No time to be distracted by that collector’s edition of The Earthsea Trilogy, my experiment couldn’t wait. There was no desk in the spare room, so I sat on the floor and opened my laptop to the project I was working on: expts/faithinreason_v2

To explain how I got there, and where it led, let me back up a bit.

§

The big idea was one that I’d been itching at for a while. What do we think about the rational powers of our fellow citizens? Do we think they are easily fooled? Or do we trust them to make reasonable choices, even when we disagree?

For any democrat, this matters. If you don’t believe in the reasonableness of others, there is no cause to try evidenced persuasion. If we give up on persuasion as a method to get agreement between ourselves and others, then we are left with either force or trickery. What we believe about other people is a fundamental question for deciding how society should be governed.

This topic of human reasonableness is one for which many commonly expressed views contain a latent anti-democratic attitude. If we claim that the majority in an election were fooled by digital advertising, or driven by ignorance, or stupidity, or otherwise conned into voting “the wrong way”, we risk both ignoring the reasons people had for voting differently from us, and downplaying the very fact that there is a space of reasons in which our own view is just one of many possible legitimate positions.

You hear many supposed progressives speak about people who disagree with them - sometimes even the majority of their country - as if they are irredeemably stupid, ignorant or morally corrupted. I worry about where the anti-democratic strain in these sentiments leads.

§

Previously, I proposed creating a simple scale to measure people’s belief in the reasonableness of others (paper, substack post). We’d found that people’s belief in the reasonableness of others was associated with a sense that other people are in general trustworthy (no surprise), but not with the sense that democracy was essential for a good society. It’s hard to know how to interpret this - is it a failure to find a meaningful connection, rather than a meaningful lack of connection? Perhaps we asked the question in a bad way meaning we missed a real association which would manifest if we had more sensitive measures.

With both associations - with generalised trust and with need for democracy - the age-old limitation of correlation-is-not-causation applies. We don’t know if people who trust more are then led to have more faith in the reasonableness of others, or if faith in the reasonableness of others drives more trust. One could cause the other, or the other way around, or both could be influenced by something else.

A host of other questions also buzz around when you invent a new measure like this. Is faith in reason something stable, something people either have, or not, like personality traits or fundamental convictions? Or is it more labile, something more like state of mind, or mood, which adjusts to every little thing (including how the questions are asked)?

These are tricky questions to answer.

§

When researchers want to know about causation their greatest ambition is to run an experiment. If you change X and Y changes, this is evidence about how X changes Y. Of course, you can only do this when it is possible, or ethical, to make changes to whatever X is.

If I’m interested in people’s faith in reason and democratic politics, how can I make any intervention which affects either of those things? I don’t have the power to change the nature of democracy just so I can see how it affects people’s beliefs about others (even if it was ethical).

For situations like these, social scientists look for natural experiments - situations where something changes and as-if-at-random some are affected and some are not.

Could I find a natural experiment where half the population were disappointed in the choices of their fellow citizens, letting me test the effect on their faith in reason?

§

Sometime in early 2024 I realised that the (then) upcoming US presidential election could be such a possibility.

It might be hard to recall, but up until the vote the 2024 election was widely regarded as too close to call¹. The US electorate is famously split between two parties, which define an (increasingly) polarised political landscape, with only statistically marginal action for third parties.

This provided a balanced situation of near-perfect ambiguity before the 2024 election. Nobody knew what was going to happen. I could predict that, whatever happened, approximately half of voters would be disappointed with the outcome (and their fellow citizens). On top of this, approximately half of anyone who believed they knew who would win would be surprised.

So this was my plan - rather than look for a natural experiment that had happened in the past, I would do a planned natural experiment, measuring US voters’ faith in reason before and after the disappointment-generating (and ambiguity-resolving) event of the presidential election of 5th of November 2024.

The core prediction was that political disappointment would reduce people’s generalised faith in the reasonableness of others, but the experiment would also let me test how stable my measure of faith in reason was, how the impact of events moved responses around, and if they interacted with the confidence of people’s predictions for the election (and whether they were right or wrong).

§

To make this work, I needed to sort out a few things.

I needed to be able to ask questions of US voters before the election, and recontact them afterwards. Participant drop out is a major headache for survey research - it is easy to find online participants, but linking responses from the same individuals across multiple time points is a technical headache. I knew that the research participation platform Prolific provided this functionality. To incentivise participants to come back I promised to pay a large fee, plus a bonus, for taking the post-election follow up. I used the bonus to get predictions from participants about the election outcome. Participants also revealed their implicit confidence in their prediction by opting into different bonus options:

Using this method I got a 98% return rate for two surveys - one a week before the election and one a week after. The choice about bonus payment also incentivised participants to report what they really believed would happen - this isn’t just idle talk, they are risking money on the table.

As well as solving the recruitment and re-recruitment issue, I also needed to settle on a short set of questions I would ask my participants, adapting and improving version 1 of the scale. I coded all my questions into the qualtrics platform, which let me do nice things like randomise the order of some questions and links up with Prolific nicely.

The revised scale items. The faith in reason score is the average across all questions (where higher is more faith in reasonableness of other people).

Next, I wanted to preregister my experiment. What’s the point of making a cool prediction if you can’t demonstrate that you made the prediction before you knew the result?

It was this need to preregister that woke me up at 2am on 28th of October. The election was on 5th of November, I wanted to collect data the week before, 29th of October (then come back a week after, 11th of November).

If I was going to do data collection on the 29th I needed to have my predictions preregistered on the 28th. I knew I was going spend all day busy, so I had to make time to record my predictions in the middle of the night.

Predictions made², I was ready to collect the data, with just one unresolved question: does the idea of a planned natural experiment even make sense? The election outcome is far from random, although no individual could know the outcome (however much they believed they did). The nature of politics also means we can’t infer that the winning side would have reacted the same way as the losing side, in an alternative universe where the election came out differently. Maybe, fundamentally, Democrats and Republicans think about the rationality of the citizenry differently? It’s not implausible.

Regardless, at that point in October 2024 I knew that soon half of my participants were going to be disappointed by the choice of their fellow citizens. I was excited to find out what this did to their faith in reason.

§

We all know how the 2024 presidential election turned out. Republican Party voters got what they wanted, Democratic Party voters were disappointed. Using the appropriate colour coding we can visualise the before-after effect on the Faith in Reason scale responses, Republicans on the left, Democrats on the right:

Of 501 participants³, 213 reported voting Democratic, 174 voting Republican, and 113 neither party (not shown here). The y-axis shows the faith in reason score for each voter. Each voter has two scores, before and after. I’ve jittered the x-axis position so individual points are more visible, creating columns of data. The black horizontal lines show the middle value (medians) for each group.

A few things to note: There are individuals on both sides who have very high or very low belief in the reasonableness of other people. Supporters of both parties are more similar in their beliefs about the reasonableness of others than they are different.

Before the election, Republicans had a slightly higher average score than Democrats. I wouldn’t read this as a meaningful superiority, but it does at least speak against the idea that Republicans have less regard for the reasonableness of others than Democrats (and maybe to the idea that Democrats may be at risk of having lower such regard than Republicans).

Post-election, the average Democratic faith in reason dropped. Using the full scale on the graph y-axis makes the change seem small, but in standardised effect size terms this is a large (d = 0.77) effect, and a solidly statistically significant one. So that’s the answer to my original question - political disappointment does knock your belief in the reasonableness of others.

And it is possible to use a national political event to meaningfully assess the causal impact on a survey measure of political-psychological attitudes.

Looking at participants’ predictions, if you split voters by their confidence in the election outcome it is the Democrats who were most confident in their prediction that had the largest drop in scores post election.⁴

Size of the before-after difference in faith in reason score, by prediction confidence (= bonus option selected at stage 1)

Here you can see the Republicans (red) didn’t change their beliefs about the reasonableness of others, regardless of their confidence in the election outcome. For the Democrats, though, those with most confidence - willing to select the bonus option which paid out $9 if their prediction was right, but only $1 if they were wrong - whose faith in reason score decreased the most between before and after the election.

Combined, it seem the combination of disappointment and surprise knocks our faith in the reasonableness of others hardest.

How would Republicans have reacted if their candidate had lost the election? We can’t know that, but it is notable that, tested the week before the election, Republicans had confidence in their fellow citizens powers of reasoning which was as great as Democrats, if not greater.

§

With this study, the disappointment isn’t just with those voters for whom the election didn’t turn out how they wanted, I’m also furiously disappointed with myself for not writing up the study. I planned and conducted the study in a rush, driven by the deadline of the election. Since then I’ve written a draft of a formal report, but haven’t been able to finish it. I wanted to preprint on the anniversary of the election, but that has come and gone. Part of this is that other projects have taken over, but part is that I find, now I’ve done the study, I am not sure what I make of it.

The study refines the Faith in Reason scale, showing that it is built on a coherent set of items, has a lot of stability across time, as well as being reactive to events. The use of the election is neat, and suggests that counter-forces are necessary, if the democracy-sustaining faith in reason is to survive across time - after all, in every election some of the population have to lose. The Republicans in my study didn’t increase their faith in reason as much as the Democrats decreased it, suggesting the election was a net loss for the belief in the reasonableness of others.

§

I’d love to hear what people think of this work. You can download the draft report here, or shoot me an email and I’ll send it over. If you have comments, please leave them below this post or hit reply. If you think the result is obvious, or interesting, or if there’s something else you’d like to know about the data, please let me know.

---

This newsletter is free for everyone to read and always will be. If you can afford it, feel free to chip in the help me keep writing by upgrading to a paid subscription (more on why here)

Keep reading for further reading and other things I’ve been thinking about.

---

Version 1 of the Faith in Reason scale:

• Stafford, T., Zhu, J., & Dommett, K. (2024, December 31). Quantifying belief in the rationality of others: the Faith in Reason scale. Working Paper. https://doi.org/10.31234/osf.io/umwxj

Version 2 (and the results in this post) are partly written up here

• Stafford. T. Belief in the reasonableness of others: the 2024 US presidential election as natural experiment. https://osf.io/6r8se/files/wf574

Other stuff….

The Gambler’s Fallacy

Last time I mentioned the Gambler’s Fallacy, Kevin Dorst has a nice explanation of why this is the behaviour you would expect from a rational actor

Kevin Dorst: Bayesians Commit the Gambler’s Fallacy

Me: Good bias

Catch-up

Also by me recently:

• Gambling with research quality How you get 244 different ways to measure performance on the same test of decision making. And what it means for the reliability of behavioural science

• Model collapse. when AI alters our behaviour and our institutions, the risk isn’t just theoretical

• The half life of trust. Or: You trusting something isn’t enough for me to trust it

Effect sizes

Methods fiends and reproducibility mavens are recommended to check out Ian Hussey’s pubpeer comment on a study with implausibly high effect sizes (of up to d = 22).

Is a standardised effect size of d = 22 a lot? Ian explains:

effect sizes of 22 are not even seen in manipulation checks or for obvious preferences such as “Chocolates are more desirable than human poop” (Cohen’s d = 4.52: Balcetis & Dunning, 2009) .

Writing

Thanks to @codingconduct for this:

And finally…

This cartoon requires you to know the Judy Blume reference, but when you do …

via @csilverman

END

Comments? Feedback? Disappointments? I am tom@idiolect.org.uk and on Mastodon at @tomstafford@mastodon.online

1. “The forecast remains extremely stable and extremely close to 50/50” (Election forecaster Nate Silver, October 31, 2024), “Really as close of a race as you can possibly get” (Lakshya Jain of Split Ticket, October 30, 2024), “A perfect coin toss” (Owen Winter of The Economist, October 30, 2024).

2. Timestamp 2:52 am. There are difference schools of thought around preregistration. One is that you should fully specify all intended analyses. This preregistration does not follow that school. I believe it is still valuable to articulate and commit what you intended before you did it, to the degree you are able to, even if not all details of a future analyses are known.

3. I originally planned to collect 1400 participants, but after doing the preregistration I realised that the bonus mechanism created risk that I would exceed my budget. If I recruited 1400 people, and enough of them choose the high bonus option, and the election went the way they expected, then I wouldn’t be able to pay everyone, so to reduce my exposure I had to recruit fewer people at stage 1.

4. Interestingly, this association between confidence and the size of the effect of the election did not hold for self-reported confidence - it only holds when confidence is measured by which bonus option participants selected, not in how confident they said they were.

Also, this plot drops the small minority of party supporters who predicted that the opposition would win.