Our intuitions about political advertising are poor

Notes on a great interview with David Broockman

This is great: The Realities of Political Persuasion with David Broockman, from the always listenable Opinion Science by Andy Luttrell.

Broockman argues that, in the context of campaigning, too much emphasis has been put on message personalisation - identifying particular segments of voters and designing ads which particularly appeal to them - and not enough on empirically testing message effectiveness. You can understand why - for consultants and strategists personalisation seems like the kind of magic which might allow big breakthroughs, and there are a range of ways of categorising people from demographics or psychology which superficially seem promising.

Broockman’s research pours cold water on this idea of personalisation. This fits with what I’ve learnt - when I’ve looked at studies which purport to show that advertising according to personality is more successful, there are either major confounds, or - when you look closely - the personalisation may arguably have an effect, but the size of the effect is not significantly larger than a non-personalised message.

Important context for this is that there are big differences between individual adverts in terms of effectiveness. This means that better and worse adverts could be deployed in terms of persuasive effect - the variance is there, if we can work out how to take advantage of it (and if we did, it wouldn’t threaten our vision of human reasonableness, because microtargeting is not mind control).

The point is that personalisation is, in practice, very very far from being some kind of terrifying ‘manipulation machine’. The difference between different ads is typically far larger than between a personalised and non-personalised version of an ad. I try and show this by reworking the data from this paper into a summary visualisation:

To understand this plot you just need to know that the up-down/vertical axis shows the persuasive effect. The green line, as it changes left to right, shows the difference between individuals in how persuasive they found the ads - a huge difference. The change in the vertical position of the orange triangles shows the differences between different adverts used in the study - a smaller difference, but still much much bigger than the difference due to personalisation (blue dots). The full context is in this post: AI-juiced political microtargeting. The point is that a non-zero effect of personalisation (the highest blue dot is higher than the lowest blue dot) can still be non-important compared to the other sources of variation (the differences due to advert or individual are far larger).

This is from a paper which says in the abstract : “Recent technological advancements, involving generative AI and personality inference from consumed text, can potentially create a highly scalable “manipulation machine” that targets individuals based on their unique vulnerabilities without requiring human input.”

The “highly scalable” part is definitely true, but I question the degree to which this is manipulation, or to which it is worth worrying about given the size of the demonstrated effects.

From the other side of the coin, the implication is that political campaigners should worry far more about being persuasive for everyone than trying to take advantage of minuscule fine-tuning that might be possible for different groups.

In the Opinion Science pod, Broockman goes on explain that despite the large differences between different messages these differences are hard to predict. They aren’t predicted by the categories that political scientists typically use, such as whether the ads are positive or negative in tone.

Further, expert political practitioners are not better at predicting the most persuasive adverts compared to the general public. Not only were the experts not better than the public at predicting which campaigning ads would be most persuasive, they weren’t consistent with each other (which is important because it suggests there isn’t even some kind of shared conventional wisdom which, although wrong, is shared by the experts).

I have a memory of a study done of the 2016 Democratic campaign which shows that the ads predicted to be most persuasive by campaign staffers were least persuasive with the voters who most needed to be swayed. Sadly I can’t find this now, but I did find this very telling article from pre-election 2016: How the Clinton campaign is slaying social media. This gloating piece neatly illustrates the potential mechanism - highly political, young, cosmopolitan and progressive campaign staffers (and journalists) mistake what works for them as what works for the generally less political, less young, less cosmopolitan and less progressive swing voters.

My take-away is that our intuitions in this area are bad.

We’re bad at discounting our own reaction to an ad to work out how it will affect people unlike us (meaning that campaigners probably leave a lot of variation in persuasive effect on the table when deciding which ads to deploy).

We’re drawn to think that microtargeting and personalisation will unlock extra persuasive power (when the evidence is that extant models of personalisation are not strongly effective, and the degree to which they are is probably in line with how normal persuasion works - information containing facts and evidence is persuasive, just as we’d hope).

And we’re constantly tempted to think of persuasion in general, and advertising in particular, as a form of manipulation. This is extremely limiting. It implicitly denies legitimacy to electoral campaigning, which is a core part of democracy, and denigrates the voters who are persuaded.

Link: The Realities of Political Persuasion with David Broockman

Paper: Broockman and colleagues: Political practitioners poorly predict which messages persuade the public.

Related, from me:

• Language models are persuasive - and that’s a good thing

  Two new studies provide insights into exactly how LLMs persuade, and what that means.

• The truth about digital propaganda

  Reasonable People #55: Our piece in New Scientist bring evidence to worries about online manipulation

• AI-juiced political microtargeting: Reasonable People #53 looking carefully at the claims in one study which uses generative AI to customise ads to personality type

• Propaganda is dangerous, but not because it is persuasive

  Reasonable People #52: I pick at the claim that propaganda “doesn’t work”.

• How persuasive is AI-generated propaganda? Reasonable People #51: Bullet review of a new paper suggesting LLMs can create highly persuasive text and will supercharge covert propaganda campaigns.

• Microtargeting is not mind control

  Reasonable People #22 exaggerated beliefs about the effectiveness of microtargeted ads obscure real risks, and real opportunities to foster public trust in politics

---

This newsletter is free for everyone to read and always will be. To support my writing you can upgrade to a paid subscription (more on why here)

Keep reading for more on fact-checking, AI, and a very cool job opportunities.

---

In case you missed it

I’m spending more time in the UK capital. Let me know if you want to get coffee :

London calling / Tools for Thought

PODCAST: Gordon Pennycook on Unthinkingness, Conspiracies, and What to Do About Them

From Sean Carroll’s Mindscape, an interview with Pennycook about his work, including on pseudo profound bullshit and using chatbots to debunk conspiracy theories. Something which came out, which I missed from the papers and seems really important, is that people like the experience of being debunked by the chatbot.

And people usually actually like it.

They’re not mad at the AI.

The AI gives them information I think is useful.

And evidence matters more than we thought it was.

Link: Gordon Pennycook on Unthinkingness, Conspiracies, and What to Do About Them

JOB: Research Scientist at Wikimedia

We’re hiring a Research Scientist strongly committed to the principles of free knowledge, open source, privacy, and collaboration to join the Research team. As a Research Scientist, you will conduct applied research on the integrity of Wikipedia knowledge, its communities and their work, and the Wikipedia model.

Examples of recent research questions you may contribute to include::

How does the platform and its community navigate election times?

What is the role of Wikipedia in the landscape of online disinformation?

What guidance can we provide to researchers studying neutrality on Wikipedia?

Closes: January 15th, full remote with some geographic limitations

Link: https://job-boards.greenhouse.io/wikimedia/jobs/7484474/

….And finally

END

Comments? Feedback? Takes from 2016 which aged really badly? I am tom@idiolect.org.uk and on Mastodon at @tomstafford@mastodon.online