There’s a dilemma we face whenever we have to rely on someone’s judgement, and I think it tells us something important about the possibility of better, ‘unbiased’, decision making.
Imagine you are comparing teachers who mark student essays. Or imagine football scouts who come back with reports of promising young players. Or peer reviewers who score funding proposals, or two friends who both tell you about different films they saw last weekend. Any situation where you have multiple reports of multiple things.
Every judge will bring to their task their own sensibility, the particular combination of taste and insight that lets them perform the judging task. And every judged object, be it essay, young player, funding proposal or film, will be a beautiful unique snowflake, but, in addition to having a particular and irreducible combination of qualities, it will also be better or worse on the scale of quality for which it is being judged.
Which is to say that better and worse exist, even in subjective domains. I am no young Lionel Messi in terms of potential football talent. Chopper Chicks in Zombietown is no Citizen Kane in terms of the heights of cinematic artistry (imdb score 4.1 out of 10). Even if the nature of quality collapses to no more than “possessing the tendency to be judged better”, we can still assume there is some true scale of quality on which each judged object could be placed. Some footballer is the best bet for the club’s upcoming season, some essays should get top marks, and if your pal recommends you go to the cinema to see Chopper Chicks in Zombietown you should only go if you can equip yourself with the same drugs they had available while enjoying the film.
Here is the root of the dilemma — how should you weight opinions from different judges? Taking all judgements at face value risks the outcome being skewed by an outlier or miscalibrated judge. Aggregating the scores from different judges to correct for miscalibration risks throwing away the unique and valuable signal that each judge provides.
As well as bringing their particular combination of taste and insight to the task, each judge must also have some tendency to give high or low scores, to be generous or ungenerous. This is one of the senses in which we use the word ‘bias’.
You can level-out bias across judges, with an adjustment. In statistics this is called ‘normalisation’, which means you can put all judges’ scores on a common scale. Crucially, normalisation assumes there is, and should be, such a common scale.
Here’s how it might look. Judge A sees Objects O1, O2, O3 and gives scores of 35, 46 and 28. Judge B sees Objects O4, O5, O6 and gives scores of 86, 79 and 82. Like this:
Mathematically, we adjust them both so they have the same mean (the midpoint of the scores they gave will now be 0) and the same variance (the unit of spread for scores they gave will always be 1).
So their scores become -0.18,+1.30,-1.12 and +1.28,-1.16, -0.12. Like this:
Graphically, you can see they retain the same ‘shape’. We’ve just stretched and shifted the axes of the graph, so to speak.
Now we’ve done that, we have the appearance of consistency. Judge A’s 28 for O3 is now -1.12, slightly above Judge B’s 79 for O5 (which is now -1.16).
You can see the problem. The adjustment has taken away something. What if Judge A really did mean to award 28 out of 100? Object 3 is a stinker. The normalisation process is, in effect, declaring that Judge A is ungenerous, because they gave low scores, but maybe they just got dealt low quality objects to judge?
You can get around this by having one judge look at everything, or by having each judge see the same set of objects (or at least some common objects), or having each judge make a very large number of judgements (so you can assume that the set of objects they see won’t be especially good or bad just by chance). All of these are some flavour of inefficient or impractical. Teachers don’t want to mark work that is already marked, football scouts can only see so many young players play, etc.
Mathematically, there are ways of balancing the bias of the judges against uncertainty around the quality of the objects. All of these ways work best when fed by large numbers; many judges, many judgements, many judged objects. In a single school, talent academy, or funding agency they come up against their limitations.
There are also methods of circumventing the need for normalisation. You can train the judges, or give them better guidance on the scale, but to the extent you don’t do this perfectly, the dilemma persists. You can ask them to rank objects rather than score them (so every judge has to say one object of the set they see is 1st, one is 2nd, etc), which is a good solution, but is also limited when each judge only sees a small number of items (and throws out information. Because a judge has to pick a best object, they can’t indicate where what they’ve seen is truly exceptionally good, or just the least-worst of a bad bunch).
I do a lot of work with research funders who face exactly this dilemma. You can only ask any reviewer to evaluate so many proposals (and getting sufficient reviews is a chronic problem in the research system). Proposals for new research are lengthy, complex, documents. Each proposal is different, across many different dimensions (feasibility, potential, risk, novelty, etc). Yet each proposal must be scored against every other proposal because only a minority can be funded.
For what it is worth, some funders normalise the scores they get from different reviews. Most don’t.
From the perspective of standardisation, normalisation looks like the obviously correct option. If you don’t normalise then some students have their essays marked harshly, some young talent is unfairly overlooked and some funding proposals miss out due to the bad luck of which reviewers they are assigned. When I talk about funding proposal scoring with many people in psychology research, they are often shocked that funders usually don’t normalise (although not that shocked, most researchers accept that funding is something of a lottery). Maybe more students (or their parents) would be shocked if they knew about the lack of normalisation in marking exams and coursework.
I think, though, I understand why normalisation isn’t universal. Part of it is legibility. Judging has two functions: to deliver a score, and to legitimate that score. A process that isn’t trusted or accepted isn’t any good, even if the scores are correct.
The ingredients for this legitimacy are a curious blend of transparency and inscrutability. It is good that people understand the process (the rule “judges give marks on a scale which goes from 0 - 10” is very clear, for example). But it is probably also good that they don’t understand the process too well (“this judge didn’t eat breakfast, so is grumpy”; “this judge holds a grudge against the whole nation of Norway”, and so on). Some things in judging we just have to accept, and normalisation opens up a new front for scepticism.
If the judging process itself can overturn a judge’s score — turning a raw 28 into a -1.12 — then maybe we should be looking a bit harder at the factors internal to the judge that produced this raw score? A can of psychological worms!
Not-normalising may be statistically sub-optimal, but it is easy to explain. Judges give a score. That’s it. Normalisation, as well as opening up a new front for doubt in the scoring process, requires a small amount of statistical thinking, both to understand the rationale and the implementation. It is only a small amount, but that incurs an explanatory cost and so puts the legibility of the process, and so the legitimacy of the process, at greater risk.
There’s an additional, deeper, reason not to normalise. This statistical standardisation of judge’s scores sweeps up differences in the way judges score and tries to even them out. Just as this brings a generous and ungenerous judge into line, it will also tend to magnify the influence of a judge who can’t see much difference between objects, and one who can. In adjusting for individual quirks in how individuals judge, it effectively overrules what those individuals mean at the time of judging. There’s no way of getting away from this. Good and bad judges alike get swept up. A judge who gave away all 10s because they are overly generous looks exactly like a judge who gave all 10s because they really did mean to award 10s to what they saw, and what they saw really was worth it.
Normalisation risks throwing out the signal baby with the noise bathwater. Discrimination, in the original sense of discerning the difference between things, is what we want from a good judge, and measures designed to balance out biases can also reduce the influence of an individual’s judgement. To the extent that an individual judge is providing good information, the collective intelligence that results from aggregating across judges suffers.
This newsletter is free for everyone to read and always will be. If you can afford it, feel free to chip in to help me keep writing for everyone: upgrade to a paid subscription (more on why here).
See below for some caveats, and some other things I’ve noticed recently.
Caveats
There’s lots more to say about this topic. A lot of it concerns schemes for accounting for the difficulties I’ve gestured at. All of them make some assumptions, and compromise legibility (especially to the statistically unsophisticated). The fundamental story is that correcting for bias risks overruling input from some individuals, not correcting for bias risks letting the idiosyncrasies of individual judges play an outsized role in the overall outcome. You can’t get away from this because it is not possible to cleanly separate ‘bias’ in judgements from ‘taste’ unless you have access to the generating mechanisms within the judge that underlie the judgement (and maybe not even then).
Two caveats:
On identifying skilled judges: You could imagine some scheme where individual judges are tracked over time and scores for their ability to score proposals (either according to how other people score them, or actual outcomes). If you did this you could avoid correcting out their ‘bias’ on an isolated occasion where they gave all 10s. This kind of scheme is obviously harder the harder or more delayed is it to place an accurate value of the object (research funding, I’m looking at you).
On bias as generous/ungenerous and more generally: My discussion mixes bias in the signal detection theory sense and bias as a general category of ‘differences in the decision making process’. There could be a sense in which having a bias in the sense of “weighting different aspects of the judged object differently” is neutral with respect to bias in the sense of generous/ungenerous, but I don’t have the statistics chops to think through this systematically (as well as it maybe not being the place in this newsletter). The general lesson, which I think I could articulate better (if you know how, let me know) is that, if you look only at scores, one cannot distinguish personal characteristics of judgement which may be useful (i.e. aspects of taste which are pertinent to the outcome, and so provide signal) from personal characteristics of judgement which are mere prejudice (i.e. aspects of taste which are irrelevant and so constitute noise).
Other things…
No reason without dialogue, no dialogue without free speech
Here at Reasonable People we honour the Voltairian principle: “Monsieur l’abbé, je déteste ce que vous écrivez, mais je donnerais ma vie pour que vous puissiez continuer à écrire” (not an actual Voltaire quote). Here are some updates on that topic:
§
I Taught a Short Story. My College Fired Me For It.
I was not asking my students to like the story. Only to reckon with it. Literature that never unsettles us may be comfortable, but incomplete—antiseptically free of debate, friction, and discovery.
§
Tony Greenstein: The most important half-hour of my life
“Below is the text of the closing speech Tony Greenstein prepared for the end of his trial at Kingston Crown Court.
After a four-day trial ending on Friday, August 21 2026, Tony was acquitted of one count of inviting support for a proscribed terrorist organisation [Hamas] under Section 12(1) of the Terrorism Act 2000.”
[From the speech]
Ask yourselves — is a Jewish secular socialist who has spent twenty years criticising political Islam and Hamas a plausible recruiting sergeant for it?
And as you weigh that question, understand what it is to be me, standing in this dock. Someone who has spent fifty years fighting fascism on the streets of this country. I never imagined it could happen here in the mother country of democracy that I would be facing 14 years imprisonment for writing a blog and tweeting under the Terrorism Act 2000.
Wander the archives
I made a button you can click to get a random Reasonable Post from the 117 or so in the archives. Try it:
Link is https://random.reasonable-people.workers.dev
Misinformation Review special issue on Fact-Checking
From the leader, ‘The future of fact-checking in the algorithmic society’:
The early 2020s brought them three interrelated yet distinct crises: financial, technological, and legitimacy crises. The contributions in this issue seek to address some of these challenges from different perspectives. Taken together they demonstrate profound transformations in factchecking activity, inviting a broader debate about how its practices and legitimacy are negotiated and renegotiated in today’s algorithmic society.
Minformation Review, Issue 3, Volume 7: Fact-checking
Andy Stirling: Illuminating neglected uncertainties.
Subhead: ‘How metascience can help rather than hinder democracy’, Andy Stirling discusses how discourse around metascience has often obscured the conditionality (which he labels ‘uncertainty’), unspoken directionality (what we want to achieve) and lack of democracy in research and research reform. Public trust in science has been promoted with insufficient attention, he says, to discussion of science’s trust in the public:
This social reciprocity of trust matters because democracy is not simply about communicating expertise more effectively. It is about creating institutions capable of engaging constructively with legitimate disagreement. Public engagement is therefore not most valuable when it manufactures consensus, as is typically assumed across too many instruments of ‘public engagement’. It is instead when ‘uninvited participation‘ illuminates competing perspectives and divergent options – and clarifies why disagreements exist – that alternative futures can be properly explored. Productive dissensus is often a healthier democratic outcome than premature agreement.
Andy Stirling: Illuminating neglected uncertainties.
…And finally
The fourth little pig’s house…
via @cmconseils, a classic from SMBC
END
Comments? Feedback? Invites to a Chopper Chicks in Zombietown watch party? I am tom@idiolect.org.uk and on Mastodon at @tomstafford@mastodon.online
AI declaration: I asked Gemini to check for spelling and grammar. For this post I also asked Claude to make the plots (link).
It’s interesting that we feel so uncomfortable that a person’s individuality expresses itself in their judgment (“bias”). Even striving for maximal objectivity is an individual choice according to some, and often they ridicule the idea of objectivity as such. And here’s the rat: does it have something to do with living in ever more diverse societies and suspecting that one would probably not agree with a random other person’s judgment?