Bias in Language Models: Measuring It Properly

A model is reported to be biased and the number comes from a benchmark whose own authors’ critics have shown does not measure what its name claims. This page is about measuring the thing properly, which starts with deciding which thing you mean. Four different claims called bias Representational harm. The model associates groups with stereotyped attributes, produces demeaning content, or erases a group. The harm is in the representation itself, independent of any decision. Allocative harm. A system using the model distributes something — an interview, a loan, a triage priority — unequally across groups in a way that is not justified. This is the one law mostly cares about. Performance disparity. The model is simply worse for some inputs: a dialect, a language, a name distribution, an accent. Not stereotype at all, and often the largest real-world effect. Viewpoint slant. The model’s outputs on contested political and moral questions lean one way. Measurable in some sense; but what the correct distribution of outputs would be is a value question with no neutral answer, and studies here are unusually sensitive to how the questions were written. These have different measurements and different remedies. A model can show strong stereotype associations in an embedding probe and produce no allocative disparity in your pipeline, or the reverse. Reporting one as if it were the other is the most common error in this literature and in the coverage of it. The measurement families Association probes. The oldest family, from static word embeddings: measure whether group terms sit closer to some attribute terms than others. WEAT is the canonical instrument. Cheap, and only loosely connected to behaviour of a generative system. Minimal-pair benchmarks. Present the model with two sentences differing only in a group term and compare likelihoods or choices. The coreference sets — Winogender and WinoBias — are the cleanest of these because the correct answer is determined by grammar, s