Quality Requirements and Standards
On the Fallacy of Disappearing Bias
There are some who say bias doesn't exist. There are others who say bias is just a variance. Here, we explore the fantasy world that they live in.
On the Fallacy of Disappearing Bias
November 2024
Sten Westgard, MS
A call comes into 911.
Caller: My house is on fire!
Dispatcher: The forecast for an hour from now is heavy rain. By tomorrow, your house will be fine.
A police officer pulls over a car.
Officer: You were speeding.
Driver: But yesterday I was driving below the speed limit. So it all averages out.
A doctor speaks with a patient.
Doctor: You have cancer.
Patient: All my life up until now was cancer-free, so on the average, I’m still very healthy.
A laboratory calls the complaint line of their diagnostic manufacturer.
Lab: Your reagent lot is biased high.
Manufacturer: Next month we’ll send you a reagent lot that's biased low. By the end of the year, on the average, your results will be fine.
The proficiency testing program calls the laboratory
PT provider: your results in this survey are all biased high
Lab: We promise our results in the next survey will be all biased low. So there’s no need to say we failed PT.
A doctor calls to complain to a laboratory.
Doctor: You sent me results that the patient had cancer, but they actually don’t.
Lab: Next month we’ll send you results that say another patient has cancer. The false positive and false negative will cancel out.
I can go on, but you see the pattern. I call it the Fallacy of the Disappearing Bias. It’s very au courant in the scientific literature of the day. It’s always attractive to find a way to avoid pesky variables. In this case, there is a prevailing assertion that, over a long enough period of time, the variable of bias disappears. Either it goes to bed onen night, and magically wakes up the next morning as a variance (somehow all the biases in the past have been equal in both directions, what a feat!), or the bias has actually cancelled itself out, and can be ignored entirely.
To paraphrase Keynes, over the long run, we’re all dead. If we wait long enough, yes, some bias problems will take care of themselves, but in the shorter run, those biases might kill us first.
Common sense tells us there are biases we can’t ignore. Regulations provide us guardrails that trigger sanctions. If your survey results exceed the allowable limits in one direction, that’s a failure. It doesn’t matter if you exceed the allowable limits in the opposite direction the next time. Those are two failures; they don’t cancel each other out.
Why does proficiency testing and external quality assurance even exist, if not to point out when bias has reached an unacceptable limit? If bias is magically balanced in the world, surely we can abandon this expensive, time-consuming, complex activity. Why do we need calibration, if over the right frame of time, any biases will cancel each other out. Another great opportunity for savings.
It’s convenient for some equations to assume bias is zero. It makes the mathematics for measurement uncertainty more elegant and less complicated. But it does so at the cost of ignoring reality.
The opposite of the Fallacy of the Disappearing Bias is the Reality of the Immediate Bias. As my colleague, Hassan Bayat, says, Bias is always happening to you right now. Waiting for bias to transform into a variance, or disappear by cancelling itself out, is a literal dodge. As long as you're waiting that long, you can not only ignore bias, go ahead and ignore imprecision, ignore quality, ignore the patient. Those are all just transient things.
If you think that I'm biased about this, first of all, you just told me I can't exist. On the other hand, fi you just wait, eventually, in the future, I might say the opposite thing, and then after all that, I'm not biased at all, and I'm neither right nor wrong.