On the Moral Weight of Probabilistic Harms

Consider an arrangement that is now ordinary. A bank deploys an automated underwriting system to approve consumer loans. The system was trained on a decade of historical decisions; it has been audited for disparate impact, regularised against the worst of the documented biases, and operated with human review on rejected applications above a certain threshold. The bank's officers can say, truthfully, that the system performs better than the loan officers it replaced on every measurable axis — default rates are lower, processing times shorter, and the rate at which protected groups are rejected has fallen.

And yet, once every ten thousand decisions or so, the system declines a loan that should have been approved, on grounds it cannot articulate, to a person whose immediate need was acute. The person loses the deposit on the apartment. Their child changes schools. Some weeks afterward, the missed loan ramifies into a job lost, a marriage strained, a small life rerouted.

The question I want to raise is whether the bank has wronged that person. Not in the legal sense; in the moral sense. The harder, more interesting version of the question is whether anyone at the bank has wronged the person — the data scientist who designed the system, the executive who approved its deployment, the analyst on duty when the decision was made. My intuition, and I suspect yours, is that somebody must have, because the harm is real and the chain of action is short. But when one tries to lay the wrong at the door of any individual, it appears to slide away.

Moral Luck in the Algorithmic Register

The intuition is not new. Bernard Williams and Thomas Nagel argued, in a now-canonical pair of essays from the 1970s, that whether an action is blameworthy depends on facts the agent does not control: the truck driver who hits a child, blamelessly, has acquired a moral burden that an otherwise identical driver who hit nothing has not.[^1] Williams and Nagel disagreed on what to do about this, but they agreed that the dependence of moral standing on factors outside the agent's control is uncomfortable, and that the discomfort is not dissolved by pointing out that it is unfair.

What is interesting about automated decision systems is that they convert what was once moral luck into a known operating parameter. The bank knows, before deployment, that the system will produce roughly one wrongful denial per ten thousand decisions. The bank can adjust the threshold and trade some loss in accuracy for a lower wrongful-denial rate. The bank can deploy a more expensive review pipeline. The wrongness is no longer a matter of luck; it is a matter of policy. And once it is a matter of policy, the structure of responsibility changes.

The bank knows the rate of wrongful denial in advance. To call it ‘unavoidable’ is no longer to describe a fact about the world; it is to make a choice about whose harm the institution is willing to absorb.

This is, I think, the moral substance of the problem. In the truck driver's case, the harm is genuinely unforeseeable; the driver could not have known that this child, at this moment, would step from behind that parked car. In the bank's case, the harm is foreseeable in aggregate. The institution knows that some person will be wronged, knows what kind of wrong it will be, and chooses to proceed anyway because the expected aggregate welfare is greater under deployment than under refusal.

That choice may even be the correct one. I do not want to argue that probabilistic harms are categorically impermissible; that would commit one to the abolition of vaccination programmes, infrastructure projects, and most of medicine. What I want to argue is that the moral structure of the bank's situation is misdescribed when one says, as is so often said, that the system makes no mistakes that a human officer would not have made. It does. It makes the mistake on a schedule.

Who Owes the Victim What

If a probabilistic harm is, in aggregate, foreseeable, then the institution that produces it acquires a particular obligation to the people who fall on the wrong side of the distribution. Shannon Vallor has argued, in Technology and the Virtues, that the moral demands of new technologies should be understood not as a new ethics, but as the application of older virtues — honesty, justice, care — to circumstances in which they have not yet been worked out.[^2] In this case the relevant virtue is, I think, a kind of institutional candour. The bank's officers ought to be able to say, plainly: we know this kind of mistake happens; here is what we do for the person to whom it happens; here is the procedure by which a wronged person can seek redress; here is what we have done in the past five years to reduce the rate.

In practice, none of these statements is available. The contemporary norm is for institutions to deny that any individual harm occurred, on the ground that the system's aggregate performance was acceptable. The wronged customer is told that the decision was made by an algorithm, that the algorithm performs to specification, and that the institution regrets the inconvenience. The moral wrong is, in effect, dissolved into the variance of the distribution. It is no one's fault because it is the rate, and the rate is acceptable, because the alternative would be worse.

I find this dissolution unconvincing. The wronged customer is not a sample from a distribution; she is a person, with a deposit lost and a child reassigned, and the bank owes her not statistical exoneration but specific repair. Cathy O'Neil's Weapons of Math Destruction is largely a catalogue of cases in which the dissolution succeeded.[^3] The cases are dispiriting partly because the wrongs are individually small and partly because the institutions involved would have responded to the same wrongs, committed by a person, with apology and compensation. The probabilistic frame allows them not to.

The Engineer's Obligation

It might be objected that the responsibility I am describing rests with the institution, not with the engineer. The engineer, after all, built the system to a specification; the question of acceptable risk is one for the bank's executives and, ultimately, for the regulator. This is not wrong, but it is, I think, incomplete. Hardt, Price, and Srebro's work on equality of opportunity in supervised learning shows that the engineer's choice of fairness criterion has substantive moral content; the same training data can be made to satisfy demographic parity, equal odds, or calibration, but cannot in general satisfy all three at once.[^4] The choice of which to satisfy — which is to say, the choice of who absorbs the residual error — is made somewhere in the engineering process, often by someone who does not realise that they are making a moral decision.

Selbst and colleagues, in a much-cited 2019 paper, describe this as the fairness trap: the conviction that fairness is a property of an algorithm, when in fact fairness is a property of the sociotechnical system in which the algorithm is embedded.[^5] The trap is comfortable because it lets the engineer think she has done her part once she has run the audit. She has not. Her part includes asking what the audit could not detect, who the victims of the residual error will be, and whether the institution she is delivering the system into has the means and the will to do right by them when the residual error lands.

A reader will see immediately that this is a counsel of perfection, and that it cannot be met fully by any individual engineer in any plausible workplace. That is true, and it does not relieve the engineer of the obligation to meet it imperfectly.

A Provisional Conclusion

Probabilistic harms have a peculiar moral grammar. They are foreseeable in aggregate, unforeseeable in particular, and dissolvable into rates — which is to say, they are well suited to the institutional habit of refusing personal responsibility for damages an institution has chosen to produce. The temptation, as the use of automated decision systems grows, is to allow more and more wrongs to take this form: to be everyone's policy and no one's fault.

The corrective, I believe, has two parts. The first is institutional: a refusal to permit the dissolution of personal wrong into statistical exoneration, accompanied by specific procedures of repair when the system has produced a particular victim. The second is professional: a willingness, on the part of engineers and the executives who direct them, to treat the design parameters of a probabilistic system as the moral choices they are.

Neither part of the corrective is novel. Both have been argued for, in different vocabularies, by every serious commentator on this subject. The argument is therefore not that these things are unknown but that they are unobserved. The work, as so often, is the observance.

[^1]: Williams, "Moral Luck," (1981); Nagel, "Moral Luck," (1979). [^2]: Vallor, Technology and the Virtues (2016), esp. chapters 2 and 4. [^3]: O'Neil, Weapons of Math Destruction (2016), pp. 41–67 on credit scoring. [^4]: Hardt, Price & Srebro (2016). [^5]: Selbst et al. (2019).