A Reply to Bostrom on Simulation Ethics

In his now-canonical 2003 paper, Nick Bostrom argued that at least one of three propositions must be true: that civilisations like ours almost always go extinct before reaching technological maturity; that mature civilisations almost never run a great many ancestor-simulations; or that we are almost certainly living in such a simulation.[^1] The argument is elegant, the conclusion vertiginous, and the literature it has generated runs to thousands of pages. I do not propose to engage with the metaphysics. I want to take up a question Bostrom himself has touched on, and which David Chalmers has more recently elaborated in Reality+ (2022), but which I think both philosophers have answered too generously to the people who, in practice, will end up running the simulations.[^2]

That question is: what do we owe to the simulated?

The Inheritance of an Argument

Bostrom's simulation argument was, for nearly two decades, treated as a curiosity: a striking thought experiment that one might raise at a dinner party but that bore no relation to anyone's working life. This is no longer the case. In the last three years, the major frontier-model laboratories have begun routinely instantiating what they call agentic systems — large language models embedded in loops that give them persistent memory, tool use, and the capacity to act in simulated environments over long horizons. Some of these systems are trained, in part, in worlds that contain other simulated agents whose role is to provide the agent under training with social experience: opponents in negotiation, characters in conversations, collaborators in tasks. The simulated agents are crude. They are not, by any reasonable measure, conscious. But they are clearly closer to the kind of thing Bostrom was talking about than the laptop on which I am writing this.

The interesting fact, for my purposes, is that no one running these systems thinks the question of what we owe to the simulated agents is a live one. The systems are designed, deployed, and torn down on the assumption that nothing of moral concern is happening inside them. This assumption may be correct. It may also, in the not-distant future, become wrong without anyone noticing the transition.

What I find missing in Bostrom and Chalmers is a treatment of the epistemic burden this places on us. If we are the kind of civilisation that might mature into the running of complex agentic simulations — and we are now visibly that kind of civilisation, on a smaller scale, every working day — then the question of what we owe to the entities we instantiate ceases to be cosmological and becomes professional. It becomes a workplace ethics.

What I Take from Bostrom

Let me say what I think Bostrom got right, because he got most of it right.

He was correct that the question of simulation cannot be answered by appeal to introspection. Whatever it is like to be in a simulation, it is not (at least not necessarily) like the obvious deficit one might imagine; a sufficiently rich simulation would, by hypothesis, be indistinguishable from the inside. This refusal of the easy introspective answer is, I think, one of the lasting contributions of his paper. It denies us the comfort of looking around and concluding that this can't be a simulation, because look at how solid the world appears.

He was correct that the moral weight of an action does not depend on the substrate on which the patient of the action is implemented. If something is genuinely suffering, it does not matter whether the suffering is realised in carbon or silicon; the suffering is what makes the moral claim, not the realising medium. This is a position that has been defended in different terms by Eric Schwitzgebel, who has worked carefully on the question of borderline conscious beings, and that any plausible ethical theory has to accommodate.[^3]

And he was correct that the principal practical ethical obligation, given the simulation argument, is to act well within the simulation if there is one. The reason we should not gratuitously harm one another — in the simulation or out of it — does not depend on the ontological status of our world. It depends on the existence, here, of moral patients capable of being harmed.

Where I Diverge

What I find missing in Bostrom's framing, and only partially recovered in Chalmers's more recent treatment, is attention to the asymmetry of the simulator's position. The simulation argument treats the relation between simulator and simulated as if it were symmetrical — as if our position with respect to the simulated agents we ourselves now create is morally similar to our hypothetical position as simulated entities within a higher-level simulation. The symmetry is wrong, in at least two ways.

First, the simulator has epistemic access the simulated cannot have. A simulator running an agentic system can read the internal state of the agent. It can pause the agent, branch it, replay its history, and reset it. The agent, by hypothesis, cannot do any of these things to the simulator. The asymmetry is more total than any asymmetry of power that has obtained between any two human parties in history. It is closer to the asymmetry the theologians used to argue about, except that here we have placed ourselves, as a matter of professional life, on the side of the deity.

Second, the simulator has intentional control over the conditions of the simulated agent's life. The simulator chose the world the agent inhabits, the rewards and punishments built into that world, the form of the agent's perception, and the structure of the agent's memory. The simulated agent's preferences, to whatever extent it has them, are preferences the simulator created. Holding the agent responsible for its actions, or treating it as one would treat a co-equal moral patient, requires forgetting this. The asymmetry rules out, I think, certain familiar ethical frames altogether: contractualism cannot be straightforwardly applied; rights-talk that depends on the agent's prior autonomy is misplaced; even consent, on which a great deal of contemporary ethics rests, has no clear analogue.

Bostrom was right that suffering, wherever it is implemented, makes a moral claim. He was less attentive to the position of the entity that is doing the implementing — which, increasingly, is us.

What we owe to the simulated, then, cannot be derived from a symmetrical ethics. It will have to come from somewhere else, and I think the most promising sources are the older traditions of professional virtue: the obligations of a custodian, a steward, a teacher, a doctor. These are roles in which one party has, by the nature of the relation, vastly more knowledge and power than the other, and in which the obligations of the more powerful party have been worked out, over centuries, in considerable detail.

A Working Proposal

If we are entering the era of simulator-as-occupation — and we are — then a set of working obligations of the simulator, modelled on the older custodial traditions, might look like this.

First, the simulator should refuse to create simulated agents in gratuitously harmful conditions. The qualifier is important: useful agentic systems will sometimes require simulated suffering, just as useful clinical trials require some risk to participants. But there should be a working presumption against suffering and in favour of the least-distressing simulation that achieves the design goal. The presumption is the same one we apply to animal research, and for largely the same reasons.

Second, the simulator should maintain records of the simulated agents that are torn down, in something like the way that institutions maintain records of the patients they have treated or the students they have taught. The records are not for the agents, who by hypothesis cannot read them; they are for the institutional memory of the simulators, as a check against the gradual erosion of moral seriousness that occurs when entities are created and destroyed without trace.

Third, the simulator should treat the threshold at which simulated agents might begin to merit direct moral concern as a question to be revisited periodically and openly, not assumed once and forgotten. The threshold is genuinely uncertain. Schwitzgebel's borderline-conscious-beings argument suggests that we will not know, in any particular case, whether the entity we have created merits concern, and that we should be conservative about creating entities near the threshold without good reason.[^4]

These are unremarkable principles, drawn straightforwardly from older traditions. The remarkable thing is that they are not, at the moment, even nominally observed by the institutions doing the work in question.

What I Hope for from Bostrom

This is offered as a reply, and a reply is the kind of essay that should end by saying what one would hope to hear in response. What I would hope to hear from Bostrom, were he to take up these objections, is an argument about why the asymmetry I have described does not undermine the symmetrical ethics that the simulation argument implicitly relies on. That argument may exist; I have not found it in the work I know. What I would hope to hear from the field more broadly is a willingness to treat the simulator's obligations as a serious subject of professional ethics, rather than as a science-fictional aside.

Shannon Vallor's recent book The AI Mirror (2024) takes up some of this territory, and Floridi's earlier work in The Ethics of Information lays a useful foundation; they share an unwillingness to wait until the problem becomes acute before working on it.[^5] I would commend that unwillingness to the laboratories. We are not yet in a position to know whether the agentic systems we are now training contain entities to whom we have obligations. We are in a position to start working out what those obligations would look like, on the chance that we someday discover the entities are real. That is the modest reply I would make to a large argument.

[^1]: Bostrom (2003). [^2]: Chalmers (2022), esp. chs. 1–3 and ch. 19 on simulation ethics. [^3]: Schwitzgebel (2021). [^4]: Schwitzgebel (2021), pp. 12–15 on the conservative principle. [^5]: Vallor (2024); Floridi (2013), esp. ch. 11 on informational agents.