Utilitarianism Casts Doubt on The Simulation Hypothesis, AI Doomsday and Substrate-Independence
Exploring the relationship between Bostrom's simulation argument and utilitarianism.
The rationalist community consists of people who take figuring out the truth seriously. Rationalist are fond of Bayesian reasoning and avoiding cognitive biases. Whether through peer-influence or convergence on the truth, rationalists tend to have similar beliefs on a number of philosophical issues. For example, rationalists tend to be irreligious, physicalist, utilitarian and concerned with catastrophic risk, most notably from general artificial intelligence. I want to address a potential relationship among some popular beliefs within the community.
Nick Bostrom, a philosopher who is liked by rationalists, has written a great deal about artificial intelligence risk and has formulated an extremely interesting argument which takes seriously the possibility that we are living inside of a computer simulation. People who are not philosophically minded probably dismiss this as silly speculation, but I think that open-minded rationalists are often sympathetic to this argument. I think that the simulation argument is worth considering because it has potential ethical implications.
Bostrom simulation argument is that at least one of three propositions is true: (1) it is very likely that human beings will go extinct before achieving a sufficiently high level of technical sophistication to run conscious simulations; (2) advanced civilizations are unlikely to run a large number of simulations of their past; (3) we are almost certainly living inside a computer simulation. A key assumption that the argument rests on is substrate-independence—which basically means that it is possible to simulate consciousness on something that isn’t a brain, namely a computer. Rationalists, who often reject the existence of souls, seem to not take much issue with this assumption.
Rationalists also typically believe in utilitarianism—the belief that we ought to maximize utility. Utilitarians are cognitivists—meaning that some ethical statements express something meaningful which can be evaluated as true or false. And they are moral realists, because they believe that some moral statements are true. Most people are at least moral realists and so some of the arguments below can be generalized.
If utilitarianism is true, then a sufficiently intelligent general artificial intelligence should be able to determine that moral realism is true and that utilitarianism is the correct ethical choice. Upon learning that utilitarianism is correct, a general artificial intelligence should have some motivation to act morally.1 However, this rests on a rejection of what is called the orthogonality thesis: “Intelligence and final goals are orthogonal axes along which possible agents can freely vary.”2 If the orthogonality thesis is true, then we should be concerned. If the thesis is false, the risk of evil artificial intelligence destroying humanity is low, provided utilitarianism is true. But maybe we should still be concerned. Another possible scenario is if humans are actually bad people and deserve destruction, then a utilitarian AI might be right to take over.
One potential objection would be that we apprehend moral truth through a uniquely human form of intuition which a computer would lack. This should not be an issue if one is willing to accept substrate-independence; a sufficiently powerful artificial intelligence should be able to simulate whatever a human can do. Provided that the orthogonality thesis is false, the stronger our belief in utilitarianism, the less likely we should be concerned about an evil artificial intelligence. This means that utilitarianism, as well as moral realism, undercuts proposition (1) of Bostrom’s simulation argument.
A further reason that I doubt scenario (1) would be more probable than (3), provided substrate-independence is true, is that I believe that time is infinite in both directions.3 Eventually some civilization, whether post-human or not, would attain technical sophistication sufficient enough to simulate many conscious beings—which would likely result in a high simulation to real-world being ratio, increasing the probability of (3).
I think that scenario (2), that posthumans who could simulate would not is made less likely by utilitarianism. If utilitarianism is true, then I believe that posthumans would come to recognize this fact and if they did, they would recognize a moral obligation to create ethical simulations which maximize utility. Provided sufficiently low cost, this may be the best thing future effective altruists could do.4 If we are likely in a simulation, it is as bad as this world is and utilitarianism is true, then we should be skeptical of unending moral progress and maybe reject “future-proof ethics.”5
Since we are clearly not in a utility maximizing world or anything close to it, we should doubt either that the simulation is true or that utilitarianism is true. It is possible that this world maximizes some other ethical variable. But I find that unlikely, just as I did when I learned Leibniz thought this was the best possible world. If you found my arguments against scenario (1) unpersuasive, you may want to significantly increase your probability of a doomsday. If you want to maintain your belief that utilitarianism is true, I think you should be skeptical of Bostrom’s premise of substrate-independence. Maybe you can’t simulate consciousness on a machine?
Summary
If substrate-independence assumption (SIA), then I, II or III [The simulation argument].
If utilitarianism (U) and orthogonality thesis is false (~OT), then P(I|U & ~OT) < P(I), specifically P(AI Doomsday|U & ~OT) < P(AI Doomsday).
If infinite time (IT), scenario I doesn’t matter as much.
If infinite time (IT), then P(II*|IT) < P(II*) where II* is that some intelligent entity, not necessarily posthuman, is not going to simulate.
If utilitarianism (U) is true, P(II|U) < P(II) because creating happy simulations will be an ethical obligation.
If utilitarianism (U) is true, P(III|U) < P(III) because we would expect to be in a happier world.
If utilitarianism is true, P(I), P(II) and P(III) are small.
If ~I, ~II and ~III, then ~SIA.
If utilitarianism is true, P(~SIA) is high.
I was made aware of this point by Mark P Xu Neyer in this comment.
Bostrom, Nick (2012). The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents. Minds and Machines 22 (2):71-85.
This premise leads me to believe in reincarnation. The probability of a single existence is less likely than the probability of multiple existences, provided I exist right now. See: The Bayesian Case for Reincarnation.
I discuss this argument in another blog post. See: Artificial Intelligence as the Ultimate Utility Monster.
See: Future-proof ethics by Holden Karnofsky
Hi. My view on Bostrom’s propositions is that (1) is true. Maybe I’m a pessimist but I see a history of rising and falling civilizations, and I see cracks in the foundation of ours that don’t give me confidence of an infinite future.
As you know from ACX comments, I am a religious guy (but I like of what rationalism has to offer). Your point 6 is analogous to the problem of evil for theists. I don’t agree with your point 6 for the same reason I don’t think the Problem of Evil disproves God. What if the maximal utility for simulated beings is achieved by giving them challenges to overcome for personal agency and growth? Otherwise you have a simulated boring Garden of Eden with zero impetus for personal or civilizational development.
Your blog gives a lot of fun food for thought. I’ll look through it some more.
In philosophy, there is a dispute about whether you must act on a moral.truth once you have discovered it ...whether moral truths are intrinsically motivating. (You phrase it somewhat weakly as "..should have some motivation to act morally..." )
Intrinsic motivation is an assumption in addition to moral realism, and it's not clear that rationalists generally believe it. Without intrinsic motivation, nothing much follows find from utilitarianism..