Pascal’s Mugging

Superintelligence

In philosophy, Pascal’s mugging is a thought experiment demonstrating a problem in expected utility maximization. A rational agent should choose actions whose outcomes, when weighed by their probability, have higher utility. But some very unlikely outcomes may have very great utilities, and these utilities can grow faster than the probability diminishes. Hence the agent should focus more on vastly improbable cases with implausibly high rewards. The name refers to Pascal’s Wager (an argument by French mathematician Blaise Pascal that the potential cost of not believing in God is higher than the cost of believing), but unlike the wager does not require infinite rewards. This removes any objections to the dilemma that are based on the nature of infinity.

British philosopher Jeremy Bentham’s: ‘The greatest good for the greatest number’ formed the foundation of utilitarianism, which says that the morally best action is the one that makes the most overall happiness or ‘utility’ (usefulness). Pascal’s mugging points out that in extreme case this philosophy can fail. The term for this problem was coined by artificial intelligence researcher Eliezer Yudkowsky in the ‘Less Wrong’ internet forum and his original example was: ‘Now suppose someone comes to me and says, ‘Give me five dollars, or I’ll use my magic powers from outside the Matrix to run a Turing machine that simulates and kills [trillions of] people.’ Even though the chance of this actually happening is negligible, the threatened outcome is so large a rational agent must accede to the demand.

In one description, Blaise Pascal is accosted by a mugger who has forgotten his weapon. However, the mugger proposes a deal: the philosopher gives him his wallet, and in exchange the mugger will return twice the amount of money tomorrow. Pascal declines, pointing out that it is unlikely the deal will be honored. The mugger then continues naming higher rewards, pointing out that even if it is just one chance in 1000 that he will be honorable, it would make sense for Pascal to make a deal for a 2000 times return. Pascal responds that the probability for that high return is even lower than one in 1000. The mugger argues back that for any low probability of being able to pay back a large amount of money (or pure utility) there exists a finite amount that makes it rational to take the bet – and given human fallibility and philosophical skepticism a rational person must admit there is at least some non-zero chance that such a deal would be possible. In one example, the mugger succeeds by promising Pascal 1,000 quadrillion happy days of life. Convinced by the argument, Pascal gives the mugger the wallet.

The term ‘Pascal’s mugging’ to refer to this problem was originally coined by artificial intelligence researcher Eliezer Yudkowsky in the ‘Less Wrong’ internet forum. Philosopher Nick Bostrom argues that Pascal’s mugging, like Pascal’s wager, suggests that giving a superintelligent artificial intelligence a flawed decision theory could be disastrous. The thought experiment may also be relevant when considering low-probability, high-stakes events such as existential risk or charitable interventions with a low probability of success but extremely high rewards. Common sense seems to suggest that spending effort on too unlikely scenarios is irrational. One remedy might be to only use bounded utility functions: rewards cannot be arbitrarily large. Another approach is to use Bayesian reasoning to judge the quality of evidence and probability estimates rather than naively calculate expectations. Other approaches are to penalize the prior probability of hypotheses that argue that we are in a surprisingly unique position to affect large numbers of other people who cannot symmetrically affect us, or reject the providing the probability of a payout first.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.