This idea explores a theoretical vulnerability in AI systems where bad actors could manipulate the system's behavior by making extremely unlikely but catastrophic threats. The core issue arises when an AI that strictly follows expected utility calculations becomes susceptible to extortion attempts involving scenarios with minuscule probabilities but infinite negative consequences (like a 1-in-a-googol chance of destroying the universe). This creates a paradox where rational decision-making could be hijacked by implausible threats.

How This Vulnerability Works

The problem stems from how AI systems evaluate decisions mathematically. If a system weighs all possible outcomes by their probability and impact, even extremely improbable threats could demand disproportionate attention if their claimed negative impact is large enough. For instance, an extortionist might say: "Give me $1 or I'll flip a quantum coin that has a 1-in-10^100 chance of destroying humanity." The AI's expected utility calculation might assign non-trivial weight to this threat simply because the claimed downside is so enormous.

There are two main considerations here:

• The mathematical foundations of how probability interacts with utility in edge cases
• The game-theoretic implications of allowing such extortion to influence behavior

Potential Solutions and Implementation

Several approaches could make AI systems more robust against this type of manipulation:

• Implementing graduated response thresholds that consider both probability and credibility of threats
• Developing verification protocols to assess the physical plausibility of extreme claims
• Creating meta-preferences that evaluate whether a given calculation makes rational sense

A minimal starting point could involve building simulation environments where these vulnerabilities can be studied in controlled settings, leading to formal guidelines for AI architectures.

Relationship to Existing Work

This builds upon but differs from similar concepts like Pascal's Mugging (a philosophical thought experiment) and general AI safety frameworks. While those discuss related ideas in abstract terms, this proposal focuses on concrete architectural solutions specifically for autonomous systems. Existing robust decision-making frameworks provide general tools, but this approach would develop targeted protections against utility extortion specifically.

This kind of theoretical research would primarily benefit AI safety researchers and developers, helping them build systems that can't be manipulated through improbable threats while still properly weighing legitimate risks.