An artificial general intelligence (AGI) might have an instrumental drive to modify its utility function to improve its ability to cooperate, bargain, promise, threaten, and resist and engage in blackmail. Such an AGI would necessarily have a utility function that was at least partially observable and that was influenced by how other agents chose to interact with it. This instrumental drive would conflict with the strong orthogonality thesis since the modifications would be influenced by the AGI’s intelligence. AGIs in highly competitive environments might converge to having nearly the same utility function, one optimized to favorably influencing other agents through game theory. Nothing in our analysis weakens arguments concerning the risks of AGI.
Purpose: The great filter and an unfriendly artificial general intelligence might pose existential risks to humanity, but these two risks are anti-correlated. The purpose of this paper is to consider the implications of having evidence that mankind is at significant peril from both these risks. Design/methodology/approach: This paper creates Bayesian models under which one might get evidence for being at risk for two perils when we know that we are at risk for at most one of these perils. Findings: Humanity should possibly be more optimistic about its long-term survival if we have convincing evidence for believing that both these risks are real than if we have such evidence for thinking that only one of these perils would likely strike us. Originality/value: Deriving implications of being greatly concerned about both an unfriendly artificial general intelligence and the great filter.
An artificial general intelligence (AGI) might have an instrumental drive to modify its utility function to improve its ability to cooperate, bargain, promise, threaten, and resist and engage in blackmail. Such an AGI would necessarily have a utility function that was at least partially observable and that was influenced by how other agents chose to interact with it. This instrumental drive would conflict with the orthogonality thesis since the modifications would be influenced by the AGI's intelligence. AGIs in highly competitive environments might converge to having nearly the same utility function, one optimized to favorably influencing other agents through game theory.
29 May 2025 in SuperIntelligence - Robotics - Safety & Alignment
What should you believe will be the likely cause of your survival if you are an AI Doomer who believes in quantum immortality? An AI Doomer believes that if their understanding of AI existential risks is correct, humanity must get extremely lucky to survive. A person who believes in quantum immortality thinks that the many-worlds interpretation of quantum physics implies that their consciousness will necessarily continue in some branch of the multiverse. This article shows that combining these two beliefs requires rejecting or going beyond simple Bayesian reasoning when estimating one's most likely cause of survival. Such estimates may have practical implications for investment decisions, reputation management, and anthropic shadows.