Envisioning the Potential Scenario of an AI-Driven Catastrophe

Leah Feiger: This is incredibly useful for me that we’re organizing it into these three categories.

Zoë Schiffer: I’m a fan of a good bucket.

Leah Feiger: I’m all about the bucket. I want to explore each of these. My initial question is, where do the nuclear codes fall? Which category do they belong to?

Brian Barrett: That’s actually one we don’t need to be concerned about, right?

Zoë Schiffer: Exactly.

Leah Feiger: Can you elaborate?

Zoë Schiffer: When you speak with AI researchers and really press for clarity, you can identify the three buckets I mentioned. But if you continue the conversation, they’ll say, “There are these other situations where AI is inside a robot, and that robot can engage with the physical world in unsettling ways.” However, those scenarios are more distant and they acknowledge it’s somewhat unlikely.

Brian Barrett: One thing I’d like to clarify, Zoë, regarding your buckets, is that I want to suggest a sub-bucket or maybe a small pail.

Zoë Schiffer: Oh, a pail!

Brian Barrett: Right, because you mentioned AI creating a bioweapon, AI hacking something, and so on, all of which is accurate. However, I think there’s a crucial distinction between actions guided by humans versus AI acting independently. Many people might wonder, “How could AI possibly release a bioweapon? That’s absurd.” And while that seems far-fetched, we must acknowledge that there are individuals out there who could use AI to develop and deploy such a weapon.

Zoë Schiffer: That scenario is completely valid. If you provide individuals with knowledge on creating a bioweapon, there will be reckless and foolish people who attempt it. Absolutely, people engage in all kinds of dangerous and malicious activities regularly.

Leah Feiger: Often inspired by information found online.

Zoë Schiffer: Right. I totally agree that this is extremely perilous. AI executing this entirely on its own does seem a bit more implausible, though we can discuss that further.

Brian Barrett: I might get criticized for this, but I want to reference the 1983 film, War Games.

Leah Feiger: No, I think this is precisely what we need to discuss. Yes, War Games is essential.

Brian Barrett: The point is, AI isn’t going to be the one pressing buttons or turning keys, but in War Games, a computer nearly initiates nuclear war by misleading the US into thinking it was under attack from Soviet missiles. So, is there a risk—though I assume we have adequate safeguards—that something like this could happen? I hope we have enough satellite intelligence to verify critical situations. While I believe it’s manageable, the real concern lies in AI triggering alerts that could lead to a disastrous chain of events.