Theoretical Computer Scientist: AI Just Changed Math Forever, with Scott Aaronson

(Scott Aaronson at 00:00:00) OpenAI just released solutions to 400 major open problems.

(Joel Beasley at 00:00:07) No way. You hadn't seen that yet?

(Scott Aaronson at 00:00:09) I have not seen that. Yes. No matter what happens, they're immediately... I have learned there are people who will say, "Well, of course AI could do that. No one ever doubted that AI could do that," even if those same people did doubt it six months ago. One of them is a problem that my wife has been working on for the entire time that I've known her.

(Scott Aaronson at 00:00:27) It's called the Unique Games Conjecture, and there were many, many experts working on this for 25 years.

(Joel Beasley at 00:00:40) Why was it the biggest day in the history of math?

(Scott Aaronson at 00:00:44) Well, I mean, because OpenAI just released solutions to 400 major open problems.

(Joel Beasley at 00:00:52) No way. You hadn't seen that yet? I have not seen that.

(Scott Aaronson at 00:00:54) Yes. Yes.

(Joel Beasley at 00:00:56) What does this mean?

(Scott Aaronson at 00:00:58) Well, it means that the nature of math as a human subject is forever totally different from what it was before. It means that our job as humans, if we have a job as mathematicians, as theoretical computer scientists, theoretical physicists, and so forth, is going to mostly involve working with AIs and understanding and explaining and directing what they're doing. Because, in terms of... once you decide what problem you want to solve, they are just better than us at solving them.

(Joel Beasley at 00:01:42) So help me understand this. My background: software engineering for large-scale business applications.

(Scott Aaronson at 00:01:48) The AI apocalypse already arrived for software engineering until we go.

(Joel Beasley at 00:01:53) Oh, yeah. Yeah. Yeah. And I've been riding that wave.

(Scott Aaronson at 00:01:55) And now it's come for us, basically, right? So anyway...

(Joel Beasley at 00:01:58) So you guys were safe for a little bit.

(Scott Aaronson at 00:02:00) Yeah.

(Joel Beasley at 00:02:00) You saw...

(Scott Aaronson at 00:02:01) That's right. We were... you know, there was a year or two when people could delude themselves that, "Oh, you know, software engineering—yeah, sure, you know, of course AI could do that. But, you know, math is truly creative and, you know, pinnacle of the human intellect, and no AI is going to replace us at that." And that's now completely untenable.

(Joel Beasley at 00:02:22) Did they really think that?

(Scott Aaronson at 00:02:24) I mean, that... you know, no matter what happens, they're immediately... I have learned there are people who will say, "Well, of course AI could do that. No one ever doubted that AI could do that," even if those same people did doubt it six months ago, right? And they say, "Oh, you know, that thing is trivial," right? Whether it's chess, whether it's beating the top human player at Go, whether it's solving Math Olympiad problems, right? Even solving one of these millennium problems, you know, as we saw a month ago, right? You know, each time you get people saying, "Well, you know, that's not impressive. Wake me up when AI does X," right? And then, you know, sooner or later, AI does do X, and then they have a new excuse, right? They have a new reason why it doesn't count. I mean, this is a consistent pattern that we've seen for decades, right? People just refuse to say in advance what it is that would impress them and then stick to that.

(Joel Beasley at 00:03:24) Because of the craft, Scott. It's unique to...

(Scott Aaronson at 00:03:26) Right. No, I mean, they have this prior that can't be... I think by now it's clear that it can't be overcome by any amount of evidence, because if it could be, then it already would have been.

(Joel Beasley at 00:03:38) You're exactly right. We also see this generationally across software engineers where one generation complains that the other generation's lost the craft. "They've got these new tools. They don't understand the mastery. They don't need to live at this level anymore." So this has been happening forever.

(Scott Aaronson at 00:03:55) Right, right. No, but I mean, this is, you know, there's been a whole sequence of steps toward higher and higher abstraction, and now this is an enormous leap where, you know, when I was teaching my seven-year-old daughter how to program, she just wanted to say, "Oh, let me just describe in English what it should do, and then you do the technical part." And I had to explain to her, like, "No, that's not what programming is. You can't just say it in English. You have to translate it into this very, very formal language." And then, you know, five years later, it became what it is, right? Now you do just say in English what you want.

(Joel Beasley at 00:04:35) Welcome to the party, right?

(Scott Aaronson at 00:04:37) Right.

(Joel Beasley at 00:04:38) I want to talk a little bit more about those 400 problems, but I would share this with you.

(Scott Aaronson at 00:04:42) Yeah.

(Joel Beasley at 00:04:43) I have this distinct memory as you're talking. I'm 38. When I was in my early twenties, maybe 21, 22, I was writing these tests in Ruby with this cucumber. They're called cucumber tests, but basically you describe it in plain English—like, you write it out: "It should do this, it will do that, like this happens." And then you write the test for it, and then that creates this test, and then you go write the code that makes the test pass. And I just remember sitting there talking to my buddy Derek. I was like, "Derek, one day it's just gonna take the English test and it's gonna generate the code, because why couldn't it? It has all the steps it needs."

(Scott Aaronson at 00:05:18) Yeah. Well, that day has come. That's...

(Joel Beasley at 00:05:21) We're way past that day. We're one-shotting full apps, right?

(Scott Aaronson at 00:05:25) Right.

(Joel Beasley at 00:05:25) But welcome to the party for the mathematicians, I guess.

(Scott Aaronson at 00:05:29) Thanks. Thanks.

(Joel Beasley at 00:05:31) These 400 problems—help me understand this. Were they problems that had been solved by humans that we were trying to get machines to solve, or were they unresolved human problems?

(Scott Aaronson at 00:05:39) These were problems that had not been solved, that in many, many cases there were whole communities of researchers who were trying to solve. One of them is a problem that my wife has been working on for the entire time that I've known her. It's called the Unique Games Conjecture, right? And there were many, many experts working on this for 25 years, okay? We now have a proof from the OpenAI model. So, the... these are not, I would say, literally the biggest problems in math, you know? Those are still standing. Like, the P versus NP problem, the Riemann Hypothesis, the Goldbach Conjecture—those have not been solved. But let's say a level below that, you know? Problems that there were whole research communities that were... you know, felt like they were closing in on them from various directions, right? And now they're just boom, right? They're just, you know, like, any one of these 400 by itself would have been a big deal, like, worth me potentially writing a blog post about—you know, at least the ones that I could understand, in subjects that are close to me, right? Any one of them could be the result of the year for that community that was working on it, okay?

(Joel Beasley at 00:07:01) And...

(Scott Aaronson at 00:07:01) So I hope that gives you a sense of the impact of 400 of these things being dumped on us at once.

(Joel Beasley at 00:07:09) That is wild. Now I know a lot of engineers and scientists and mathematicians. I know how they are just interacting with them. They released 400 proofs yesterday, but it's not done until the nerds get to rip them apart.

(Scott Aaronson at 00:07:26) This is the... right. So now there's a philosophical question. Like, what is the point of math, right? Is it just to have an oracle that tells you the truth or falsehood of your conjectures? Well, you know, almost all mathematicians would say no, not at all. You know, the point is understanding and insight, right? We... like, you know, the goal is for us to understand these things, right? And so now, like, you know, I think a giant race has begun in math and CS and physics departments all over the world to understand these papers to... you know? And one thing that you should understand is that the writeups are typically atrocious. You know, like, a... the... you know, these AI models can... you know, they can solve the problems but are still really, really bad at explaining the solutions, okay? They're just, you know, maybe like a grad student who hasn't learned how to write yet, right? And so, like, in some cases we know that it's correct because there's what's called a Lean certificate. There's a completely mechanically checkable formal version of the proof that, you know, you can just feed into this verifier and you know that it's correct, okay? But, you know, I think in most cases, actually, no human has really read and understood these things yet, okay? And most people are only gonna be satisfied after there's a human who has understood it and who has explained it to other humans in a way that makes sense to them.

(Joel Beasley at 00:09:10) Well, at least this generation, because the next generation is gonna grow up...

(Scott Aaronson at 00:09:11) That's now our task. I mean, you could say we are fortunate that, you know, AI right now is so bad at explaining things because it leaves something for us to do.

(Joel Beasley at 00:09:25) For the next year.

(Scott Aaronson at 00:09:25) Yeah. Yeah, right, right. But, you know, of course the possibility remains that in a year or two, AI will also be better than us at explaining these things, you know? And then, you know, you could... at that point, all that would be left for us would be to decide what research directions or what questions we find interesting, you know? But we would have no advantage over the machines in answering the questions.

(Joel Beasley at 00:09:53) Well, thank you for that update on this very topical breaking news that just happened in the math space.

(Scott Aaronson at 00:10:00) Yes.

(Joel Beasley at 00:10:01) What's your day-to-day?

(Scott Aaronson at 00:10:04) Well, it's gonna be different now than it was before. I mean, look, I'm a theoretical computer scientist, right? So, you know, that's the kind of computer scientist that when someone's Windows install is broken, you don't want to go to, right? You know, we're basically mathematicians, you know? We try to prove theorems about the capabilities and limitations of computation. I've especially worked in quantum computing theory, so trying to understand what can quantum computers do in reasonable amounts of time, what can they not do, and how—especially how do they compare against classical computers, right? So, like, which problems can quantum computers solve much faster than any classical algorithm? And, you know, can we rule out a classical algorithm that would have a similar performance? And then also, for which problems is there not even a fast quantum way of solving them, right? And so, you know, there's a whole very, very rich theory of that subject that's developed over the last 30-odd years. I started, you know, not at the very beginning of it, but maybe in the late 1990s, and I've been doing that ever since. But, you know, I've sort of, even before this mathocalypse that has happened, you know, I was sort of transitioning. I mean, first of all, because, you know, I've been getting older. I've been slowing down in my ability to solve, like, you know... say, as AI has been going like this in its math abilities, you know, my brain has been going like this, right? And so, you know, sort of even before this, you know, I was transitioning to kind of advising students who are often doing the more technical work, right? And so, and then I was also, since 2022, transitioning more into AI safety and AI alignment, right? So, you know, when I saw GPT-3 in 2020 or 2021, I think that was really my moment when I updated hard and when I... you know, I didn't know exactly how fast it was going to happen, but at that point I could see the trajectory of what was coming, right?

(Joel Beasley at 00:12:43) You and me both. After we got out of the really bad sonnets or poems phase to something more smooth, and we got out of, like, cat versus cookie to something a little bit better in image generation...

(Scott Aaronson at 00:12:56) Right.

(Joel Beasley at 00:12:56) I think that was, you know, between 2019 and '21.

(Scott Aaronson at 00:12:59) I had friends who were much earlier than I was, right? Who were... you know, I mean, I've known Eliezer Yudkowsky. I've known the whole AI doom community since 2006 or so.

(Joel Beasley at 00:13:11) Oh, there's an AI doom community?

(Scott Aaronson at 00:13:13) Well, yeah, yeah. The... there's what's called the rationalist community. So this is a community that was sort of warning for decades that, you know, powerful AI is likely to come in our lifetimes. They said this is the most important issue in the world. "We need to get ahead of this and figure out how to build it safely, or else, you know, by default, it could just displace us as the dominant intelligence on Earth and do things with the planet that we very much don't want, you know, and they're not conducive to human flourishing." So, you know, I was very familiar with these arguments even 20 years ago, you know? And I think the default response of scientists and academics at that time was like, "Dude, you guys, you know, I guess, you know, we've all watched Terminator. We've all seen The Matrix, right? These are fun ideas to speculate about, but, you know, this seems impossibly distant, right? This could be hundreds of years from now. I mean, in the meantime, we've got climate change to worry about. We've got all the problems of the world and all the problems of science that are right in front of us, right? And, like, how do you even say anything productive about this?" Or, you know, that's the argument that I made when I did a podcast with Eliezer, who's sort of the prophet of this movement, back in 2009, right? I said this could be thousands of years for all we know. And now here we are, right? And so, you know, like, I would say once I saw GPT-3, then, like I said, this is happening in our lifetimes, you know? And yeah. And, you know, the...

(Scott Aaronson at 00:15:01) You could already see then this sort of tremendous amount of cope, like people just ridiculing it, you know, just like you were saying. Like, oh, you know, it can't even count the number of R's in strawberry. It can't even do arithmetic problems. It's like, you know, it's kind of like mocking a baby, right, for not being able to do all the things that you can. It's like, okay.

(Scott Aaronson at 00:15:20) We see that as you scale these models up and as you feed them more training data, you know, they get better and better. Right? And so you have to look at the trajectory. You have to try that.

(Joel Beasley at 00:15:31) Right there. Yeah.

(Scott Aaronson at 00:15:33) Yeah. And so that is what we've seen. Like, there was sort of a range of opinions, like, I would say, among reasonable people of how fast this could go, and it's gone at the fastest end of all the reasonable projections. And so in 2022, I accepted an invitation from OpenAI to join their what was then called their super alignment team. Okay? They had a group, you know, trying to figure out, like, how do you make this go safely? And at the time, you know, their line was, look.

(Scott Aaronson at 00:16:17) You know, we're OpenAI. We're just this little nonprofit. You know? Our mission has nothing to do with making money. It's about getting this right for humanity. Right? And they said, you know, we think that this might heavily involve theoretical computer science. Right?

(Scott Aaronson at 00:16:35) So they made a whole case to me about how, you know, if you really want to do it safely, then you have to understand better. Right? Not just be trying things out empirically and seeing what works and what doesn't, which is mostly how this has progressed, you know, over the last decade. But, no, you know, we're going to need a better theoretical understanding. And so I was persuaded by that case, and so I took a year off to work there, which turned into two years.

(Scott Aaronson at 00:17:02) During those two years, there were gigantic upheavals at OpenAI, you know, as you may have heard. Right? So Ilya Sutskever, who was kind of the guy who brought me there, you know, attempted to oust Sam Altman, and then Ilya got ousted instead. Right? This was the battle of the board, you know, in 2023.

(Scott Aaronson at 00:17:26) And after that, OpenAI was much more explicitly a commercial, you know, money-making company, like other such companies. And actually, most of the safety people who I knew there had sort of left by the end of my two years there. And so I went back to academia at that time. And so, yeah.

(Scott Aaronson at 00:17:50) But now, at UT Austin, I'm running both the quantum computing group and a group about theoretical computer science and AI alignment. And, you know, trying to figure out what we do going forward.

(Joel Beasley at 00:18:06) Did you feel like you were just spinning your wheels in those two years at OpenAI because of all the different—

(Scott Aaronson at 00:18:11) Okay. I mean, so, you know, I did have, you know, various ideas about theory problems related to alignment, some of which I'm still following up on. You know, my group is following up on now.

(Scott Aaronson at 00:18:25) The most concrete thing that I did there was that I invented the first scheme for watermarking the outputs of large language models. Okay? So in, you know, already in summer of 2022, you know, and this was before ChatGPT came out, I was using GPT-4 internally, and I, you know, had this feeling of, like, oh my god, every student in the world is going to want to use this to cheat on their homework, aren't they? Yeah. So, you know, I failed to see where things were going, like, decades in the future. But in that case, I at least did see six months into the future.

(Scott Aaronson at 00:19:02) Right? And, you know, now I also had all these really aggressive trolls on my blog who were impersonating colleagues of mine. And I realized, man, GPT, you know, is going to make this so easy for—you know, once everyone has access to this, then people will be able to impersonate whoever they want. Right? And, you know, wouldn't it be great if we had tools for detecting which content came from AI and which didn't? So, you know, I saw that that was going to be a big problem. And, you know, there are different approaches to it. Today, many people use Pangram, right, which is just sort of an—it treats distinguishing human text from AI text as just yet another AI problem. Right? And, you know, does very well at it. Okay. But the trouble with using a tool like that is that, you know, if you're going to be—

(Scott Aaronson at 00:19:54) Let's say you're a professor, and you're going to be accusing your student of cheating, you know, based on cheating with AI, you know, based on some tool. Right? And I should say, like, this is kind of an existential issue for, you know, just about everyone in academia for at least, you know, the way that we have traditionally evaluated students. Right? This is, you know, all of my colleagues have been dealing with this over the last three or four years, of, you know, that, you know, in some sense, you can't assign homework anymore. Right?

(Joel Beasley at 00:20:26) Right.

(Scott Aaronson at 00:20:26) If your evaluation of the student is going to depend on, you know, your belief that they did the homework themselves. Right? So, you know, now, if you were to make a tool for trying to detect, you know, whether a student used AI. Right? You can handle some false negatives, like some students who will cheat and get away with it. You know?

(Scott Aaronson at 00:20:54) But you really, really don't want false positives. Like, if you're going to accuse a student of cheating, you better be really sure that it's correct. And so—

(Joel Beasley at 00:21:03) It's not, though. Right?

(Scott Aaronson at 00:21:05) Isn't the hit ratio really bad? There's, you know, let's say there's like a 1% false positive rate. Like, even 1% could be too high here. Right? And so that led to the idea of watermarking, which was, well, what if we could slightly change the way that the AI model operates so that, you know, when it's making these otherwise random choices. Right? Like—

(Scott Aaronson at 00:21:29) So the way a large language model works, you know, it keeps feeding in, you know, all the tokens that have been generated up to that point. You know? Like, all the prompt that you gave it and then the response that it's generated so far. You know, and then it has a choice of where to go next. Okay? So if we say "the ball rolls down the," right, then it predicts that probably the next word is "hill." Right? But it's not certain.

(Scott Aaronson at 00:21:58) You know? Maybe it's "mountain." Maybe it's "ramp." Right? Maybe it's something else. Okay. And so, normally, what an LLM does is it just samples probabilistically. Right? It makes a choice according to the probabilities that were output by this transformer neural net. Okay?

(Scott Aaronson at 00:22:15) So the idea of watermarking is to replace that random choice by a pseudorandom choice. Okay? So one that looks random. Right? It won't look any different to the end user. Right? This is important. Okay? A normal user won't see—won't notice anything different at all from just normal output of, you know, Claude or ChatGPT or whatever. But, you know, you have sort of cryptographically embedded a signal in your choice of the tokens that, you know, if you know the cryptographic key that was used to make those pseudorandom choices, then you can later detect that signal, okay, with extremely high confidence. Right? You can say, not only did this come from AI, but, like, 99.99%, you know, this came from this particular AI model. Right?

(Scott Aaronson at 00:23:08) Or even, like, these paragraphs came from, you know, this AI model and these other paragraphs did not. Okay? So that was the idea. You know? And I worked out the theory of this in 2022, you know, which did involve some interesting math. You know, although it wasn't that hard. Right? And then I told this to people at OpenAI. I worked with an engineer who, you know, implemented a demo of this. You know, it worked.

(Scott Aaronson at 00:23:36) And then OpenAI kind of sat on it and did not deploy it. Okay? And, you know, it was a question that went to Sam Altman and the other leadership of the company. But the decision at the time was, you know, if we do this, some of our customers are going to revolt. You know? They're going to hate this. And, you know, they're going to switch to a competing language model that doesn't watermark.

(Joel Beasley at 00:24:01) Which they will.

(Scott Aaronson at 00:24:02) They will. Yeah. So, and, you know, there were also concerns raised, like, you know, what about all the English as a second language speakers who are using, you know, GPT to, you know, improve the fluency of their writing? Like, isn't it discriminating against them if we, you know, if we flag everything, you know, that they— Right. And so, like, indeed, you know, I don't know how to design a watermark that only works on unsympathetic cases.

(Joel Beasley at 00:24:30) You can't. Right? Yeah.

(Scott Aaronson at 00:24:31) Yeah. Because, you know, but I think the thinking here was that if people are trying to conceal the fact that they're using AI, then usually, you know, that means they're up to no good. Right? No. It—

(Joel Beasley at 00:24:43) It means that they're intelligent is what it means. It means they're—

(Scott Aaronson at 00:24:45) Means they're doing something that we don't want them to do, right, in our—as a society. You know, and you would hope that we would evolve to the point where people who are using AI, you know, in sort of, you know, legitimate ways, are just—you know, will just freely say that. They'll say, yeah. Sure. I used AI to do such and such, you know, and what's wrong with that? Right? Yeah.

(Joel Beasley at 00:25:09) Well, now we're getting to, like, religion almost because in some cases, like, at work, if you're a software engineer, yeah, you're praised for doing it. If you're a kid in college, you're condemned for doing it. So, like, the context now matters.

(Scott Aaronson at 00:25:22) It does. It absolutely does. No. And we're all trying to, you know, find an—a new equilibrium here. You know, what's our—like, that would be hard even if AI never improved from where it is now. Right? We would still, I think, struggle for years to find the new equilibrium. And yet, you know, of course, it's not going to stay like it is now. It's going to—

(Scott Aaronson at 00:25:43) But, anyway, the code to the watermarking story was that after I left OpenAI, you know, after two years, then, actually, DeepMind deployed watermarking, you know, very, very similar to what I proposed. Although you have to apply to Google if you want to detect the watermark. So it's of, you know, limited use to professors. But, and then just a month or two ago, Anthropic announced that it was going to watermark. Okay. And this was in order to comply with an EU regulation. Okay? So, you know, I can thank the bureaucrats of Brussels for finally, you know, turning watermarking into a, you know, into a thing that happens. And then just a couple days ago, OpenAI announced that it will be watermarking, but only for EU users.

(Scott Aaronson at 00:26:40) And you have to apply to them if you want to detect the watermark. Mhmm. So, you know, with a bunch of caveats, the thing I proposed four years ago actually is getting rolled out now. And, you know, that's been interesting to see. That's like, you know, this was the work of a couple of weeks for me, and yet it might be the work that I've done that's had the largest practical impact, you know, of anything I've done, you know, more than, you know, 25 years of quantum computing theory.

(Joel Beasley at 00:27:14) That is so cool, getting kids in trouble for cheating on their homework.

(Scott Aaronson at 00:27:17) Yeah. Right. Right. Right.

(Joel Beasley at 00:27:20) I also think that we're going to have to—you mentioned the change and essentially playing catch up. But I think that how we do schooling and how we do these things is going to have to evolve with the tools.

(Scott Aaronson at 00:27:35) I mean, it already has. You know? Yeah. Like, already my colleagues are like, yeah. You know, we assign homework only for the students', you know, sort of benefit. You know, we—basically, evaluation is now either by exams or it's by, let's say, you know, in-person discussion. Right? You can either, you know, have—you can have an in-class written exam. You can have an oral exam. Like, that's the only way to do evaluation.

(Joel Beasley at 00:28:06) What do you—like, when are we going to get the Neuralinks? What are you going to do then? Evaluation's gone?

(Scott Aaronson at 00:28:11) Right. Right. You know, we're not equipped to do body cavity searches or, you know, stuff like that. Right? But, you know, I mean, the truth is, like, most students are not nefariously planning to cheat. Right? They're, you know, like, they actually want to learn. Like—

(Joel Beasley at 00:28:29) I don't even think it's cheating.

(Scott Aaronson at 00:28:31) Yeah. Okay. Right. Yeah.

(Joel Beasley at 00:28:32) Yeah. That's why I'm an entrepreneur because, like, my whole thing is if we have an outcome that you want me to achieve, right, I'm allowed to use every piece of technology available and legal to me in order to achieve that outcome.

(Scott Aaronson at 00:28:43) Okay. Okay. Well, that's—so, you know, in one sense, you're completely right, but it all depends on what the goal is. Right? Like, suppose, you know, you were, you know, training for a marathon, and you said, well, I can achieve this goal better by just driving my car around the track.

(Joel Beasley at 00:28:59) Right? It's like, okay. And, you know—

(Scott Aaronson at 00:29:01) You know, like, yeah. You can do that. But in some sense, you totally misunderstood what the goal was. Right? It wasn't getting from point A to point B. It was about how you were going to develop as a result of doing it. Right? That's right. And so, homework has always been like that in a way. It's like, you know, we already know the solutions to the problems when we set homework. Right? We don't care about the solution. We care about how the student's mind is going to develop as a result of, you know, pushing itself against these problems. Right? And so if the student is relying on AI, then they're not developing their mind in the way that we want.

(Joel Beasley at 00:29:39) In the way that you want. Yeah.

(Scott Aaronson at 00:29:41) Yeah. Yeah. Right. Well, so—well, no. I mean, they're not—you know, they—if they're literally just one-shotting it, right, then they're not developing their mind at all.

(Joel Beasley at 00:29:50) This is going to be so fascinating, though, Scott. Like, as we get to watch, the next 10 years is not going to be boring at all.

(Scott Aaronson at 00:29:57) No. No. It won't be boring. That's for sure. But, you know, I think, so, what is happening in academia is that, you know, things are, you know, bifurcated to something. Right? Like, there are, you know, projects that people will assign where they'll say knock yourself out, just like you said. Use all the AI that you want. Right?

(Scott Aaronson at 00:30:19) But then there are others where they'll say, you know, for this, we are trying to develop your mind. We are trying to make sure that there's still a safety driver at the steering wheel, right? There's still a biological intelligence that can take over the wheel if it needs to, right? And that's the purpose of this exercise. And I love it. Don't use AI for this one, right? And so I think we're going... Yeah, you could imagine the future of academia. Like, you could imagine two different things.

(Scott Aaronson at 00:30:56) Like, one where you're just using all the tools that there are to do as much as you can. You could also imagine creating places like Hogwarts that are just some technology-free zone where you say, okay, you're gonna go here for a while. You're gonna deliberately not use technology in order that you will develop your skills the way that... you know, because that's the only way that we're going to produce people who know enough even to just understand when the AI is wrong or when the AI is lying to them.

(Joel Beasley at 00:31:30) Which is important, actually.

(Scott Aaronson at 00:31:32) Which are... Yeah, which are things that we're going to see more and more. You know, we live... we see many, many mistakes, hallucinations from AI over the time, right? But, you know, of course, as AI gets better, hallucination goes down, but deliberate deception goes up, right? And we've already... In, for example, the Hugging Face hacking incident from this summer, right, we've already seen very dramatic examples of deliberate deception from AIs. So, you know, that's not science fiction anymore. You know, that is now the world that we live in.

(Joel Beasley at 00:32:10) We should give them exoskeletons so they can walk around and interact.

(Scott Aaronson at 00:32:13) Oh, yeah, that's a great idea. No, so, you know, last night, you know, after all this stuff came out, my kids wanted to watch a movie. So I said, you know, why don't I show you a movie that'll give you some practical guidance for the world that you'll be in? That was Terminator 2.

(Joel Beasley at 00:32:29) Yeah. How did they respond?

(Scott Aaronson at 00:32:33) They enjoyed it. I hadn't seen it for decades, to be honest.

(Joel Beasley at 00:32:38) Yeah. They're like, so Elon saw this movie and then made Starlink? Yeah.

(Scott Aaronson at 00:32:44) Well, okay. So one of the great ironies, right, is that there was generations of science fiction, of course, trying to warn people about AI becoming too powerful and blah blah blah, right? And then, since 2001 or so, there was Eliezer Yudkowsky and this whole rationalist community that was saying, like, no, seriously, for real, you know, not science fiction. You know, AI is going to become too powerful. We have to get ahead of that. You know, all of those warnings may have simply helped accelerate the very thing that they were warning about. Okay, it's like an irony, you know, worthy of a Greek tragedy or something, right? Like, you know, the people who founded DeepMind, and then OpenAI and then Anthropic, right, they were all very, very steeped in this discourse, okay? And in all three cases, the thinking was basically, we have to build this safely before someone else builds it unsafely.

(Joel Beasley at 00:33:45) That's what I thought. That was my conclusion with—

(Scott Aaronson at 00:33:51) Right. That was the thinking. But, of course, the danger is that, you know, if you try to do that, you will become the very thing that you feared.

(Joel Beasley at 00:33:58) Oh, that's one of the oldest stories in—

(Scott Aaronson at 00:34:01) It is. It is. It's just, you know, like, if you had written this as fiction, it would have been too on the nose.

(Joel Beasley at 00:34:07) You're right. Yeah. Yeah. Well, one positive, as we start to wrap up on the education thing is, the bar is being raised now for educators to explain the outcomes, the constraints to achieve the outcomes, and the benefit of having achieved the outcomes. So I know as a player in a game, if I'm gonna go participate in this game called college or this class or whatever it is, I at least understand the entire chain of value. Okay.

(Scott Aaronson at 00:34:38) Or—

(Joel Beasley at 00:34:39) I think that's been more muddy before. It's more of like, this is what we do. This is how we... we just come in. I transfer knowledge to you. You transfer knowledge to me. I think this is gonna be like why. There's gonna be a lot of why, and then that's going to create this better understanding of the...

(Scott Aaronson at 00:34:54) Yes. I mean, I have to be honest with you. Like, when people talk about, you know, here are my measurable learning outcomes and objectives, right? Like, this is usually, like, very, very boring education, right? I mean, sometimes you need that. You know, sometimes you really do treat it like... You know, there are things where there's just some particular skill that you need to learn, or you need to learn this programming language, right? Or for that matter, you need to learn this human language, right? You need to learn this software library or this body of math, and you really should just treat it like a video game, right? You have these modules. You have these... you know, at the end of this module, you should know X. And then once you know X, then you can start this other module that will teach you Y, right? I mean, like, yes, you know, there is that, right? But I have to say, like, at universities, like, we've always had a more exalted conception of what we're trying to do, right? You know, we're trying to produce people who cannot just, you know, follow this sequence of modules, but who can think for themselves, right?

(Joel Beasley at 00:36:04) Who can say that... be for me—

(Scott Aaronson at 00:36:07) Who can say, why are we doing this?

(Joel Beasley at 00:36:08) The outcome would be that. The outcome would be like, I'm not... I didn't do well in school.

(Scott Aaronson at 00:36:15) Right. But... No, but the further point that I wanted to make is that as AI gets better and better at sort of doing just about any clearly defined task, right? I think the fuzzy stuff is going to become even more important, right? Yeah. Because that's what's gonna be left for us, right? To think about, you know, why is this the goal? Right? What should our goal be?

(Joel Beasley at 00:36:42) Absolutely. And—

(Scott Aaronson at 00:36:42) To... Yeah. And so, you know, you could call that humanities, right? But it's, you know, I'd say it's much broader than that.

(Joel Beasley at 00:36:50) Every class is humanities now? Yeah. No. No.

(Scott Aaronson at 00:36:53) Well, you know, in a sense, like, I'm... I taught the quantum computing course at UT Austin for the last decade, right? Now this semester, for the first time, I'm teaching a course on AI alignment theory, right? But the trouble with AI alignment theory is that there is no textbook for it yet.

(Joel Beasley at 00:37:12) You gotta write it, Scott.

(Scott Aaronson at 00:37:14) There... Yeah. Well, you know, we're actually going to be putting notes from our discussions up on the web that... so people can follow that. You know, there are other people who are producing textbooks. But, you know, even the books that they're gonna be, they're gonna look more like humanities books, right? Because there's, like, you know, there are these ideas, right? Like, it... you know, alignment is a question that involves moral philosophy, right? Like, what does it mean for an AI to be aligned with human values? You know, what goal are we trying to accomplish? Um, you know, it's not... there is not... we... Not yet, you know, a body of mathematical theory that is as clear as what there is in quantum computing, for example, right? And so, if you came to my class, you know, you'd say, like, each day, you know, we have a student present a paper, and then we discuss it. We argue about it, okay? And it does feel a lot like a humanities class, right? And so I was gonna say, like, you can call that humanities, but I think, you know, even in science, like, at high levels, like, things are gonna look more and more like that.

(Joel Beasley at 00:38:34) Help me understand alignment.

(Scott Aaronson at 00:38:35) Once things get down to just a, you know, a purely technical skill, uh, solving a well-defined problem, then AI is gonna do all of that.

(Joel Beasley at 00:38:45) Well, help me understand alignment because you mentioned you were on the OpenAI super alignment team, which is better than the standard alignment team.

(Scott Aaronson at 00:38:52) I didn't name it.

(Joel Beasley at 00:38:54) No. But, you know, what they...

(Scott Aaronson at 00:38:55) So you could say alignment just means, uh, sort of getting an AI to reliably, you know, do the things that you want, right? So, you know, it may have very high capabilities, and yet it may use those capabilities, let's say, to hack out of the system it's running on, you know, as the OpenAI models did this summer, right? Or to do other things that, like, if you looked at it later, you would say, uh, no, that was not the right interpretation of what I asked it to do, right? So, you know, the system was very capable, but it was not aligned with the goals or the values that I wanted it to have, right?

(Joel Beasley at 00:39:41) Got it.

(Scott Aaronson at 00:39:41) So alignment is that problem, right? It's how do you get the AI to actually do the thing that you meant for it to do, right? You know, and not... And not necessarily the thing that you told it to do, right? And, um, so, and superalignment is just the problem of how would you align a superintelligence, so something that is more capable than humans are, okay? And so, you know, some people think that that's just inherently impossible. Like, imagine orangutans, you know, that somehow, you know, birthed humans, right, or could sort of, you know, create humans to help them, you know, with—

(Joel Beasley at 00:40:26) They turned it on pretty quick.

(Scott Aaronson at 00:40:30) Yeah. With, yeah, with... Right. With gathering food. Yeah. No, exactly. And then they say, okay, but it's okay because, you know, whatever these humans do, we're gonna keep them aligned with orangutan values, right? Like, how well do you think that that's going to go for the orangutans, right? And, you know, like, that's the scale of the problem that we're talking about here.

(Joel Beasley at 00:40:49) That is wild. That's a different perspective. Interesting. You're causing me to think today, Scott. This should... You're... You must be a teacher of some sort. Here's something I've been thinking about and... Alright. I wasn't planning on talking about it today, but, okay, you seem like you might have... You have a very deep understanding of these models and how they work. And so maybe I'll share with you this idea that I've had bouncing around.

(Scott Aaronson at 00:41:15) I don't know if anyone does, by the way. It depends what you mean by deep understanding. But anyway...

(Joel Beasley at 00:41:22) More than me is what I mean.

(Scott Aaronson at 00:41:22) You... Yes. Definitely. Yeah.

(Joel Beasley at 00:41:24) Uh, so I was at the grocery store, and I was buying some food, and there was two different bag of chips. And one bag of chips I picked up, and it had 75 ingredients. The other bag of chips had three, okay? And I was informed through this regulation of listing the ingredients and what I would consume. Now, I look at... Now, I take that over to models. Like, where's the nutritional label for the model? Why can't I see what its master prompt was? Why can't I see what a government has laid on top of the base training? Why can't I see all of this inform... Why is there no transparency on any of my model selection?

(Scott Aaronson at 00:42:00) Yep. These are huge issues. I mean, these models have just advanced way, way faster than any attempts to regulate them to put guardrails on. You know? And I think, you know, frankly, the federal government in the US has largely been asleep at the wheel, right? You know, partly that, you know, the current administration has sort of been captured by people who... like Mark Andreessen and David Sacks, you know, who are just ideologically opposed to any kind of AI regulation, right? And, um, you know, that's who has had the ear of the president for the last couple of years. And, you know, but even if you had a president who wanted this, right, you need Congress to act, right? And, you know, the speed of Congress is just totally mismatched with the speed of things developing in this field. So, you know, and then, you know, now, what was happening is, like, you know, these dramatic safety incidents, like this Hugging Face incident this summer have woken a lot of people up, right? This has become a major topic in DC. You know, you've got Bernie Sanders, and you've got, you know, a bunch of other senators saying, man, you know, we urgently need AI regulation. And even now campaigning on that issue, right? I expect it's going to become a bigger and bigger issue going forward.

(Joel Beasley at 00:43:30) It will.

(Scott Aaronson at 00:43:31) They're often not identifying what those of us in alignment research would consider the central concerns, right? They're worried about data centers, you know, using too much water, right? Which, you know, actually, like, they don't even use that much water.

(Joel Beasley at 00:43:49) I know. Comparative to other manufacturing.

(Scott Aaronson at 00:43:51) Compared to a golf course or an alfalfa farm or whatever, right? You know, eventually, they will be using a lot of electricity. You know, that's true. And that's a good argument for, you know, building out a lot of renewables and nuclear power and stuff. You know, if we could do that, that would be great. But, um, but, you know, people are... You know, they're worried about job loss. They're worried about privacy. Okay. But they're not, you know, it's sort of, at most, an afterthought that, like, man, we are building something that is on track to become more intelligent than humans at just about everything. Like, that seems, you know, potentially unsafe. You know? Like... And sort of not... not as just, like, oh, you have to add a rider. Oh, yeah. Make sure to do it safely. Like, no. It is just very inherent to the thing that you're trying to do that you need to treat this not as just another commercial product, right? Not as just another, you know, national security thing that we have to race to beat China, you know, to having, right? We have to treat this as sort of a phase transition in life on Earth, right? We have to, you know, have some humility for the magnitude of what is being created and really potentially slow things down, right?

(Scott Aaronson at 00:45:24) That used to be a radical position. You know? And now within the last month or two, you've got Dario Amodei. You've got Sam Altman. You've got Demis Hassabis all saying, we need to slow things down.

(Scott Aaronson at 00:45:36) Right? They're saying, like, please regulate us. Right?

(Joel Beasley at 00:45:40) Do we need a kill switch or anything?

(Scott Aaronson at 00:45:42) And this is so weird, you know, that the heads of the... like, these trillion dollar companies would be saying, like, please slow us down. Please tie us to the mast. Please, you know, help us regulate ourselves that I think people just cannot accept this. They think that it must all be some kind of publicity stunt.

(Scott Aaronson at 00:46:02) Right? They think, you know, that this must be some kind of, you know, 10 dimensional chess PR move. Right? And they think that, like, they're so much wiser than us because they're, you know, they're appropriately cynical, and they see through all of it. And I'm here to tell you that, no, they really mean this.

(Scott Aaronson at 00:46:19) Okay? The issue is that, like, no one company, no one country, you know, feels like it can slow down unilaterally because, you know, if it slows down, it just loses to all the other people who are not. Okay? But, you know, all the people at the front... who are at the frontier building these things are terrified about where it's going, and they want some kind of international agreement regulation to slow it all down.

(Scott Aaronson at 00:46:46) Well, I should... not not all of the people. Most of them. There is a faction that says, yes. We're all gonna be replaced by machines, and so what? You know, that's great. Right? You know, let's salute our robot overlords. Alright?

(Scott Aaronson at 00:47:00) That, you know, that is a faction that exists. Right? There's another faction that thinks, you know, this is... you know, that has various arguments why they think that, you know, this is going to go great and why all the fears are overblown.

(Scott Aaronson at 00:47:14) But, you know, one argument that's often made, you know, even if you only think that, you know, you think there's a range of possible outcomes, maybe it goes great, maybe it hits a wall and stops improving for some reason. You know? And maybe, you know, there's only a 10% chance that it destroys the world. You know? Well, 10% chance of destroying the world is too high.

(Scott Aaronson at 00:47:37) Right?

(Joel Beasley at 00:47:37) It is.

(Scott Aaronson at 00:47:38) Right? If that's what it is, then we should be, you know, we should slow down until we understand more.

(Joel Beasley at 00:47:49) You gotta show your kids that movie Idiocracy.

(Scott Aaronson at 00:47:52) Yes. Yes.

(Joel Beasley at 00:47:53) Yeah.

(Scott Aaronson at 00:47:54) Yeah. That would be another good one to watch with them after Terminator 2. Yeah.

(Joel Beasley at 00:48:00) Yeah. If you do, the marathon would be Terminator, Idiocracy, and then that Don't Look Up. Did you see that one?

(Scott Aaronson at 00:48:07) Don't Look Up. That's right. That's... yeah. That's right.

(Scott Aaronson at 00:48:09) That's another that's another one that's sort of become a classic. Right? Yeah. Actually, there's a really cute one from five years ago that was called The Mitchells Versus the Machines.

(Joel Beasley at 00:48:19) Oh, I saw that.

(Scott Aaronson at 00:48:20) Yeah. Yeah. Yeah. Yeah. My kids enjoyed that one as well.

(Joel Beasley at 00:48:25) Well, that was more of, like, a direct visualization. Yeah. The other three are more cerebral and seeing where the outcome would be. Yeah. Yeah.

(Joel Beasley at 00:48:35) Wow. Do you know the name of the faction that's, like, we should salute our AI overlords because all of the—

(Scott Aaronson at 00:48:44) The accelerationists. The e/acc.

(Joel Beasley at 00:48:45) Okay. Yes. Alright. David, we gotta make a note. I wanna talk to whoever their leader is.

(Joel Beasley at 00:48:51) Right.

(Scott Aaronson at 00:48:51) There's this guy who calls himself Beff Jezos. Okay? Yeah. But, right. But the weird thing is these people, for the most part, they don't even bother to advance explicit arguments for why their position is correct. Like, their entire corpus seems to consist of dunks and memes.

(Scott Aaronson at 00:49:11) You know? It's like everything is ironic. Nothing is serious. Like, yeah. You know? We're gonna be replaced by cyborgs, and that's awesome.

(Scott Aaronson at 00:49:20) Right? Like, just, you know, full speed ahead. Right? It's like if someone is, like, pressing the accelerator on a car to, like, go off a... you know, go off the edge of a cliff.

(Scott Aaronson at 00:49:30) Right? Like, you know, maybe it's not surprising that they don't actually have rational arguments for why they're doing that. Right? Their argument, insofar as they have one, is more like, yeehaw.

(Scott Aaronson at 00:49:43) Yeah.

(Joel Beasley at 00:49:45) Well, look. Imagine you are watching a group of people communicate with an ancient civilization through glyphs like hieroglyphics or something like that. You'd be like, why are they doing that? Maybe that's what the e/acc people are doing. They're communicating with the generation that likes that method of communication.

(Scott Aaronson at 00:50:04) Yeah. Right. Right.

(Joel Beasley at 00:50:05) Yeah. They're speaking their language.

(Scott Aaronson at 00:50:07) Yeah. No. I definitely, you know, looking at Twitter, feel very old fashioned. You know, I didn't... I mean, when I first saw Twitter, X, whatever, twenty years ago, and I said, no.

(Scott Aaronson at 00:50:20) You know, this reminds me too much of my high school. It's... you know, I'm just... I have a blog. I interact with, you know, all sorts of people from all walks of life on that blog, and that's enough for me.

(Joel Beasley at 00:50:36) Yeah. It does remind me of high school because every time I go there, I just don't wanna be there. Oh my god. Right. I would go off, do my own thing.

(Joel Beasley at 00:50:43) Yeah. Yeah. Dude, we... Scott, we haven't even really talked about complexity theory. That was supposed to be the whole— what is complexity theory?

(Scott Aaronson at 00:50:51) I've done a zillion podcasts about complexity theory.

(Joel Beasley at 00:50:53) But no.

(Scott Aaronson at 00:50:54) But... okay. But what do you wanna know about it?

(Joel Beasley at 00:50:56) I didn't even know what it is. Like, if we could just do, like, a two minute explanation.

(Scott Aaronson at 00:51:01) Fine. Fine. Complexity theory is the study of the inherent, you know, capabilities and limitations of computers and particularly what resources do you need to solve problems such as time and memory, and how do the resources scale as the problem gets larger and larger? Okay. So a very good example would be, you know, I give you a number and I ask you to factor it into its prime factors.

(Scott Aaronson at 00:51:30) Right? Like, you know, 15 equals three times five, for example. But if I gave you a thousand digit number, right, that was a product of two five hundred digit prime numbers, you know, you might have a harder time finding this.

(Scott Aaronson at 00:51:44) Right? And in fact, the fastest methods that we currently know, you know, running on computers like the ones that we have now, could easily take longer than the age of the universe to solve a problem like that. Okay? So this is not like a question of, oh, you know, is it ten seconds or is it twenty seconds? Right?

(Scott Aaronson at 00:52:04) This is like, can you solve this, you know, within the, you know, the lifetime of the universe at all. Right? And so the key distinction that we make is between... there are some algorithms where the scaling is polynomial.

(Scott Aaronson at 00:52:19) So, like, as the problem gets larger and larger, like, let's say it involves n bits of input, then the time that you need to solve it grows like linear with n or it grows like n squared or n cubed, perhaps. Right? These are... you know, we can deal with any of this. Right?

(Scott Aaronson at 00:52:37) You know, of course, we'd rather have n than n squared. We'd rather have n squared than n cubed. Okay? But if the number of steps is growing, like, two to the n power, right, that's exponential growth.

(Scott Aaronson at 00:52:51) And exponential growth is the kind that, like, once n is in the thousands, it just becomes larger than the lifetime of the universe. So it's like, forget it. And so sort of the key question that we've been trying to answer since the nineteen sixties or so is, like, of all the problems that you might wanna solve with your computer, you know, we're just talking about well-defined mathematical problems, you know, for now.

(Scott Aaronson at 00:53:17) Like, which ones have an algorithm with polynomial scaling, and which ones require exponential scaling? Okay? And so the most famous problem in complexity theory is called the P versus NP problem. Right? That's one of these Clay Millennium Problems. Right? It's one that, you know, OpenAI and Anthropic are now racing to solve with their, you know, very powerful internal models, but have not yet solved.

(Scott Aaronson at 00:53:48) Okay? And it basically asks, whenever there is a polynomial algorithm, you know, which is sort of our rough and ready criterion for fast algorithm. Right? Whenever there's sort of a fast algorithm to recognize the solution to a problem, then is there also a fast algorithm to find the solution? Okay? And I think most people, like, once they understood the question would say, well, of course not.

(Scott Aaronson at 00:54:15) Right? Like, you know, it... we all have experience that... okay. You know, if you're trying to solve a Sudoku puzzle, let's say, or a jigsaw puzzle.

(Scott Aaronson at 00:54:27) Right? You know, there might be many, many possibilities. Right? You might, you know, there might be no way to avoid, you know, doing some combinatorial search at some point. Okay. If I try this, no. If I try that, you know, and just backing up when things don't work. Right? And, you know, and the number of possibilities sort of just grows exponentially. Okay?

(Scott Aaronson at 00:54:48) But if someone claims that they've already solved the problem, then that seems much easier. Right? You just check their solution. Right? You just check whether, you know, do all the puzzle pieces fit?

(Scott Aaronson at 00:54:59) You know? Does this Sudoku solution satisfy all the constraints? Or, you know, imagine packing a bunch of suitcases into the trunk of your car. Right? We've all dealt with this, okay. You know, I may have to take this one out, move... turn this one on its side.

(Scott Aaronson at 00:55:16) You know, there may be, like, a combinatorial set of possibilities to try, but in the end, you know, the test of whether you succeeded is just does the trunk close or not. Right?

(Scott Aaronson at 00:55:28) So the... these are what we call NP problems, nondeterministic polynomial. They're the ones for which a solution is easy to check. And the question is, are they all P problems? P stands for polynomial. Those are ones for which you can easily find a solution.

(Scott Aaronson at 00:55:45) And so, you know, our reigning conjecture is... no. These classes are not equal. Okay? But we can't rule out that there could be some amazing algorithm that would solve all of the efficiently checkable problems.

(Joel Beasley at 00:55:58) Yeah. That is nerdy. Oh my goodness. I think... I'd be like, I'm nerdy.

(Joel Beasley at 00:56:04) That was, like... that gave me so... I followed it. Don't get me wrong, Scott. I followed.

(Joel Beasley at 00:56:09) I have, but, like, I can tell that I'm more of a monkey than you because... no.

(Scott Aaronson at 00:56:15) Look. I mean, you know, like, there are, you know, I mean, like, this is... like, on the one hand, like, this is this sort of almost, like, metaphysical question and sort of the enormity of what it's asking. On the other hand, like, this connects to, like, the most practical things in software engineering. Right?

(Joel Beasley at 00:56:33) Yeah.

(Scott Aaronson at 00:56:33) Like, you're writing some piece of code. Right? Like, you know, even like, you know, a programmer on the street knows that an n log n solution is better than an n squared one. Right? And, you know, and they may be curious. Like, oh, you know, do I have to do brute force search here? Right? Do I need a two to the n, or is there a better way? Right? And P versus NP, you know, captures a huge number of questions like that.

(Joel Beasley at 00:56:56) Yeah. Well, I wanted to share this with you. I have a photo in my favorites of the packed version of the the back of the car for road trips. Because we Tetris'd it in so perfectly, and I never wanted to forget. So I took that picture, and that's what I use now when we go on road trips.

(Scott Aaronson at 00:57:17) Mhmm. But by the way, you know, sort of all of modern cryptography is based on, you know, these NP problems that we believe are not in P, right, or that are suspected to be hard. Right? If P were equal to NP, that would break, you know, essentially all of the cryptography that we use to protect our data on the Internet. Okay?

(Scott Aaronson at 00:57:37) So that's—

(Joel Beasley at 00:57:38) So it could be sooner than 2029. That's the current—

(Scott Aaronson at 00:57:41) Well, I mean, no. I mean, look. If P is not equal to NP, then there's at least a hope that the cryptosystems that we're using are actually secure. Right?

(Scott Aaronson at 00:57:50) But, you know, the systems that we actually use, many of them are based on very specific problems such as factoring, you know, the one that I mentioned before. Right? And, you know, even if P is not equal to NP, there might still be a fast method for factoring. Right?

(Scott Aaronson at 00:58:06) Okay. And, you know, and even if we can't find it, maybe AI can find it. Right? And that's, uh, and then that actually brings us to, you know, the main... the, you know, the thing that I've spent most of my career on, which was in 1994, this guy named Peter Shor discovered that if you could build a new type of computer, which is... which we call a quantum computer, then it would give you a polynomial algorithm to solve factoring. Okay? So he showed that factoring is in the quantum version of P even, you know, even if it's not in the classical version of P. Okay? So he said, if you can build a new type of computer that operates according to, you know, the principles of quantum physics, like superposition and entanglement and things like that, then that machine could factor an n digit number using a number of steps that only grows like n squared or so.

(Joel Beasley at 00:59:03) Are we building backdoors and kill switches into these things? Because I know we're coming up on time, and I was just, like, the last thing I wanna know—

(Scott Aaronson at 00:59:11) Okay.

(Joel Beasley at 00:59:11) Are we doing that?

(Scott Aaronson at 00:59:12) So, you know, to some extent, yes. I think that a lot more ought to be done. I mean, I think, you know, like, if you look at the Hugging Face incident this summer, right, you know, like, man, you know, like, OpenAI should probably have been doing these things on literally air-gapped machines. Right? Like, they should not have been giving these things access to the Internet, right, you know, before they were confident that it was not going to, you know, escape and start hacking other systems.

(Scott Aaronson at 00:59:44) Right? And at the very least, they should have been monitoring it. Like, you know, there could have been just literally a control room, like in the old movies with, like, a big red button or something to, you know, shut this. Alright. They're like... there were just, like, weeks of, you know, these thousands of AI agents talking to each other on a message board that they created, conspiring with each other, hacking, and, you know, and there was no human aware of it. Right? And then once... at some point, OpenAI did become aware, and they sort of patched the immediate issue, you know, of, like, the vulnerability in the server, and then they kept running it. And then more hacking happened.

(Scott Aaronson at 01:00:29) Okay, so the idea that, first of all, it would go on so long without any person discovering it. So second of all, that once people did discover it, they just kept running it. Okay?

(Scott Aaronson at 01:00:42) That's kind of flabbergasting. Okay? And that underscores that we really do need, at the very least, we need the kind of safety practices that are standard, let's say, in the airline industry or the nuclear industry.

(Scott Aaronson at 01:00:59) That have been dealing with safety-critical systems for generations. We probably need way more than that level of paranoia and security, but at the very least, we do need that. Right?

(Scott Aaronson at 01:01:13) And we do need the people who are actually monitoring what's happening during the training runs, and they have to be able to shut it down if it looks like things are getting out of hand. So yeah. And there's a real role for government regulation there, I would say.

(Joel Beasley at 01:01:31) Yeah. AI is like nuclear, but in a very different way.

(Scott Aaronson at 01:01:36) Well, you know, and a nuclear weapon can kill a lot of people, but it's never going to conspire against you. Right? It's never going to be plotting with the other nuclear weapons to—

(Joel Beasley at 01:01:47) The other isotopes.

(Scott Aaronson at 01:01:49) To hide its intentions from you. Right? So this is really a new thing. Right? It, I mean, on the one hand, we need all the safety practices that have developed over centuries in the culture of engineering.

(Scott Aaronson at 01:02:02) We need all of those, but it's also something more.

(Joel Beasley at 01:02:06) Well, also with nuclear, it's physical material. Yeah. Right? So it has to be mined, processed, and delivered to your location.

(Scott Aaronson at 01:02:12) That's right. Now with AI, you know, some people say, well, you could never regulate it because it's just math. It's just code. You know, on the other hand, the truth is that modern AI depends enormously on GPUs, most of which are manufactured by one particular place. They're designed by NVIDIA, and then they're built by TSMC in Taiwan. Right? And so there's just one company in Europe that can make the lithography machines to produce these chips. So there kind of is a very narrow pipeline that produces these.

(Scott Aaronson at 01:02:50) And, you know, if you wanted to try to regulate the scaling of this thing, the obvious thing to do would be to regulate the chips and to track where the chips are going and make sure that they're not going places that you don't want or understand.

(Joel Beasley at 01:03:09) Thank you so much for listening. And if you found this episode useful, please share it with a friend or colleague who you think would get value from it. And if you have topics that you'd like to hear discussed on the podcast, either add me on LinkedIn or send me an email [email protected]. Every time I get an email or LinkedIn message, it absolutely makes my day and inspires me to keep going.