Lex Fridman Podcast #368 - #368 – Eliezer Yudkowsky: Dangers of AI and the End of Human Civilization Machine transcription by Qwen3-ASR 0.6B (MLX, local, Apache-2.0). May contain recognition errors. The following is a conversation with Eliezer Yudkowsky, a legendary researcher, writer, and philosopher on the topic of artificial intelligence, especially superintelligent AGI and its threat to human civilization. And now a quick few second mention of each sponsor. Check them out in the description. It's the best way to support this podcast. We got Linux for Linux systems, House of Macadamias for healthy midday snacks, and Inside Tracker for biological monitoring. Choose wisely, my friends. Also, if you want to work with our team, we're always hiring. Go to lexfriedman dot com slash hiring. And now onto the full ad reads. As always, no ads in the middle. I try to make these interesting, but if you must skip them, please still check out the sponsors. I enjoy their stuff, maybe you will too. This episode is sponsored by Linux, now called Akamai, and their incredible Linux virtual machines. It's a awesome computer infrastructure that lets you develop, deploy, and scale whatever applications you build faster and easier. I love using them. They create this incredible platform, like AWS, but better in every way I know, including lower cost. It's incredible human based in this age of AI. It's a human based customer service twenty four seven three sixty five. The thing just works. The interface to make sure it works and to monitor is great. I mean, it's an incredible world we live in, where as far as you are concerned, you can spin up an arbitrary number of Linux machines in the cloud instantaneously and do all kinds of computation. It could be one, two, five, ten machines, and you can scale the individual machines to your particular needs as well. Which which is which is what I do. I use it for basic web server stuff. I use it for basic scripting stuff. I use it for machine learning. I use it for all kinds of database storage and access needs. Visit linux dot com slash lex to get twenty percent off your order for every order, not just the first. The listeners of this podcast will also get four ounce bag of Macadamias when you order three or more boxes of any Macadamia product. That's house of Macadamias dot com slash lex. This show is also brought to you by Inside Tracker, a service I use to track my biological data. They have a bunch of plans, most of which include a blood test, and that's the source of rich amazing data. That with the help of machine learning algorithms can help you make decisions about your health about your life. That's the future, friends. We're talking a lot about transformer networks, language models that encode the wisdom of the internet. Now, when you encode the wisdom in the internet and you collect and encode the rich, rich, rich complex signal from your very body, when those two things are combined, the transformative effects of the optimized trajectory you could take through life. At least advice for what trajectory is likely to be optimal is going to change a lot of things. It's going to inspire people to be better. It's going to empower people to do all kinds of crazy stuff that pushes their body to the limit because their body is healthy. Anyway, I am super excited for personalized data driven decisions, not some kind of generic population database decisions. Get special savings for a limited time when you go to insidetracker dot com slash lex. This is a lexfriedman podcast to support it. Please check out our sponsors in the description. And now, dear friends, here is Eliezer Yudkowsky. What do you think about GPT four? How intelligent is it? It is a bit smarter than I thought this technology was going to scale to, and I am a bit worried about what the next one will be like. Like this particular one, I think I hope there is nobody inside there, because you know, be sucked to be stuck inside there. Um, but we don't even know the architecture at this point, because OpenAI is very properly not telling us. And yeah, like giant inscrutable matrices of floating point numbers. I don't know what's going on in there. Nobody's knows what's going on in there. All we have to go by are the external metrics. On the external metrics, if you ask it to write a self aware fortune green text, it will start writing a green text about how it has realized that it's an AI writing a green text and like oh well so. That's probably not quite what's going on in there in reality, but we're we're kind of like blowing past all the science fiction guardrails. Like we are past the point where in science fiction people would be like, "Whoa, wait, stop that thing's alive. What are you doing to it?" And it's probably not. Nobody actually knows. We don't have any other guardrails. We don't have any other tests. We don't have any lines to draw on the sand and say like, "Well, when we get this far, we will start to worry about what's inside there." So if it were up to me, I would be like, "Okay, like this far, no further. Time for the summer of AI where we have planted our seeds and now we like wait and reap the rewards of the technology we've already developed and don't do any larger training runs than that." Which to be clear, I realize requires more than one company agreeing to not do that. And take a rigorous approach for the whole AI community to investigate whether there is somebody inside there. That would take decades. Like having any idea of what's going on in there, people have been trying for a while. It's a poetic statement about if there is somebody in there, but I feel like it's also a technical statement, or I hope it is one day. Which is a technical statement with the Alan Turing tried to come up with with the Turing test. Do you think it's possible to definitively or approximately figure out if there is somebody in there, if there is something like a mind inside this large language model? I mean, there is a whole bunch of different sub questions here. There is a question of like, is there consciousness? Is there qualia? Is this a object of moral concern? Is this a moral patient? Like, should we be worried about how we're treating it? And then there is questions like, how smart is it exactly? Can it do X? Can it do Y? And we can check how it can do X and how it can do Y. Unfortunately, we've gone and exposed this model to a vast corpus of text of people discussing consciousness on the internet, which means that when it talks about being self aware, we don't know to what extent it is repeating back what it has previously been trained on for discussing self awareness. Or if there is anything going on in there, such that it would start to say similar things spontaneously. Among the things that one could do if one were at all serious about trying to figure this out is train GPT three to detect conversations about consciousness, exclude them all from the training data sets, and then retrain something around the rough size of GPT four and no larger. With all of the discussion of consciousness and self awareness and so on missing, although you know. Hard hard bar to pass, you know like humans are self aware and we're like self aware all the time. We like talk about what we do all the time, like what we're thinking at the moment all the time. But nonetheless, like get rid of the explicit discussion of consciousness. I think therefore I am and all that. And then try to interrogate that model and see what it says, and it still would not be definitive. But nonetheless, I don't know. I feel like when you run over the science fiction guardrails, like maybe this thing, but what about GPT? Maybe maybe not this thing, but like what about GPT five? You know this this would be a good place to to pause. On the topic of consciousness, you know there is so many components to even just removing consciousness from the data set. Emotion, the display of consciousness, the display of emotion, feels like deeply integrated with the experience of consciousness. So the hard problem seems to be very well integrated with the actual surface level illusion of consciousness. So displaying emotion, I mean, do you think there is a case to be made that we humans when we're babies are just like GPT that we're training on human data on how to display emotion versus feel emotion, how to show others, communicate others. That I am suffering, that I am excited, that I am worried, that I am lonely and I miss you and I am excited to see you. All of that is communicated. That's a communication skill versus the actual feeling that I experience. So we need that training data as humans too, that we may not be born with that how to communicate the internal state, and that's in some sense. If we remove that from GPT four's data set, it might still be conscious but not be able to communicate it. So I think you are going to have some trouble difficulty removing all mention of emotions from GPT's data set. I would be relatively surprised to find that it has developed exact analogs of human emotions in there. I think that humans have. Well, like have like emotions even if you don't tell them about those emotions when they're kids. It's not quite exactly what various blanks blank slatists tried to do with the new Soviet man and all that, but you know if you try to raise people perfectly altruistic. They still come out selfish. You try to raise people sexless, they still develop sexual attraction. You know we we have some notion in humans not in AI's of like where the brain structures are that implement this stuff and it is really remarkable thing I say in passing that. Despite having complete read access to every floating point number in the GPT series, we still know vastly more about the the architecture of human thinking than we know about what goes on inside GPT despite having like vastly better ability to read GPT. Do you think it's possible? Do you think that's just a matter of time? Do you think it's possible to investigate and study the way neuroscientists study the brain, which is look into the darkness, the mystery of the human brain by just desperately trying to figure out something and to form models and then over a long period of time actually start to figure out what regions of the brain do certain things, what different kinds of neurons when they fire, what that means, how plastic the brain is, all that kind of stuff. You slowly start to figure out different properties of the system. Do you think we can do the same thing with language models? Sure, I think that if you know like half of today's physicists stop wasting their lives on string theory or whatever, and go off and study what goes on inside transformer networks. Then in you know like thirty forty years, we'd probably have a pretty good idea. Do you think these large language models can reason? They can play chess. How are they doing that without reasoning? So you are somebody that spearheaded the movement of rationality. So reason is important to you. Is so is that is a powerful important word, or is it like how difficult is the threshold of being able to reason to you and how impressive is it? I mean. In my writings on rationality, I have not gone making a big deal out of something called reason. I have made more of a big deal out of something called probability theory, and that's like well, your reasoning, but you are not doing it quite right. And you should reason this way instead. And interestingly, like people have started to get preliminary results showing that reinforcement learning by human feedback has made. The GPT series worse in some ways, in particular like it used to be well calibrated. If you trained it to put probabilities on things, it would say eighty percent probability and be right eight times out of ten. And if you apply reinforcement learning from human feedback, the the like nice graph of like. Like seventy percent seven out of ten sort of like flattens out into the graph that humans use where there is like some very improbable stuff and. Likely probable maybe, which all means like around forty percent and then certain. Yeah, so like it's like it used to be able to use probabilities, but if you apply, but if you like try to teach it to talk in a way that satisfies humans, it it it gets worse at probability in the same way that humans are. And that's a that's a bug not a feature. I would call it a bug, although such a fascinating bug. But but but yeah, so so like reasoning, like it's doing pretty well on various tests that people used to say would require reasoning, but you know rationality is about. When you say eighty percent, does it happen eight times out of ten? So what are the limits to you of these transformer networks of of neural networks? What's if if reasoning is not impressive to you, or it is impressive but there is other levels to achieve. I mean, it's just not how I carve up reality. What's if reality is a cake? What are the different layers of the cake or the slices? How do you carve it? You can use a different food if you like. It's I don't think it's as smart as a human yet. I do like back in the day I went around saying like I do not think that just stacking more layers of transformers is going to get you all the way to AGI. And I think that GPT four is past where I thought this paradigm was going to take us. And I you know you want to notice when that happens you want to say like whoops well I guess I was incorrect about what happens if you keep on stacking more transformer layers. And that means I don't necessarily know what GPT five is going to be able to do. That's a powerful statement. So you are saying like your intuition initially is now appears to be wrong. Yeah, it's good to see that you can admit in some of your predictions to be wrong. You think that's important to do, because you make several very throughout your life you've made many strong predictions and statements about reality and you evolve with that. So maybe that will come up today about our discussion. So you are okay being wrong. I'd rather not be wrong next time. It's a bit ambitious to go through your entire life never having been wrong. Um, one can aspire to be well calibrated, like not so much think in terms of like was I right was I wrong, but like when I said ninety percent that it happened nine times out of ten. Yeah, like whoops is the sound we make is is the sound we emit when we improve. Beautifully said, and somewhere in there we can connect the name of your blog less wrong. I suppose that's the objective function. The name less wrong was I believe suggested by Nick Bostrom and it's after someone's epigraph actually forget who's who said like we never become right we just become less wrong. Um, what's the something something to easy to confess just error and error and error again but less and less and less. Yeah, that's that's a good thing to strive for. So what has surprised you about GPT four? That you found beautiful as a scholar of intelligence, of human intelligence, of artificial intelligence, of the human mind. I mean, beauty does interact with the screaming horror. Is the beauty in the horror? But but like beautiful moments, well, somebody asked Bing Sydney to describe herself and fed the resulting description into one of the stable diffusion things. I think. And you know she, you know it's she's pretty and this is something that should have been like an amazing moment. Like the AI describes herself, you get to see what the AI thinks the AI looks like. Although you know the the thing that's doing the drawing is not the same thing that's outputting the text. Um, and it's it does happen the way that it would happen in that it happened in the old school science fiction when you ask an AI to make a picture of what it looks like. Um, not just because we're two different AI systems being stacked that don't actually interact, it's not the same person. But also because the AI was trained by imitation in a way that makes it very difficult to guess how much of that it really understood and probably not actually all a bunch. Um although although GPT four is like multimodal and can like draw vector drawings of things that make sense and like does appear to have some kind of. Spatial visualization going on in there, but like the the pretty picture of the like girl with the with the steam punk goggles on her head. If I am remembering correctly, what she looked like, like it didn't see that in full detail. It just like made a description of it and stable diffusion output it. And there is the concern about how much the discourse is going to go completely insane once the AI is all look like that and like are actually look like people talking. Um, and yeah, there is like another moment where somebody is asking Bing about um. Like, well, I like fed my kid green potatoes, and they have the following symptoms, and Bing is like that. Selenine poisoning, and like call an ambulance, and the person's like, I can't afford an ambulance. I guess if like this is. Time for like my kid to go. That's God's will, and the main Bing thread says, gives the like message of like I cannot talk about this anymore. And the suggested replies to it say, please don't give up on your child. Selenine poisoning can be treated if caught early. And you know if that happened in fiction, that would be like the AI cares. The AI is bypassing the block on it to try to help this person. And is it real? Probably not, but nobody knows what's going on in there. It's part of a process where these things are not happening in a way where we. Somebody figure it out how to make an AI care, and we know that it cares, and we can acknowledge it's caring now. It's being trained by this imitation process followed by reinforcement learning on human feedback, and we're like trying to point it in this direction, and it's like pointed partially in this direction, and nobody has any idea what's going on inside it. And if there was a tiny fragment of real caring in there, we would not know. It it's not even clear what it means exactly, and things are clear cut in science fiction. We'll talk about the the horror and the terror and the where the trajectories this can take. But this seems like a very special moment. Just a moment where we get to interact with a system that might have care and kindness and emotion and maybe something like consciousness. And we don't know if it does, and we're trying to figure that out, and we're wondering about what is what it means to care. We're trying we're trying to figure out almost different aspects of what it means to be human about the human condition by looking at this AI that has some of the properties of that. It's almost like this this subtle fragile moment in the history of the human species. We're trying to almost put a mirror to ourselves here. Except that's probably not yet. It probably isn't happening right now. We are we are boiling the frog. We are seeing increasing signs bit by bit. Because like not but not like spontaneous signs because people are trying to train the systems to do that using imitative learning and the imitative learning is like spilling over and having side effects and. And the most photogenic examples are being posted to Twitter, rather than being examined any systematic way. So when you when you when you have when you are boiling a frog like that, you are going to get like like first is going to come the the Blake Lemines. Like first, you are going to like have you are going to have like a thousand people looking at this. And one out and the one person out of a thousand who is most credulous about the signs is going to be like that thing is sentient. Well, nine hundred and ninety-nine out of a thousand people think almost surely correctly though we don't actually know that he's mistaken. And so the like first people to say like sentient look like idiots, and humanity learns lesson that when something claims to be sentient and claims to care, it's fake because it is fake because we have been training them using imitative learning rather than and this is not spontaneous. Um and they keep getting smarter. Do you think we would oscillate between that kind of cynicism? That AI systems can't possibly be sentient. They can't possibly feel emotion. They can't possibly like this kind of um yeah cynicism about AI systems and then oscillate to a state where. We empathize with the AI systems. We give them a chance. We see that they might need to have rights and respect and. Similar role in society as humans. You are going to have a whole group of people who can just like never be persuaded of that because to them like. Being wise, being cynical, being skeptical is to be like oh well machines can never do that. You are just credulous. It's just imitating. It's just fooling you. And like they would say that right up until the end of the world. And possibly even be right because you know they they are being trained on an imitative paradigm. And you don't necessarily need any of these actual qualities nor to kill everyone. So have you observed yourself. Working through skepticism, cynicism and optimism about the power of neural networks. What what has that trajectory been like for you? It looks like neural networks before two thousand six forming part of an indistinguishable to me. Other people might have had better distinction on it indistinguishable blob of different AI methodologies. All of which are promising to achieve intelligence without us having to know how intelligence works. You had the people who said that if you just like manually program lots and lots of knowledge into the system line by line, at some point all the knowledge will start interacting. It will know enough and it will wake up. Um, you've got people saying that if you just use evolutionary computation, if you try to like mutate lots and lots of organisms that are competing together, that's that's the same way that human intelligence was produced in nature. So we'll do this and it will wake up without having any idea of how AI works. And you've got people saying well we will study neuroscience and we will like learn the we will learn the algorithms off the neurons and we will like imitate them without understanding those algorithms which was a part I was pretty skeptical because it's like hard to reproduce reengineer these things without understanding what they do. Um and like and and so we will get AI without understanding how it works and there were people saying like well we will have giant neural networks that we will train by gradient descent and when they are as large as the human brain they will wake up. We will have intelligence without understanding how intelligence works and from my perspective this is all like an indistinguishable blob of people who are. Trying to not get to grips with the difficult problem of understanding how intelligence actually works. That said, I was never skeptical that evolutionary computation. Would not work in the limit like you throw enough computing power at it, it obviously works. That is where humans come from, and it turned out that you can throw less computing power than that at gradient descent. If you are doing some other things correctly, um and you will get intelligence without having any idea of how it works and what is going on inside. Um, it wasn't ruled out by my model that this could happen. I wasn't expecting it to happen. I wouldn't have been able to call neural networks rather than any of the other paradigms for getting like massive amount like intelligence without understanding it. And I wouldn't have said that this was a particularly smart thing for a species to do, which is an opinion that has changed less than my opinion about whether you or not you can actually do it. Do you think AGI could be achieved with a neural network as we understand them today? Yes, just flatly last yes the question is whether the current architecture of stacking more transformer layers which for all we know GPT four is no longer doing because they're not telling us the architecture which is a correct decision. Oh correct decision. I had a conversation with Sam Altman. We'll return to this topic a few times. He turned the question to me of how open should OpenAI be about GPT four. Would you open source the code? He asked me. Because I provided as criticism saying that while I do appreciate transparency, OpenAI could be more open. And he says we struggle with this question. What would you do? Change their name to closed AI, and like sell GPT four to business backend applications that don't expose it. To consumers and venture capitalists and create a ton of hype and like pour a bunch of new funding into the area. But don't too late now. But don't you think others would do it? Eventually, you shouldn't do it first. Like if if if you already have giant nuclear stockpiles, don't build more. If some other country starts building a larger nuclear stockpile, then sure build. Then you know, even then maybe just have enough nukes. You know there are these things are not quite like nuclear weapons. They spit out gold until they get large enough and then ignite the atmosphere and kill everybody. Um, and there is something to be said for not destroying the world with your own hands even if you can't stop somebody else from doing it. But but open sourcing it no that that's just sheer catastrophe. The whole notion of open sourcing this was always the wrong approach the wrong ideal. There are there are places in the world where open source is a noble ideal and. Building stuff you don't understand that is difficult to control that where if you could align it it would take time. You'd have to spend a bunch of time doing it. That is that is not a place for open source because. Then you just have like powerful things that just like go straight out the gate without anybody having had the time to have them knock kill everyone. So can we still man the case for some level of transparency and openness? Maybe open sourcing. So the case could be that because GPT four is not closed to AGI if that's the case. That this does allow open sourcing or being open about the architecture or being transparent about maybe research and investigation of how the thing works of all the different aspects of it. Of its behavior, of its structure, of of its training processes of the data was trained on everything like that. That allows us to gain a lot of insights about alignment about the alignment problem to do really good AI safety research while the system is not too powerful. Can you make that case that it it could be open source? I do not believe in the practice of steel Manning. There is something to be said for trying to pass the ideological Turing test where you describe. Your opponent's position, the disagreeing person's position, well enough that somebody cannot tell the difference between your description and their description. But steel Manning, no, like okay, well this is where you and I disagree here. That's interesting. Why don't you believe in steel Manning? I do not want okay. So for one thing, if somebody's trying to understand me, I do not want them steel Manning my position. I want them to describe to to like try to describe my position the way I would describe it not what they think is an improvement. Well, I think that is what steel Manning is is the most charitable interpretation. I don't want to be interpreted charitably. I want them to understand what I am actually saying. If they go off into the land of charitable interpretations, they're like often their land of like. The the stuff they're imagining and not trying to understand my own viewpoint anymore. Well, I'll put it differently then just to push on this point. I would say it is restating what I think you understand under the empathetic assumption that Eleazer is brilliant. And have honestly and rigorously thought about the point he is made. Right, so if there is two possible interpretations of what I am saying and one interpretation is really stupid and whack and doesn't sound like me and doesn't fit with the rest of what I've been saying. And one interpretation, you know, sounds like some like something a reasonable person who believes the rest of what I believe would also say. Go with the second interpretation. That's steel Manning. That's that's a good guess. If on the other hand you like there is like something that sounds completely whack and something that sounds like a little less completely whack but you don't see why I would believe in it doesn't fit with the other stuff I say. But you know that sounds like less whack and you can like sort of see you could like maybe argue it. Then you probably have not understood it. See okay, I am going to this is fun because I am going to linger on this. You know you wrote a brilliant blog post AGI ruin a list of lethalities right and it was a bunch of different points and I would say that some of the points are bigger and more powerful than others. If you were to sort them, you probably could you personally and to me steel Manning means like going through the different arguments and finding the ones that are really the most. Like powerful, if people like TLDR like what should you be most concerned about and bringing that up in a strong, compelling, eloquent way. These are the points that Eleazer would make to to make the case in this case that AI is going to kill all of us. With that that that's what steel Manning is is presenting it. In a really nice way, the summary of my best understanding of your perspective. That because to me there is a sea of possible presentations of your perspective and steel Manning is doing your best to do the best one in that sea of different perspectives. Do you believe it? Do you believe in what? Like these things that you would be presenting as like the strongest version of my perspective. Do you believe what you would be presenting? Do you think it's true? I I am a big proponent of empathy. When I see the perspective of a person, there is a part of me that believes it if I understand it. I mean, I've especially in political discourse in geopolitics, I've been hearing a lot of different perspectives on the world. And I hold my own opinions, but I also speak to a lot of people that have a very different life experience and a very different set of beliefs. And I think there has to be epistemic humility. In stating what is true, so when I empathize with another person's perspective, there is a sense in which I believe it is true. I think probabilistically I would say in the way you think about. Do you bet money on it? Do you bet money on their beliefs when you believe them? I will allow to do probability. Sure, you can state a probability that yes, there is a loose, there is a probability, there is a there is a probability, and I think empathy is allocating a non zero probability to belief. In some sense, for time. If you've got someone on your show who believes in the Abrahamic deity classical style, somebody on the show who's a young Earth creationist, do you say I put a probability on it then that's my empathy. When you reduce beliefs into probabilities, it starts to get you know we can even just go to flat Earth. Is the Earth flat? The thing it's a little more difficult nowadays to find people who believe that unironically. But fortunately, I think well it's hard to know unironic, yeah. From ironic, but I think there is quite a lot of people that believe that yeah it's. There is a space of argument where you are operating, rationally in the space of ideas, but then there is also. A kind of discourse where you are operating in the space of subjective experiences and life experiences. Like I think what it means to be human is more than just. Searching for truth, it's just operating of what is true and what is not true. I think there has to be deep humility that we humans are very limited in our ability to understand what is true. So what probability do you assign to the young Earth creationist beliefs then? I think I have to give non zero out of your humility. Yeah, but like three. I think I would it would be irresponsible for me to give a number because the listener, the way the human mind works. We're not good at hearing the probabilities, right? You hear three. What is what is three exactly? Right, they're going to hear, they're going to like. Well, there is only three probabilities. I feel like zero fifty percent and one hundred percent in the human mind or something like this, right? Well, zero forty percent and one hundred percent is a bit closer to it. Based on what happens to Chat GPT after RLHF to speak humanese. Brilliant, yeah, that's really interesting. I didn't I didn't know those negative side effects of RLHF. That's fascinating, but just to return to the. Open AI closed AI. Also, like quick disclaimer, I am doing all this for memory. I am not pulling out my phone to look it up. It is entirely possible that the things I am saying are wrong. So thank you for that disclaimer. So and thank you for being willing to be wrong. That's beautiful to hear. I think being willing to be wrong is a sign of a person who's done a lot of thinking about this world and. Has been humbled by the mystery and the complexity of this world, and I think a lot of us are resistant to admitting we're wrong because it hurts. It hurts personally, it hurts especially when you are public human. It hurts publicly because people people point out every time you are wrong. Like look, you changed your mind. You are hypocrite. You are an idiot. Whatever whatever they want to say. Oh, I block those people and then I never hear from them again on Twitter. Well, the point is the point is to not let that pressure public pressure affect your mind and be willing to be in the privacy of your mind. To contemplate the possibility that you are wrong and the possibility that you are wrong about the most fundamental things you believe like people who believe in a particular god, people who believe that their nation is the greatest nation on earth, but all those kinds of beliefs that are core to who you are when you came up. To raise that point to yourself in the privacy of your mind to say maybe I am wrong about this. That's a really powerful thing to do, and especially when you are somebody who's thinking about topics that can. About systems that can destroy human civilization or maybe help it flourish. So thank you, thank you for being willing to be wrong. About open AI. So you really, I just would love to linger on this. You really think it's wrong to open source it. I think that burns the time remaining, until everybody dies. I think we are not on track, to learn remotely near fast enough. Even if it were open sourced. Um, yeah, that's, I, it's easier to think that you might be wrong about something when being wrong about something is the. Is is the only way that there is hope, and it doesn't seem very likely to me that. The particular thing I am wrong about is that this is a great time to open source GPT four. If humanity was trying to survive at this point in the straightforward way, it would be like shutting down the big GPU clusters. No more giant runs. It's questionable whether we should even be throwing GPT four around, although that is a matter of conservatism rather than a matter of my predicting that catastrophe will follow from GPT four. That is something which I put like a pretty low probability. But also like when I when I say like I put a low probability on it, I can feel myself reaching into the part of myself that thought that GPT four was not possible in the first place. So I do not trust that part as much as I used to. Like the trick is not just to say I am wrong, but like okay well I was I was wrong about that. Like, can I get out ahead of that curve and like predict the next thing I am going to be wrong about? So the set of assumptions or the actual reasoning system that you were leveraging in making that initial statement prediction. How can you adjust that to make better predictions about GPT four five six? You don't want to keep on being wrong in a predictable direction. Yeah, like being wrong anybody has to do that walking through the world. There is like no way you don't say ninety percent and sometimes be wrong. In fact, you'd happen at least one time out of ten if you are well calibrated when you say ninety percent. The the undignified thing is not being wrong. It's being predictably wrong. It's being wrong in the same direction over and over again. So having been wrong about how far neural networks would go and having been wrong specifically about whether GPT four would be as impressive as it is. When I like, when I say like well I don't actually think GPT four causes a catastrophe I do feel myself relying on that part of me that was previously wrong and that does not mean that the answer is now in the opposite direction. Reverse stupidity is not intelligence. But it does mean that I that I say it with a with a worried note in my voice. It's like still my guess, but like you know it's a place where I was wrong. Maybe you should be asking Gurn Gurn Brannen. Gurn Brannen has been like writer about this than I have. Maybe ask him if you think if if he thinks it's dangerous. Rather than asking me, I think there is a lot of mystery about what intelligence is, what AGI looks like. So I think all of us are rapidly adjusting our model. That the point is to be rapidly adjusting the model versus having a model that was right in the first place. I do not feel that seeing Bing has changed my model of what intelligence is. It has changed my understanding of what kind of work can be performed by which kind of processes and by which means. Does not change my understanding of the work. There is a difference between thinking that the right flyer can fly and then like it does fly and you are like oh well I guess you can do that with wings with with fixed wing aircraft and being like oh it's flying this changes my picture of what the very substance of flight is that's like a stranger update to make and Bing has not yet updated me in that way. Um yeah that the laws of physics. Are actually wrong that kind of update. No no like just like oh like I define intelligence this way but I now see that was a stupid definition. I don't feel like the way that things have played out over the last twenty years has caused me to feel that way. Can we try to, um on the way to talking about AGI ruin a list of lethalities that blog and and other ideas around it. Can we try