諾姆·布朗：撲克中 AI 與人類的對抗及策略性談判博弈

Lex Fridman Podcast #344 – Noam Brown: AI vs Humans in Poker and Games of Strategic Negotiation Machine transcription by Qwen3-ASR 0.6B (MLX, local, Apache-2.0). May contain recognition errors. The following is a conversation with No Brown, research scientist at Fair Facebook AI Research Group at Meta AI. He co-created the first AI system that achieved superhuman level performance in No Limit Texas Hold'em both heads up and multiplayer. And now recently he co-created an AI system that can strategically out negotiate humans using natural language in a popular board game called Diplomacy, which is a war game that emphasizes negotiation. And now a quick few second mention of each sponsor. Check them out in the description. It's the best way to support this podcast. We got True Classic T's for shirts, Audible for audiobooks, Inside Tracker for bio monitoring and ExpressVPN for ISP privacy. Choose wisely, my friends. And now onto the full ad reads as always no ads in the middle. I try to make this interesting but if you skip them, please still check out the sponsors. I enjoy their stuff, maybe you will too. This show is brought to you by True Classic T's, high quality soft slim fitted T shirts for men. They also make all the other menswear staples like polo's, workout shirts and boxers. I mostly wear their black T shirt. And that's what I am wearing right now. I have a ton of them. I enjoy how it feels. I enjoy how it looks. That's like the standard default look of the the guy behind the keyboard. I guess programmer. I wonder when the T-shirt was created. Looking up on Wiki, the T-shirt evolved from undergarments used in the 19th century and in the mid 20th century transitioned from undergarments to general use casual clothing. Okay, wonderful. Anyway, go to trueclassic dot com and enter code Lex to get 25% off. This episode is brought to you by Audible, an audiobook service that has given me hundreds if not thousands of hours of education through listening to audiobooks. I am currently re-listening for the. I don't know how many times I've re-listened to this book. How many times I've read it, but George Orwell's 1984, not nearly as many times as Animal Farm. I don't know why exactly, but Animal Farm just as a story just connects with me—a basic fable of creatures being not so good to each other and being greedy and also being good to each other and having personalities and just in a simple way that sort of whatever animals are made in a children like story—it's simple but it can still get to the profound and I think I like to think of life in those terms. Anyway, I am currently listening to 1984. It's really well sort of spoken voice or whatever on Audible, so I highly recommend it. New members can try free for 30 days at audible dot com slash Lex or text Lex to 500 500. This show is also brought to you by Inside Tracker, a service I use to track biological data. Did you know there is 1.2 to 1.5 gallons of blood in a human being? It's approximately 10% of adults' weight. I remember like reading some insane stats on the volume of blood pumped a day through the human body. It's kind of it's just incredible. Sort of this whole machinery that interacts with oxygen. I mean, just with the chemistry of Earth interacts with it. It uses energy ultimately from the sun and the photosynthesis is involved with the plants. The whole like system is just freaking incredible and it's just so full of data. Like imagine being able to measure all of that about life on Earth. Yeah, so Inside Tracker is like early steps in being able to measure directly from the human body. Get 200 off Inside Tracker's ultimate plan or 34% off the entire store when you go to insidetracker dot com slash Lex. This show is also brought to you by ExpressVPN. I use them to protect my privacy on the internet. I've used them for many many years. Through thick and thin, when I was lost in this too big world of ours and looking for the truth out there and protecting my privacy all along. I remember being really excited to learn that it can work on Android that it can work on Linux. So it works on any operating system and it just doesn't feel like it's there. Everything is super fast whether you have the VPN or not. It's kind of incredible. It's incredible how this technology works and it it conceals your location from the perspective of the website or service that you are using online. It's it's an incredible tool should be the first layer of protection of your privacy on the internet. Go to expressvpn dot com slash Lex Pod for an extra three months free. This is the Lex Friedman podcast. To support it, please check out our sponsors in the description. And now, dear friends, here is No Brown. You've been a lead on three amazing AI projects. So we got Libras that solved or at least achieved human level performance on No Limit Texas Hold'em poker with two players heads up. You got Pluribus that solved No Limit Texas Hold'em poker with six players. And just now you have Cicero. These are all names of systems that solved or achieved human level performance on the game of Diplomacy, which for people who don't know is a popular strategy board game. It was loved by JFK, John F Kennedy and Henry Kissinger and many other big famous people in the decade since. So let's talk about poker and diplomacy today. First poker, what is the game of No Limit Texas Hold'em and how is it different from chess? Well, No Limit Texas Hold'em poker is the most popular variant of poker in the world. So you know you go to a casino, you play, sit down at the poker table. The game that you are playing is No Limit Texas Hold'em. If you watch movies about poker like Casino Royale or Rounders, the game that they're playing is No Limit Texas Hold'em poker. Now it's very different from Limit Hold'em in that you can bet any amount of chips that you want and so the stakes escalate really quickly. You start out with like one or two dollars in the pot and then by the end of the hand you've got like a thousand dollars in there maybe. So the option to increase the number very aggressively very quickly is always there. Right, the no limit aspect is there is no limit to how much you can bet. You know you in Limit Hold'em, there is like two dollars in the pot you you can only bet like two dollars but if you got ten thousand dollars in front of you you are always welcome to put ten thousand dollars into the pot. So I got a chance to hang out with Phil Hellmuth who plays all these different variants of poker. And correct me if I am wrong, but it seems like No Limit rewards crazy versus the other ones rewards more kind of calculated strategy or or no because you are sort of looking from an from an analytic perspective is is strategy also rewarded in No Limit Texas Hold'em. I think both variants reward strategy, but I think what's different about No Limit Hold'em is it's it's much easier to get jumpy. You know you go in there thinking you are going to lose. You know you are going to play for like a hundred dollars or something and suddenly there is like you know a thousand dollars in the pot. A lot of people can't handle that. Can you define jumpy? When you are playing poker, you always want to choose the action that's going to maximize your expected value. It's kind of like kind of like with investing right like if you are ever in a situation where you are the amount of money that's at stake is is going to have a material impact on your life, then you are going to play in a more risk averse style. You know if somebody makes a huge bet, you are going to if you are playing No Limit Hold'em and somebody makes a huge bet, there might come a point where you are like this is too much money for me to handle like I can't risk this amount and that's what throws a lot of people off. So that's the big difference I think between No Limit and Limit. What about on the action side when you are actually making that big bet? That's what I mean by crazy. I was I was trying to refer to the the technical the the technical term of crazy meaning use the big jump in the bet to completely throw off the other person in terms of their ability to reason optimally. I think that's right. I think one of the key strategies in poker is to put the other person into an uncomfortable position and if you are doing that then you are you are playing poker well and there is a lot of opportunities to do that in No Limit Hold'em you know you can have like fifty dollars in there you throw in a thousand dollar bet and you know that sometimes if you do it right it puts the other person in a really tough spot now it's also possible that you make huge mistakes that way and so it's really easy to lose a lot of money in No Limit Hold'em if you don't know what you are doing but there is a lot of upside potential too. So when you build systems AI systems that play these games, we'll talk about poker, we'll talk about diplomacy. Are you are you drawn in in part by the beauty of the game itself? AI aside, or is it to you primarily a fascinating problem set for the AI to solve? I am drawn in by the beauty of the game. When I I started playing poker when I was in high school and the idea to me that there is a correct an objectively correct way of playing poker and if you could figure out what that is then you are you know you are making unlimited money basically. That's like a really fascinating concept to me. And so I was fascinated by the strategy of poker, even when I was like 16 years old. It wasn't until like much later that I actually worked on poker AI's. So there was a sense that you can solve poker like in the way you can solve chess for example or checkers. I believe checkers got solved right. Yeah checkers checkers is completely solved optimal strategy. It's impossible to beat the AI. Yeah and so in that same way you could technically solve chess. You could solve chess. You could solve poker. You could solve poker. So this is this gets into the concept of Nash equilibrium. So what is Nash equilibrium? Okay, so in any finite two player zero sum game, there is an optimal strategy that if you play it you are guaranteed to not lose an expectation no matter what your opponent does. And this is kind of a radical concept to a lot of people, but it's true in chess, it's true in poker, it's true in any finite two player zero sum game. And to give some intuition for this, you can think of rock paper scissors. In rock paper scissors, if you randomly choose between throwing rock paper and scissors with equal probability, then no matter what your opponent does, you are not going to lose an expectation. You are not going to lose an expectation in the long run. Now the same is true for poker. There exists some strategy, some really complicated strategy that if you play that you are guaranteed to not lose money in the long run and I should say this is for two player poker, six player poker is a different story. Yeah it's a beautiful giant mess. When you say in expectation, you are guaranteed not to lose in expectation. What does in expectation mean? Poker is a very high variance game, so you are going to have hands where you win, you are going to have hands where you lose, even if you are playing the perfect strategy, you can't guarantee that you are going to win every single hand, but if you play for long enough, then you are guaranteed to at least break even and and in practice probably win. So that's in expectation the size of your stack generally speaking. Now that doesn't include anything about the fact that you can go broke. It doesn't include any of those kinds of normal real world limitations. You are talking in a in a theoretical world. What about the the zero sum aspect? How big of a constraint is that? How big of a constraint is finite? So finite's not a huge constraint. So I mean most games that you play are finite in size. It's also true actually that there exists this like perfect strategy in many infinite games as well. Technically the game has to be compact. There are like some edge cases where you don't have an Nash equilibrium in a two player zero sum game. So you can think of a game where like you know if we're playing a game where whoever names the bigger number is the winner, there is no Nash equilibrium to that game. 17. Yeah, 18. You beat. You win again. You are good at this. Played a lot of games. Okay, so that's and then the zero sum aspect the zero zero sum aspect. So there exists a Nash equilibrium in non two player zero sum games as well. And by the way, just to clarify what I mean by two player zero sum, I mean there is two players and whatever one player wins the other player loses. So if we're playing poker and I win fifty dollars, that means that you are losing fifty dollars. Now outside of two player zero sum games, there still exists Nash equilibrium, but they're not as meaningful because you know you can think of a game like risk. If everybody else at the and on the board decides a team up against you and take you out, there is no perfect strategy you can play that's going to guarantee that you win there. There is just nothing you can do. So outside of two player zero sum games, there is no guarantee that you are going to win by playing a Nash equilibrium. Have you ever tried to model in the other aspects of the game, which is like the pleasure you draw from playing the game and then if you are a professional poker player, if you are exciting even if you lose. The you know the money you would get from the attention you get to the sponsors and all that kind of stuff is that that would be a fun thing to model to model in. I wish that would make it sort of super complex to include the human factor and this in its full complexity. I think you bring up a couple good points there. So I think a lot of professional poker players, I mean they get a huge amount of money not from actually playing poker but from the sponsorships and having a personality that people want to tune in and watch that that's a big that's a big way to to make a name for yourself in poker. I just wonder from an AI perspective if you create and we'll talk about this more maybe an AI system that also talks trash and all that kind of stuff that that becomes part of the function to maximize so it's not just optimal poker play maybe sometimes you want to be chaotic maybe sometimes you want to be suboptimal and you lose the the chaos and maybe sometimes you want to be overly aggressive because the audience loves that. Yeah, fascinating. I think I think what you are getting at here is that there is a difference between making an AI that wins a game and an AI that's fun to play with. Yeah, yeah, and or fun to watch. So those are all different things, fun to play with and fun to watch. Yeah, and I think you know I've I've heard talks from like game designers and and they say like you know people that work on AI for actual recreational games that people play and they say yeah there is a big difference between trying to make an AI that actually wins and you know you look at a game like Civilization the way that the AI's play is not optimal for trying to win they're they're playing a different game they're trying to have personalities they're trying to be fun and engaging and that makes for a better game. Yeah and we also talk about NPCs. I just talked to Todd Howard who is the the creator of Fallout and the Elder Scrolls series and Starfield a new game coming out and the creator what I think is the greatest game of all time which is Skyrim and the NPCs there the AI that governs that whole game is very interesting but the NPCs also are super interesting and considering what language models might do to NPCs in an open world RPG role playing game. It's super exciting. Yeah, honestly, I am. I think this is like one of the first applications where we're going to see like real consumer interaction with large language models. I guess let Skyrim Elder Scrolls six is in development now. They're probably like pretty close to finishing it, but I would not be surprised at all if Elder Scrolls seven was using large language models for their NPCs. No, they're not. They're I mean I am not saying anything. Not saying anything. Okay, this is me speculating, not you. No, but but there is they're just releasing the Starfield game. They do one game at a time. Yeah, and so whatever it is, whenever the date is, I don't know what the date is. Calm down, but it would be I don't know like 2024, 25, 26. So it's actually very possible that would include language models. I was listening to this this talk by a gaming executive when I was in grad school, and one of the questions that a person in the audience asked is why are all these games so focused on fighting and killing? And the person responded that it's just so much harder to make an AI that can talk with you and cooperate with you than it is to make an AI that can fight you. And I think once this technology develops further and you can have you can reach a point where like not every single line of dialogue has to be scripted, it unlocks a lot of potential for new kinds of games like much more like positive interactions that are not so focused on fighting and I am really looking forward to that. It might not be positive, it might be just drama. So you you'll be in like a Call of Duty game instead of doing the shooting, you'll just be hanging out and like arguing with an AI about like like passive aggressive and then you won't be able to sleep that night. You have to return and continue the argument that you were emotionally hurt. I mean, yeah, I think that's actually an exciting world. Whatever, whatever is the drama, the chaos that we love the push and pull of human connection. I think it's possible to do that in the video game world, and I think you could be messier and make more mistakes in the video game world, which is why it would be a nice place and and also it doesn't have a deep of a as deep of a real psychological impact because inside video games it's kind of understood that you are in a not a real world so whatever crazy stuff AI does we have some flexibility to play just like with a game of diplomacy it's a game this is not real geopolitics not real war it's a it's a game so you could you can have a little bit of fun a little bit of chaos okay back to Nash equilibrium how do we find the Nash equilibrium? All right, so there is different ways to find a Nash equilibrium. So the way that we do it is with this process called self play. Basically, we have this algorithm that starts by playing totally randomly and it learns how to play the game by playing against itself. So it will start playing the game totally randomly and then it you know if it's playing poker it'll eventually like get to the end of the end of the game and make fifty dollars, and then it will like review all the decisions that it made along the way and say what would have happened if I had chosen this other action instead. You know if I had raised here instead of called, what would the other player have done? And because it's playing against a copy of itself, it's able to do that counterfactual reasoning. So it can say, okay, well if I took this action and the other person takes this action and then I take this action and eventually I make one hundred fifty dollars instead of fifty. And so it updates the regret value for that action. Regret is basically like how much does it regret having not played that action in the past, and when it encounters that same situation again, it's going to pick actions that have higher regret with higher probability. Now it'll just keep simulating the games this way. It'll keep you know accumulating regrets for different situations, and in the long run, if you pick actions that have high regret with higher probability in the correct way, it's proven to converge to a Nash equilibrium. Even for super complex games, even for imperfect information games. It's true for all games. It's true for it's true for chess. It's true for poker. It's particularly useful for poker. So this is the the method of counterfactual regret minimization. This is counterfactual regret minimization. That doesn't have to do with self play. Has to do with just any if you follow this kind of process self play or not. You will be able to arrive at an optimal set of actions. So this counterfactual regret minimization is a kind of self play. It's a principled kind of self play that's proven to converge to Nash equilibrium even in imperfect information games. Now you can have other forms of self play and people use other forms of self play for perfect information games. Um, where you have more flexibility the algorithm doesn't have to be as theoretically sound. Um, in order to converge to to that class of games because there's it's a simpler setting. Sure. So I kind of in in my brain the word self play has mapped to neural networks but we're speaking something bigger than just neural networks it could be anything. The self play mechanism is just the mechanism of a system playing itself. Exactly. Yeah, self play is not tied specifically to neural nets. It's it's a kind of reinforcement learning basically. Okay. And I would also say this process of like trying to reason oh what would the value have been if I had taken this other action instead. This is very similar to how humans learn to play a game like poker right like you probably played poker before and with your friends you probably ask like oh would you have called me if I raised there. You know and that's that's a person trying to do the same kind of like learning from a counterfactual that the AI is doing. Okay and if you do that at scale you are going to be able to learn an optimal policy. Yeah. Now where the neural nets come in I said like okay if it's in that situation again then it will choose the action that has has high regret now the problem is that poker is such a huge game you know I think no limit Texas Hold'em the version that we were playing has 10 of the 161 different decision points which is more than the number of atoms in the universe squared. That's heads up. That's heads up. Yeah. 10 of the 161 you said. Yeah I mean it depends on the number of chips that you have the stacks and everything but like the version that we were playing was 10 of the 161. Which I assume would be a somewhat simplified version anyway because the. But there's some like step function you had for like bets. Oh no no that's that's I'm saying like we played the the full game you can bet whatever amount you want and the bot maybe was constrained in like what it considered for bet sizes but the the person on the other side could bet whatever they wanted. Yeah I mean 161 plus or minus 10 doesn't matter. Um and so the the way neural nets help out here is you know you don't have to run into the same exact situation because that's never going to happen again the odds of you running into the same exact situation are pretty slim but if you run into a similar situation then you can generalize from other states that you've been in that kind of look like that one and you can say like well these other situations I had high regret for this action and so maybe I should play that action here as well. Which is the more complex game chess or poker or go or poker do you know. That is a controversial question. Okay I'm gonna. Was like somebody screaming on Reddit right now it depends on which subreddit you're on is it chess or is it poker. I'm sure like David Silver is going to get really angry at me. Yeah I'll I'll say I'm going to say poker actually and I think for a couple reasons. They're not here to defend themselves. So first of all you have the imperfect information aspect and so it's um it. We we can go into that but like once you introduce imperfect information, uh things get much more complicated. So we should say maybe you can describe what is seen to the players what is not seen, uh in the game of Texas Hold'em. Yeah so Texas Hold'em you get two cards face down that only you see um and so that's the hidden information of the game the other players also all get two cards face down that only they see. Um and so you have to kind of as you're playing reason about like okay what do they think I have what do they have what do they think I think they have that kind of stuff and um that's that's kind of where bluffing comes into play right because the fact that you can bluff the fact that you can bet with a bad hand and still win is because they don't know what your cards are. And that's the that's the key difference between a perfect information game like poker sorry like chess and go um and imperfect information games like poker. This is what trash talk looks like. The implied statement is the game I solved is much tougher. But yeah so when you're playing I'm just going to do random questions here so when you're playing your opponent under imperfect information. Is there some degree to which you're trying to estimate the range of hands that they have, or is that not part of the algorithm? So what are the different approaches to the imperfect information game. So the key thing to understand about why imperfect information makes things difficult is that you have to worry not just about which actions to play but the probability that you're going to play those actions. So you think about um rock paper scissors for example rock paper scissors is an imperfect information game. Um because you don't know what I'm about to throw. I do but yeah usually not yeah. And so you can't just say like oh I'm just going to throw rock every single time because the other person is going to figure that out and notice a pattern and then suddenly you're going to start losing. And so you don't just have to figure out like which action to play you have to figure out the probability that you play it. And really importantly the value of an action depends on the probability that you're going to play it. So if you're playing rock every single time that value is really low but if you're. Never playing rock you play rock like one percent of the time and suddenly the the other person is probably going to be throwing scissors and when you throw rock the value of that action is going to be really high. Now you take that to poker what that means is the value of bluffing for example if you're the kind of person that never bluffs and you have this reputation as somebody that that never bluffs and suddenly you bluff there's a really good chance that that bluff is going to work and you're going to make a lot of money. On the other hand if you got a reputation like if they see you play for a long time and they see oh you're the kind of person that's bluffing all the time when you bluff they're not going to buy it and they're going to call you down you're going to lose a lot of money. And that finding that balance of how often you should be bluffing is the key challenge of a game of poker. And um you contrast that with a game like chess. It doesn't matter if you're opening with the queen's gambit 10 of the time or 100 of the time the value the expected value is the same. So um so that's that's why we need these algorithms. That understand not just we have to figure out what actions are good but the probabilities we need to get the exact probabilities correct and that's actually when we created the bot Lebratus Lebratus means balanced because the algorithm that we designed. Was designed to find that right balance of how often it should play each action. The balance of how often in the key sort of branching is the bluff or not the bluff. Is that is that a good crude simplification of the major decision in poker. It's a good simplification I think that's like the main tension but it's it's not just how often to bluff or not to bluff it's like how often should you bet in general how often should you what what kind of bet should you make. Um should you bet big or should you bet small and with which with with which hands and so this is where the idea of of a range comes from because when you are bluffing with a particular hand in a particular spot. You don't want there to be a pattern for the other person to pick up on you don't want them to figure out oh whenever this person is in this spot they're always bluffing and so you have to reason about okay would I also bet with a good hand in this spot. You want to be unpredictable so you have to think about what would I do if I had this different set of cards. Is there explicit estimation of like a theory of mind that the other person has about you or is that just a emergent thing that happens. The way that the bots handle it that are really successful they have an explicit theory of mind so they are explicitly reasoning about what are. What's the common knowledge belief what does what do you think I have what do I think you have what do you think I think you have um it's explicitly reasoning about that is there multiple use there. So maybe that's jumping ahead to six players but is there a stickiness to the person so it's an iterative game you're playing the same person. There is there's a stickiness to that right you're gathering information as you play it's not every every every hand is in your hand is there um a continuation in terms of estimating what kind of player I'm facing here. That's a good question so you could approach the game that way the way that the bots do it they don't and the way that humans approach it also expert human players the way they approach it is to basically assume that you know my strategy so. I'm going to try to pick a strategy where even if I were to play it for 10 000 hands and you could figure out exactly what it was you still wouldn't be able to beat it basically what that means is I'm trying to approximate the Nash equilibrium I'm trying to be perfectly balanced because if if I'm playing the Nash equilibrium even if you know what my strategy is like I said I'm still unbeatable in expectation. So so that's what that's what the bot aims for and that's actually what a lot of expert poker players aim for as well to start by playing the Nash equilibrium and then maybe if they spot weaknesses in the way you're playing then they can deviate a little bit to take advantage of that. They aim to be unbeatable in expectation okay so who's the greatest poker player of all time and why is it Phil Hellmuth so this is for Phil. So he's known um at least in part for maybe playing suboptimally and he still wins a lot it's a bit chaotic so maybe. Can you speak from an AI perspective about the genius of his madness or the madness of his genius. So playing suboptimally playing chaotically um as a way to make you hard to pin down about what your strategy is. So okay the thing that I should explain first of all with like Nash equilibrium it doesn't mean that it's predictable the whole point of it is that you're trying to be unpredictable. Now I think when somebody like Phil Hellmuth might be really successful is not in being unpredictable but in being able to um. Take advantage of the other player and figure out where they're being predictable or guiding the other player into thinking that you have certain weaknesses and then and then understanding how they're going to change their behavior they're going to deviate from a Nash equilibrium style of play to try to take advantage of those perceived weaknesses and then counter exploit them so you kind of get into the mind games there. So you think about at least heads up poker as as a dance between two agents I guess are you playing the cards are you playing the the player. So this this gets down to a big argument in the poker community and the academic community for a long time there was this debate of like what what's called GTO game theory optimal poker or exploitative play. And um up until about like 2017 when we did the Lebratus match I think actually exploitative play had the advantage a lot of people were saying like oh this whole idea of game theory it's just nonsense and if you really want to make money you got to like look into the other person's eyes and read their soul and figure out what cards they have. But what happened was people started adopting the game theory optimal strategy um and they were making good money and they weren't trying to adapt so much the other player they were just trying to play the Nash equilibrium and then what really solidified it I think was the Lebratus the Lebratus match. Where we played our bot against four top heads up no limit Hold'em poker players and the bot wasn't trying to adapt to them it wasn't trying to exploit them it wasn't trying to do these mind games it was just trying to approximate the Nash equilibrium and it crushed them. I think you know we were playing for 50 100 blinds and over the course of about 120 000 hands it made close to two million dollars 120 000 hands 120 000 hands against humans. Yeah and this was this was fake money to be clear so there was real money at stake there was 200 000 first of all all money is fake but um that's that's that's a different conversation. We give it meaning it's an it's a it's it's a phenomenon that gets meaning from our complex psychology as a human civilization. It's emergent from the collective intelligence of the human species but that's not what you mean you mean like there's literally you can't you can't buy stuff with it okay can you actually step back and take me through that competition. Yeah okay so when I was in grad school there was this thing called the annual computer poker competition where every year all the different research labs that were working on AI for poker would get together they would make a bot they would play them against each other. And we made a bot that actually won the 2014 competition the 2016 competition and so we decided we're going to take this bot build on it and play against real top professional heads up no limit Texas Hold'em poker players. So we invited four of the world's best um players in this specialty and we challenged them to 120 000 hands of poker over the course of 20 days. Um and we had 200 000 200 000 dollars in prize money at stake where it would basically be divided among them depending on how well they did relative to each other. So we wanted to have some incentive for them to play their best. Did you have a confidence 2014 16 that this is even possible how much doubt was there. So and we did a competition actually in 2015 where we also played against professional poker players and the bot lost by by a pretty sizable margin actually now there were some big improvements from 2015 to 2017 and so. Can you speak to the improvements is it computational nature is it the algorithm the the methods it was it was really an algorithmic approach that that was the difference so. 2015 it was much more focused on trying to come up with a strategy upfront like trying to solve the entire game of poker like and then just have a lookup table where you're saying like oh I'm in this situation what's the strategy. Um the approach that we took in 2017 was much more search based it was trying to say okay well let me in real time try to compute a much better strategy than what I had precomputed by playing against myself during self play. What is the search space for poker? What are you searching over? What's that look like? There's different actions like raising calling yeah what are the actions. Um is it just a search over actions. So in a game like chess the the search is like okay I'm in this chess position and I can like you know move these different pieces and see where things end up in poker what you're searching over is the actions you can take for your hand the probabilities that you take those actions and then also the probabilities that you take other actions with other hands that you might have. Um and and that's kind of like a hard to wrap your head around like why are you searching over these like other hands that you might have and like trying to figure out what you would do with those hands. Um and the idea is is again you you want to you want to always be balanced and unpredictable and so if your search algorithm is saying like oh I want to raise with this hand well in order to know whether that's a good action like let's say it's a bluff you know let's say you have a bad hand and you're saying like oh I I think I should be betting here with this really bad hand and bluffing. Well that that's only a good action if you're also. Betting with a strong hand otherwise it's an obvious bluff so if your action in some sense maximizes your unpredictability so that action could be mapped by your opponent to a lot of different hands then that's a good action. Basically what you want to do is put your opponent into a tough spot so you want them to always have some doubt like should I call here should I fold here and if you are raising in the appropriate balance between bluffs and good hands then you're putting them into that tough spot and so that's what we're trying to do we're always trying to search for a strategy that would put the opponent into a difficult position. Can you give a metric that you're trying to maximize or minimize does this have to do with the regret thing what we're talking about in terms of putting your opponent in a maximally tough spot. Yeah ultimately what you're trying to maximize is your expected winning so like your expected value the amount of money that you're going to walk away from assuming that your opponent was playing optimally in response so you're going to assume that your opponent is is also playing like as as well as possible a Nash equilibrium approach because if they're not then you're just going to make more money right like anything that deviates like by definition the Nash equilibrium is the strategy that does the best in expectation. And so if you're deviating from that then you're just they're going to lose money and since it's a two