克里斯·拉特納：編譯器、LLVM、Swift、TPU 與機器學習加速器

[00:00:00] The following is a conversation with Chris Flattener, currently he's a senior director at Google working on several projects, including CPU, GPU, GPU accelerators for for swift returns to flow and all kinds of machine learning compiler magic going on behind the scenes. He's one of the top experts in the world on compiler technologies, which means he deeply understands the intricacies of how hardware and software come together to create efficient code. He created that allow VM Compiler Infrastructure Project and the clang compiler. [00:00:33] He led major engineering efforts at Apple, including the creation of the swift programming language. He also briefly spent time at Tesla as vice president of Autopilots Software during the transition from auto pilot hardware one to hardware to when Tesla essentially started from scratch to build an in-house software infrastructure for autopilot. I could have easily talked to Chris for many more hours. Compiling code down across the levels of abstraction is one of the most fundamental and fascinating aspects of what computers do. [00:01:05] And he is one of the world experts in this process, its rigorous science and its messy, beautiful art. This conversation is part of the Artificial Intelligence podcast, if you enjoy it. Subscribe on YouTube, iTunes or simply connect with me on Twitter at Lux. Friedman spelled Frid. And now here's my conversation with Chris Laettner. What was the first program you've ever written? Our first program back and when was it I think I started as a kid and my parents got a basic programming book, and so when I started it was typing out programs from a book and seeing how they worked and then typing them in wrong and trying to figure out why they were not working. [00:02:08] Right. That kind of stuff. [00:02:10] So basically, what was the first language that you remember yourself maybe falling in love with? Like really connecting with? I don't know. [00:02:17] I mean, I feel like I've learned a lot along the way and each of them have a different special thing about them. So I started in Basic and then went like GW Basic, which was the thing back in the dark days and then upgraded to Kubasik. And eventually Quick Basic, which are all slightly more fancy versions of Microsoft, basically made the jump to Pascall and started doing machine language programming and assembly and Pasko, which was really cool through Pasko. [00:02:43] It was amazing for its they're eventually going to see C++ and then kind of did lots of other weird things. [00:02:50] So I feel like it took the dark path, which is the you could, you could have gone lisp. Yeah. [00:02:57] You get a higher level sort of functional philosophical hippy root. Instead you went into like the dark arts of the straight into the machine. [00:03:07] So started with Basic Pasko and then assembly and then wrote a lot of assembly and y eventually eventually did small talk and other things like that. But that was not the starting point. [00:03:19] But so what, what is this journey to see is that in high school. Is that in college I was in high school. [00:03:25] Yeah. So and then that was, it was really about trying to be able to do more powerful things than what Pascall could do and also to learn a different world. So it was really confusing to me with pointers and the syntax and everything. And it took a while but. Pascale's much more. Principals in various ways seems more. I mean, it has its historical roots, but it's not as easy to learn with pointers. There's this memory management. [00:03:56] Thing they have to become conscious of is that the first time you start to understand that there's resources that you're supposed to manage well, so you have that in Pascal as well. [00:04:05] But in Pasko, these like the carrot and sort of the star. And there's some small differences like that, but it's not about point or arithmetic. And and C, you end up thinking about how things get laid out in memory a lot more. And so in Pascal, you have allocating and allocating and owning the memory. But just the programs are simpler and you don't have to. [00:04:27] Well, for example, Pascal has a string tape and so you can think about a string instead of an array of characters which are consecutive in memory. And so it's a little bit of a higher level abstraction. [00:04:37] So let's get into it. Let's talk about LVM ceiling and compilers. Sure. So can you tell me first what all the ceiling are and how is it that you find yourself the creator and lead developer or one of the most powerful compiler optimization systems in use today? [00:04:57] Sure. So. I guess they're different things, so. Let's start with what is the compiler it's a good place to start. What are the phases of a compiler where the parts. Yeah, what is it? So what does even a compiler used for? [00:05:10] So the way the way I look at this is you have a two sided problem of humans that need to write code and then you have machines that need to run the program that the human wrote. And for lots of reasons, the humans don't want to be writing in binary and want to think about every piece of hardware. And so at the same time that you have lots of humans, you also have lots of kinds of hardware. And so compilers are the art of allowing humans to think of the level of abstraction that they want to think about and then get that program, get the thing that they wrote to run on a specific piece of hardware. [00:05:43] And the interesting and exciting part of all this is that there's now lots of different kinds of hardware, chips like X 86 and her PC and arm and things like that. [00:05:53] But also High-Performance accelerators for machine learning and other things like that are also just different kinds of hardware, GPS. These are new kinds of hardware. [00:06:00] And at the same time, on the programming side of it, you have your basic you have C, you have JavaScript, you have Python, you Swift. You have like lots of other languages that are all trying to talk to the human in a different way to make them more expressive and capable and powerful. And so. Compilers are the thing that goes from one to the other now and then from the very beginning to the end to end, and so you go from what the human wrote and programming languages end up being about, expressing intent not just for the compiler and the hardware, but the programming languages job is really to to capture an expression of what the programmer wanted that then can be maintained and adapted and evolved by other humans, as well as by the interpreter, by the compiler. [00:06:46] So so when you look at this problem, you have on the one hand, humans, which are complicated, you have hardware which is complicated. [00:06:53] And so compilers typically work in multiple phases. [00:06:56] And so the software engineering challenge that you have here is try to get maximum reuse out of the the amount of code, the right, because these compilers are very complicated. And so the way it typically works out is that you have something called a front end or a parser that is language specific. And so you have a C parser and that's what Clangers or C++ or JavaScript or Python or whatever. That's the front end. Then you'll have a middle part, which is often the optimizer, and then you have a late part, which is hardware specific. [00:07:30] And so compilers end up there's many different layers often, but these three big groups that are very common and compilers and what LVM is trying to do is trying to standardize that middle and last part. And so one of the cool things about LVM is that there are a lot of different languages that compile through to it. And so things like Swift, but also Julia Rust. Clang for c c++ subjective c like these are all very different languages, and they can all use the same optimization infrastructure which gets better performance and the same cogeneration infrastructure for hardware support. [00:08:05] And so LVM is really that that layer that is common that all these different specific compilers can use. [00:08:12] And is it is it a standard like a specification or is it literally an implementation set implementation? [00:08:19] And so it's I think there's a couple of different ways of looking at it. [00:08:23] Right, because it depends on which angle you're looking at it from. LVM ends up being a bunch of code. OK, so it's a bunch of code that people reuse and they build compilers with. [00:08:33] We call it a compiler infrastructure because it's kind of the underlying platform that you build a concrete compiler on top of. [00:08:39] But it's also a community. And the Muslim community is hundreds of people that all collaborate. And one of the most fascinating things about LVM over the course of time is that we've managed somehow to successfully get harsh competitors in the commercial space to collaborate on shared infrastructure. And so you have Google and Apple, you have AMD and Intel, you have Invidia and AMD. On the graphics side, you have Quray and everybody else doing these things. [00:09:09] And like all these companies are collaborating together to make that shared infrastructure really, really great. [00:09:15] And they do this not out of the goodness of their heart, but they do it because it's in their commercial interests of having really great infrastructure that they can build on top of and facing the reality that it's so expensive that no one company, even the big companies, no one company really wants to implement it all themselves, expensive or difficult? [00:09:33] Both. That's a great point, because it's also about the skill sets. Right. And these are the skill sets are very hard, hard to find. [00:09:43] How big is the LVM? It always seems like with open source projects, the kind of open source. [00:09:50] Yes, it's open source. It's about it's 19 years old now. So it's fairly old. [00:09:55] It seems like the magic often happens in a very small circle of people. Yes, I'd like at least the early birth and whatever. [00:10:03] Yes. So LVM came from a university project, and so I was at the University of Illinois and there it was myself and my advisor and then a team of two or three research students in the research group. And we built many of the core pieces initially. I then graduated and went to Apple and an apple brought it to the products, first in the open geographic stack, but eventually to the C compiler rom and eventually built clang and eventually built Swiftian. [00:10:30] These things along the way, building a team of people that are really amazing compiler engineers that helped build a lot of that. And so as it was gaining momentum and as was using it being open source and public and encouraging contribution, many others, for example, at Google came in and started contributing. And in some cases, Google effectively owns Clanged now because it cares so much about C++ and the evolution of that that ecosystem. And so it's investing a lot in the C++ world and the tooling and things like that. [00:11:00] And so likewise, Invidia cares a lot about Khuda. And so Kuda uses clang and uses LVM for for graphics and GPP. [00:11:12] And so when you first started as a massive project, I guess, do you think is going to go as far as it went where you crazy ambitious about it now? Seems like a really difficult undertaking. A brave one. [00:11:26] Yeah, no, it was nothing like that. So I mean, my goal when I went to the University of Illinois was to get in and out with a non thesis masters and a year and get back to work. [00:11:35] So I was not I was not planning to stay for five years and and build this massive infrastructure. I got nerds sniped into staying and a lot of it was because all of it was fun. I was building cool stuff and learning really interesting things and facing both software engineering challenges, but also learning how to work on a team and things like that. [00:11:57] I had worked at many companies as interns before that, but it was really a different a different thing to have a team of people that were working together and trying to collaborate and version control. And it was it was just a little bit different. [00:12:09] Like I said, I just talked to Don Knuth and he believes that two percent of the world population have something weird with their brain, that they're geeks. They understand computers, the connected computer, he put it exactly two percent. [00:12:21] OK, so this specific guy is very specific. He says I can't prove it, but it's very empirically there. [00:12:30] Is there something that attracts you to the idea of optimizing code? And it seems like that's one of the biggest, coolest things about all of you. [00:12:38] Yeah, that's one of the major things that does. So I got into that because of a person, actually. So when I was in my undergraduate, I had an advisor or a professor named Steve Victo, and he I went to this little tiny private school, the. The seven or nine people in my computer science department, students in my in my class, so it's a very tiny, very small school, it was kind of a work on the side of the math department kind of thing at the time. [00:13:08] I think it's evolved a lot in the many years since then. [00:13:10] But but Steve Ragdoll was a compiler guy and he was super passionate and he has passion rubbed off on me. And one of the things I like about compilers is that they're large, complicated software pieces. [00:13:26] And so one of the culminating classes that many computer science departments, at least at the time, did, was to say that you take algorithms and data structures and all these core classes. But then the compilers class was one of the last classes you take because it pulls everything together and then you work on one piece of code over the entire semester. And and so you keep building on your own work, which is really interesting, is also very challenging because in many classes, if you don't get a project done, you just forget about it and move on to the next one and get your, you know, your B or whatever it is. [00:13:58] But here you have to live with the decisions you make and continue to reinvest in it. [00:14:02] And I really like that. And and so I did an extra study project with them the following semester and he was just really great. And he was also a great mentor and a lot of ways. And so from from him and from his advice, he encouraged me to go to graduate school. I wasn't super excited about going to grad school. [00:14:20] I wanted the master's degree, but I didn't want to be an academic. [00:14:24] And but like I said, I kind of got tricked into saying and was having a lot of fun, and I definitely do not regret it. [00:14:31] What aspects of compilers were the things you connected with? So LVM, there's also the other part. [00:14:39] This is really interesting. If you're interested in languages parsing and yeah. Just analyzing like. Yeah, analyzing language, breaking it up, parsing and so on. Was that interesting to you? Were you more interested in optimization? For me it was more so. [00:14:52] I'm not really a math person. I could do math. I understand some bits of it when I get into it. But math is never the thing that that attracted me. And so a lot of the parser, part of the compiler has a lot of good formal theories that Don, for example, knows quite well. Still waiting for his book on that. [00:15:08] But but the but I just like building a thing and and seeing what it could do and exploring and getting to do more things and then setting new goals and reaching for them and and with in the case of in the case of LVM, when I started working on that, my research advisor that I was working for was a compiler guy. And so he and I specifically found each other because we're both interested in compilers. And so I started working with them and taking this class. [00:15:36] And a lot of LVM initially was it's funny implementing all the standard algorithms and all the other things that people had been talking about and were well known. And they were in the the curricula for advanced days and compilers. And so just being able to build that was really fun. And I was learning a lot by sort of reading about just building. And so I enjoyed that. [00:15:57] So he said composing these complicated systems. Q Even just with language, try to describe, you know, how you turn a C++ program. Yes. [00:16:08] Into code like what are the hard parts, why is it so hard? [00:16:11] So I'll give you examples of the hard parts along the way. So C++ is a very complicated programming language is something like 400 pages in the spec. So C++ by itself is crazy. [00:16:23] Can you just say pause? What makes a language complicated in terms of what's syntactically like us? [00:16:30] So it's what they call syntax. So the actual how the characters are arranged. Yes, it's also semantics how it behaves. It's also in the case of C++, there's a huge amount of history. [00:16:40] C++ built on top of C, you play that forward and then a bunch of suboptimal in some cases decisions were made and they compound and then more and more and more things keep getting added to C++ and it will probably never stop. [00:16:54] But the language is very complicated from that perspective. And so the interactions between subsystems is very complicated. There's just a lot there. And when you talk about the front end, one of the major challenges which playing as a project, the C C++ compiler that I built, I and many people built all the challenges we took on was. [00:17:13] We looked at Jesse, Jesse at the time was like a really good industry standardized compiler that had really consolidated a lot of the other computers in the world and was was a standard, but it wasn't really great for research. [00:17:27] The design was very difficult to work with and it was full of global variables and other other things that made it very difficult to reuse in ways that it wasn't originally designed for. And so with closing, one of the things that we want to do is push forward on better user interface. So make error messages that are just better than Jinx's. And that's actually hard because you have to do a lot of bookkeeping in an efficient way to do that. [00:17:50] We want to make compile time better. And so compile time is about making it efficient, which is also a really hard way of keeping track of extra information. [00:17:57] We wanted to make new tools available, so refactoring tools and other analysis tools that the GCSE never supported. Also leveraging the extra information we kept, but enabling those new classes are tools that then get built into ideas. [00:18:13] And so that's been one of the one of the areas that clangers really helped push the world forward in is in the tooling for C and C++ and things like that. [00:18:22] But C++ and the front end pieces complicated. And you have to build syntax trees and you have to check every rule in the spec and you have to turn that back into an error message to the human human that the human can understand when they do something wrong. [00:18:34] But then you start doing the what's called lowering. So going from C++ in the way that it represents code down to the machine. And when you do that, there's many different phases you go through. [00:18:46] Often there are, I think, Alabama, something like 150 different what are called passes and the compiler that the code passes through and these get organized in very complicated ways which affect the generate code in the performance and compile time and many of the things. [00:19:03] What are they passing through? So after you do the the clang parsing, what's what's the graph? [00:19:11] What does it look like? What's the structure here. [00:19:13] Yeah. So in, in the parser it's usually a tree and it's called an abstract syntax tree. And so the idea is you, you have a node for the plus that the human wrote in their code or the function call. You'll have a node for call with the function that they call in the arguments they pass, things like that. This then gets lowered into what's called an intermediate representation, and intermediate representations are like LVM has one and there it's a it's what's called a control photograph. [00:19:44] And so you represent each operation in the program as a very simple like this is going to add two numbers. This is going to multiply two things. [00:19:53] Maybe we'll do a call, but then they get put in what are called blocks. And so you get blocks of these straight line operations where instead of being nested like in a tree, it's straight line operations. And so there's a sequence and we're going to these operations and then in the block or outside, that's within the block. [00:20:10] And so it's a straight line sequence of operations within the block. [00:20:12] And then you have branches like additional branches between blocks. And so when you write a loop, for example, in a syntax tree, you would have a four node like four for statement in a C language for node. And you have a pointer to the expression for the initialize are appointed to the expression, for the incremental pointer, to the expression for the comparison appointer to the body. OK, and these are all nested underneath it. In a controller graph, you get a block for the code that runs before the loop. [00:20:43] So the initialized code and you have a block for the body of the loop. And so the the body of the loop code goes in there, but also the increment, other things like that. [00:20:52] And then you have a branch that goes back to the top and a comparison and branch that goes out. And so it's more of an assembly level kind of representation. But the nice thing about this level of representation is it's much more language independent. [00:21:05] And so there's lots of different kinds of languages with different kinds of, you know, JavaScript has a lot of different ideas of what is false, for example, and all that can stay in the front end. [00:21:17] But then that middle part can be shared across all of those. [00:21:21] How close is that intermediate representation to a 10 year old network, for example? Are they? Because everything you describe is a kind of echoes of neural network graph. Yes, we are. [00:21:34] The neighbors are what they're quite different in details, but they're very similar in idea. So one of the things that all networks do is they learn representations for data at different levels of abstraction and then they transform those through layers. Right. [00:21:51] So the compiler does very similar things. But one of the things the compiler does is it has relatively few different representations where a neural network, often as you get deeper, for example, you get many different representations and each, you know, layer or set of ops is transforming between these different representations. In a compiler, often you get one representation and they do many transformations to it. And these transformations are often applied iteratively. [00:22:16] And for programmers that are familiar types of things, for example, trying to find expressions inside of a loop and pulling them out of a loop so they execute four times or find redundant computation or find a constant folding or other simplifications turning, you know, to times ex into shift left by one. And things like this are all of all the examples of the things that happen. [00:22:40] But compilers end up getting a lot of things improving and other kinds of algorithms that try to find higher level properties of the program that then can be used by the optimizer. [00:22:49] So what's the biggest bang for the buck with optimization? What's today? Yeah, well, no, not even today at the very beginning, the 80s, I don't know, but yeah. [00:23:00] So for the 80s, a lot of it was things like register allocation. [00:23:03] So the idea of in in a modern like a microprocessor, what you'll end up having is you'll end up having memory which is relatively slow, and then you have registers relatively fast, but registers, you don't have very many of them. [00:23:17] And so when you're writing a bunch of code, you're just saying like compute this put in temporary variable computers, computers, computers, put in temporary variable over a loop. I have some other stuff going on. Well, now you're running on an X eighty six like a desktop PC or something. Well, it only has in some cases, some modes it registers. Right. And so now the compiler has to choose what values get put in, what registers at one points in the program. [00:23:41] And this is actually a really big deal. So if you think about you have a loop and an inner loop, the excuse millions of times, maybe if you're doing loads in stores inside that loop, then it's going to be really slow. But if you can somehow fit all the values inside that loop in registers, now it's really fast. And so getting that right requires a lot of work because there's many different ways to do that. And often what the compiler ends up doing is it ends up thinking about things in a different representation than what the human wrote. [00:24:09] Right. You wrote into X while the compiler thinks about that is for different values, each which have different lifetimes across the function that it's in. And each of those could be put in a register or memory or different memory or maybe in some parts of the code recompute it instead of extraordinarily loaded. And there are many of these different kinds of techniques that can be used so that adding almost like a time. [00:24:31] Dimensioned is trying to trying to optimize across time, so it's considering when when you're programming, you're not thinking. [00:24:39] Yeah, absolutely. And so the risk error. Made the risk so so risk gips risc the risk risk chuffs as opposed to the risk chips, made things more complicated for the compiler because what they ended up doing is ending up adding pipelines to the processor or the processor can do more than one thing at a time. But this means that the order of operations matters a lot. [00:25:04] And so one of the classical compiler techniques that you use is called scheduling. And so moving the instructions around so that the processor can keep its pipelines full instead of stalling and getting blocked. And so there's a lot of things like that that are kind of bread and butter. A compiler techniques have been studied a lot over the course of decades now, but the engineering side of making them real is also still quite hard. And you talk about machine learning. This is this is a huge opportunity for machine learning because many of these algorithms are full of these like hokey handwrote heuristics, which work well on specific benchmarks that don't generalize and full of magic numbers. [00:25:41] And, you know, I hear there's some techniques that are good at handling that. [00:25:45] So what would be the if you were to apply machine learning to this? What's the thing you try to optimize is ultimately the running time? [00:25:55] Of course, you can pick your metric and there's there's running time, there's memory use. There's there's lots of different things that you can optimize for. A good size is another one that some people care about in the embedded space. [00:26:05] Is this like the thinking into the future or somebody's actually been crazy enough to try to have machine learning based parameter tuning for the optimization of composers? [00:26:18] So this is something that is, I would say, research right now. There are a lot of research systems that have been applying search in various forms and using reinforcement learning as one form, but also for search has been tried for quite a while. [00:26:31] And usually these are small, small problem spaces. [00:26:35] So find the optimal way to generate a matrix multiplier for a GPU, something like that, where you say there there's a lot of design space of do you unroll loops a lot? Do you execute multiple things in parallel? [00:26:49] And there's many different confounding factors here because graphics cards have different numbers of threads and registers and execution parts and memory bandwidth and many different constraints to interact and nonlinear ways. And so search is very powerful for that and. It gets used in in certain ways, but it's not very structured. This is something that we need. We as an industry need to fix these set 80s, but they. So have there been like big jumps in improvement and optimization? Yes. [00:27:19] Since then. What? [00:27:21] Yeah. So it's largely been driven by hardware. So Heartwell hardware and software. [00:27:27] So in the mid 90s, Java totally changed the world and I'm still amazed by how much change was introduced, by the way, or in a good way. So like reflecting back, Java introduced things like it all at once, introduced things like compilation. [00:27:43] None of these were novel, but it pulled it together and made it mainstream and and made people Envestnet compilation, garbage collection, portable code, safe code, like memory, safe code, like a very dynamic dispatch execution model. [00:27:58] Like many of these things which had been done in research systems and had been done in small ways and various places really came to the forefront, really changed how things worked and therefore changed the way people thought about the problem. JavaScript was another major world change based on the way it works, but also on the hardware side of things. [00:28:19] Multicore and vector instructions really change the problem space and are very they don't remove any of the problems the Campo's faced in the past, but they they add new kinds of problems of how do you find enough work to keep a four wide vector busy? Or if you're doing a matrix multiplication, how do you do different columns? Are that matrix and at the same time? And how do you maximally utilize the the arithmetic compute that one core has? And then how do you take it to multiple cores? [00:28:50] And how did the whole virtual machine thing change the computation pipeline. The yeah. So, so what the Java virtual machine does is it splits just like I was talking about before, where you have a front end that passes the code and then you have an intermediate representation that gets transformed, which I ever did, was they said we will pass the code and then compile to what's known as Java bytecode, and that bytecode is now a portable code representation that is industry standard and lock down and can't change. [00:29:19] And then the the back part of the compiler, the does optimization and co-generation can now be built by different vendors and Java bytecode can be shipped around across the wire. [00:29:30] It's memory safe and relatively trusted. [00:29:33] And because of that it can run in the browser and that's why it runs in the browser. Right. And so that way you can be in, you know, again back in the day you write Java applet and you'd use a as a as a web developer, you'd build this mini app to run a webpage. Well, a user of that is running a Web browser on their computer. You download that that Java bytecode, which can be trusted, and then you do all the compiler stuff on your machine. [00:29:58] So you know that you trust that that was a good idea. [00:30:00] A bad idea. It's a great idea. I mean, it's a great idea for certain problems. And I'm very much a believer that technology is itself neither good nor bad. [00:30:07] It's how you apply it. [00:30:10] You know, this would be a very, very bad thing for very low levels of the software stack. [00:30:14] But but in terms of solving some of these software portability and transparency or portability problems, I think it's been really good. [00:30:21] Now, Java ultimately didn't win out on the desktop and like there are good reasons for that, but it's been very successful on servers. And in many places it's been a very successful thing over over decades. [00:30:33] So what has been low VMS and ceilings, improvements in optimization, that is throughout its history. [00:30:45] What are some moments we had set back and really proud of what's been accomplished? [00:30:50] Yeah, I think that the interesting thing about LVM is not the innovations and compiler research. It has very good implementations of various important algorithms, no doubt. [00:31:01] And and a lot of really smart people have worked on it. But I think that the thing that's most profound about LVM is that through standardization, it made things possible that otherwise wouldn't have happened. OK, and so interesting things that have happened with LVM, for example, Sony has picked up LVM and used it to do all the graphics compilation and their movie production pipeline. And so now they're able to have better special effects because of LVM. [00:31:26] That's kind of cool, that's not what it was designed for, right, but that's that's the sign of good infrastructure when it can be used in ways it was never designed for because it has good layering and software engineering and composable and things like that, which is where, as you said, it differs from JC. [00:31:43] Yes, Jesus is also great in various ways, but it's not as good as infrastructure technology. It's it's you know, it's really a C compiler or it's or it's a forgery compiler. It's not it's not infrastructure in the same way now, is it? [00:31:57] Now you can tell I don't know what I'm talking about because I keep saying cieling. [00:32:01] You can always tell when a person is close by the way you pronounce something. I don't think. [00:32:07] Have I ever used Klencke entirely possible. Have you? Well, so you've used code. It's generated probably. So Clang is in Alabama, used to compile all the apps on the iPhone effectively and the OS as it compiles Google's production server applications. It's used to build like Game Cube games and PlayStation four and things like that, as a user I have, but just everything I've done that I experienced to Linux has been, I believe, always geeky. [00:32:40] You know, I think Linux stole the hostages. This and is the reason for that. [00:32:45] There's a big I mean, it's a combination of technical and social reasons. Many, many Linux developers, D. D clone. But the distribution's for lots of reasons. Eustachy historically. [00:32:59] And they've not switched. No, because I just anecdotally online, it seems that LVM has either reached a level GGC or superceding different features or whatever. [00:33:10] And the way I would say it is that they're so close it doesn't matter. Yeah, exactly. Like they're slightly better in some ways slightly worse than other, but it doesn't actually really matter anymore that level. [00:33:20] So in terms of optimization, breakthrough's, it's just been solid, incremental work. Yeah. Yeah. [00:33:27] Which is which describes a lot of compilers. The hard thing about compilers in my experience, is the engineering, the software engineering, making it so that you can have hundreds of people collaborating on really detailed low level work and scaling that. [00:33:42] And that's that's really hard. And that's one of the things I think LVM has done well. And that kind of goes back to the original design goals with it to be modular and things like that, and incidentally, I don't want to take all the credit for this. Right. I mean, some of the the best parts about LVM is that it was designed to be modular. And when I started, I would write, for example, or register Elkader and then something much smarter than me would come in and pull it out and replace it with something else that they would come up with. [00:34:09] And because it's modular, they were able to do that. And that's one of the challenges with what you see, for example, is replacing subsystems is incredibly difficult. It can be done, but it wasn't designed for that. And that's one of the reasons that Alvin's been very successful in the research world as well. [00:34:25] But in the in a community sense, would of an Rassam right from Python, just retired from. [00:34:35] What does a benevolent dictator for life, right, so in managing this community of brilliant, compassionate folks is there I did it for a time at least following you to to approve things. [00:34:48] Oh, yeah. [00:34:49] So, I mean, I still have something like an order of magnitude more patches in LVM than anybody else. And many of those I wrote myself. [00:34:59] But you still write I mean, you still you still close to the to the I don't know what the expression is to the metal. You still write code. [00:35:08] Yeah. Sorry, I could not as much as I was able to in grad school, but that's an important part of my identity. But the way the LVM has worked over time is that when I was a grad student, I could do all the work and steer everything and review every patch and make sure everything was done exactly the way my opinionated sense felt like it should be done. And that was fine. [00:35:28] But as a scale, you can't do that. Right. And so what ends up happening is LVM has a hierarchical system of what's called owners. These code owners are given the responsibility not to do all the work, not necessarily to review all the patches, but to make sure that the patches do get reviewed and make sure that the right things happening architecturally in their area. [00:35:49] And so what you'll see is you'll see that, for example, hardware manufacturers end up owning the the the hardware specific parts of their their their hardware. That's very common. [00:36:02] Leaders in the community that have done really good work and actually become the de facto owner of something. And then usually something else is like, how about we make them the official code of honor? [00:36:12] And then and then we'll have somebody to make sure that the patches get reviewed in a timely manner. And then I was like, yes, that's obvious. And then it happens. Right. And usually this is a very organic thing, which is great. And so I'm nominally the top of that stack still. [00:36:25] But I don't spend a lot of time reviewing patches. [00:36:28] What I do is I help negotiate a lot of the the technical disagreements that end up happening and making sure that the community as a whole makes progress and is moving in the right direction and and doing that. So we also started a non-profit six years ago, seven years ago. [00:36:46] It's times gone away. And the nonprofit, the LVM Foundation nonprofit, helps oversee all the business sides of things and make sure that the events that the community has are funded and set up and run crackly and stuff like that. But the foundation is very much stays out of the technical side of where where the project is going. Right. [00:37:06] That sounds like a lot of it is just organic and just. [00:37:10] Yeah, well, and this is LVM is almost 20 years old, which is hard to believe. Somebody pointed out to me recently that LVM is now older than Jinksy was when LVM started. [00:37:21] All right, so time has a way of getting away from you, but the good thing about that is it has a really robust, really amazing community of people that are in their professional lives right across lots of different companies. But it's a it's a community of people that are interested in similar kinds of problems and have been working together effectively for years and have a lot of trust and respect for each other. And even if they don't always agree that, you know, we're able to find a path forward. [00:37:48] So then in a slightly different flavor of effort, you started Apple in 2005 with the task of making, I guess, LVM production ready and then eventually 2013 through 2017, leading the entire developer tools department. We're talking about LVM, Xcode, Objective C to Swift. [00:38:11] So in a quick overview of your time there, what were the challenges, first of all, leading such a huge group of developers, what was the big motivator dream mission behind creating Swift, the early birth of its from projects and so on, and Xcode? [00:38:31] So these are different questions. [00:38:33] I know, but I want to talk about the other stuff. I'll stay I'll stay on the technical side, then we can talk about the big team pieces. OK, sure. So it's to really oversimplify many years of hard work. [00:38:46] Alvin started, joined Apple, became a thing, became successful and became deployed. But then there's a question about how how do we actually past the source code? So LVM is the back part, the optimizer and the code generator. And LVM is really good for Apple. As it went through a couple of hundred transitions. I joined right at the time of the Intel transition, for example, and 64 bit transitions and then the transition to arm with the iPhone. [00:39:10] And so LVM was very useful for some of these kinds of things. But at the same time, there's a lot of questions around developer experience. [00:39:17] And so if you're a programmer pounding out at the time, Objective C-code, the error message, you get, the compile time, the turnaround cycle, the the tooling and the idea were not great or not as good as they could be. [00:39:30] And so, you know, as I occasionally do, I'm like, well, OK, how hard is it to write a C compiler? [00:39:37] Right. And so I'm not going to commit to anybody. And I could tell anybody. I'm just going to just do it on nights and weekends and start working on it. [00:39:44] And then, you know, I built up and see, there's this thing called the preprocessor, which people don't like, but it's actually really hard and complicated and includes a bunch of really weird things like try graphs and other stuff like that that are that are really nasty. [00:39:58] And it's the crux of a bunch of the performance issues in the compiler sort of working on the parser and kind of got to the point where I'm like, oh, you know what we can actually do? [00:40:06] This is everything that this is impossible to do, but it's actually just hard. It's not impossible. And I eventually told my manager about it and he's like, oh, wow, this is great. [00:40:16] We do need to solve this problem. Oh, this is great. We can get you one other person to work with you on this, you know, and so the team is formed and it starts taking off. [00:40:25] And C++, for example, huge, complicated language. People always assume that it's impossible to implement and it's very nearly impossible, but it's just really, really hard. [00:40:35] And the way to get there is to build it one piece at a time, incrementally. [00:40:39] And and there that was only possible because we were lucky to hire some really exceptional engineers that that new various parts of it very well and and could do great things. [00:40:49] Swift was kind of a similar thing. [00:40:51] So Swift came from we were just finishing off the first version of C++ support in Klang. [00:40:59] And C++ is a very formidable and very important language, but it's also ugly and of ways. And you can't implement C++ without thinking there has to be a better thing. Right. [00:41:11] And so I started working on Swift again with no hope or ambition that would go anywhere just to see what could be done. Let's play around with this thing. It was, you know, me in my spare time not telling anybody about it kind of a thing. And it made some good progress. I'm like, actually, it would make sense to do this. At the same time, I started talking with the senior VP of software at the time, a guy named Transfer there. [00:41:34] And Bertran was very encouraging. He was like, well, you know, let's let's have fun. Let's talk about this. And he was a little bit of a language guy. [00:41:40] And so he helped guide some of the early work and encouraged me and like, got things off the ground and eventually told my manager and told other people and and it started making progress. [00:41:56] The the complicating thing was swift, was that the idea of doing a new language is not obvious to anybody, including myself. [00:42:04] And the tone at the time was that the iPhone was successful because of Objective C. Right. Oh, interesting. In spite of or just because of and you have to understand that at the time Apple was hiring software people that loved Objective C, right. [00:42:22] And it wasn't that they came despite Objective C, they loved objectively, and that's why they got hired. [00:42:27] And so you had a software team that the leadership and in many cases went all the way back to next where Objective C really became real. [00:42:36] And so they, quote unquote grew up writing Objective C and many of the individual engineers all were hired because they love to see. [00:42:45] And so this notion of, OK, let's do new language was kind of heretical in many ways. Right. [00:42:51] Meanwhile, my sense was that the Alawite community wasn't really in love with objectivity. Some people were and some of the most outspoken people were. [00:42:57] But other people were hitting challenges because it has very sharp corners and it's difficult to learn. [00:43:03] And so one of the challenges of making swift happen that was totally non-technical is the the social part of what do we do? [00:43:14] Like if we do a new language, which at Apple, many things happen that don't ship. Right. So if we if we ship it, what what what is the metrics of success? [00:43:22] Why would we do this? Why wouldn't we make objectively better if objectively as problems, let's file off those rough corners and edges. And one of the major things that became the reason to do this was this notion of safety, memory, safety. [00:43:36] And the way Objective C works is that a lot of the object system and everything else is built on top of pointers and see objectively as an extension on top of C. [00:43:47] And so pointers are unsafe and if you get rid of the pointers, it's not objective C anymore. [00:43:53] All right. And so fundamentally, that was an issue that you could not fix safety or memory safety without fundamentally changing the language. [00:44:02] And so once we got through that part of the the mental process and the thought process, it became a design process of saying, OK, well, if we're going to do something new, what what is good? [00:44:13] Like, how do we think about this and what are we like and what are we looking for? [00:44:16] And and that that was a very different phase of it. [00:44:19] So. Well, what are some design choices early on and swift like we're talking about Brace's. [00:44:26] Are you making a type language? You know, all those kinds of things. Yeah. [00:44:30] So so some of those were obvious given the context. [00:44:33] So I type language, for example, objectiveness, a type language and going with another type of language wasn't really seriously considered. [00:44:41] We wanted we wanted the performance and we wanted refactoring tools and other things like that to go with type languages. [00:44:46] Quick dumb question. [00:44:48] Yeah. Was it obvious? I think this would be a dumb question, but was it obvious that the language has to be a compiler language? Not a yes. [00:44:57] That's not a dumb question. Earlier, I think late 90s, Apple is seriously considering moving its development experience to Java. But that's what started in 2010, which was several years after the iPhone, was when the iPhone was definitely on an upward trajectory and the iPhone was still extremely and is still a bit memory constrained. [00:45:18] And so being able to compile the code and then ship it and then have having standalone code that is not yet compiled is a very big deal and it's very much part of the apple value system. Now, JavaScript is also a thing. [00:45:34] I mean, it's not it's not that this is exclusive and technologies are good depending on how they're applied. Right. [00:45:41] But in the design of SWIFT saying, like, how can we make objectively better objectively of statically comp. And that was the contiguous natural thing to do. [00:45:49] Just skip ahead a little bit now, right back. Just just as a question, as you think about today in 2019 in your work at Google and so on, is again, compilations, static compilation. The right the still the right thing. Yeah. [00:46:06] So the funny thing, after working on compilers for a really long time, is that and this is one of the things that LVM has helped with is that I don't look as a compilation as being static or dynamic or interpreted or not. This is a spectrum. [00:46:24] And one of the cool things about Swift is that Swift is not just statically compiled, it's actually dynamically compiled as well. And it can also be interpreted that nobody's actually done that. And so what what ends up happening when you use Swift in the workbook, for example, and Coolabah and Jupiter is it's actually dynamically compiling the statements as you execute them. And so this gets back to the the software engineering problems, right. Where if you layer the stack properly, you can actually completely change how and when things get compiled because you have the right abstractions there. [00:46:58] And so the way that a CoLab workbook works was swift is that we start tapping into it. [00:47:04] It creates a process, a Unix process, and then each line of code you type in, it compiles it through the Swift compiler, the front end part and then sends it through the optimizer compiles machine code and then injects it into that process. And so as you're typing new stuff, it's putting it's like squirting a new code and overwriting and replacing and updating code in place, and the fact that it can do this is not an accident like it was designed for this. [00:47:32] But it's an important part of how the language was set up and how it's layered. And and this is a non obvious piece. And one of the things was swift. That was, for me, a very strong design point is to make it so that you can learn it very quickly. And so from a language design perspective, the thing that I always come back to is the UI principle of progressive disclosure of complexity. And so in Swift, you can start by saying print quote, hello, world quote. [00:48:00] Right. [00:48:01] And there's no solution. Just like Python. [00:48:03] One line of code, no man, no files, no header files, no. Public static, class void blah blah blah. String like Java has. Right. One line of code. Right. And you can teach that and great. [00:48:15] They can say, well, let's introduce variables and so you can declare variable far Savar X equals four. What is a variable. You can use X, X plus one. This is what it means. Then you can say, well how about contraflow. Well this is one if statement is this is what a fourth statement is. [00:48:29] This is what a wild statement is, then you can say let's introduce functions and and many languages like Python have had this this kind of notion of let's introduce small things and they can add complexity, they can introduce classes and they can add generics in the case as well. And then you can add modules and build out in terms of the things that you're expressing. But this is not very typical for compiled languages. And so this was a very strong design point. [00:48:55] And one of the reasons that Swift in general is designed with this factoring of complexity in mind so that the language can express powerful things. [00:49:03] You can write from Rantes if you want to, but it has a very high level feel, which is really this perfect blend, because often you have very advanced library writers that want to be able to use the the nitty gritty details. But then other people just want to use the libraries and work at a higher abstraction level. It's kind of cool there so that you can just interoperability. [00:49:26] I don't think I pronounce that word enough, but you can just drag in Python. It's just you can import like so this is the most important number, how do you make that happen? Yeah, let's say is that as easy as it looks or is it. Yes. [00:49:43] Look, that's not that's not a stage magic act or anything like that. [00:49:46] I don't mean from the user perspective, I mean from the implementation perspective to make it happen. [00:49:51] So it's easy once all the pieces are in place, the way it works. So if you think about it, dynamically typed language like Python, you can think about it in two different ways. You can say it has no type's right, which is what most people would say. Or you can say it has one type. Right. And you can say it has one type. And it's like the python object and the python object is passed around. And because there's only one type, it's implicit. [00:50:14] OK. And so what happens was Swift and Python talking to each other. [00:50:18] Swift has lots of types, has arrays and it has strings and all classes and that kind of stuff. [00:50:24] But it now has a python object type. So there is one python object type. And so when you say import Nampo, what you get is a python object, which is the module. [00:50:37] Then you say expeditor says, OK, hey, hey, python object. I have no idea what you are. [00:50:42] Give me your remember. OK, cool, and just it just uses dynamic stuff, talks to the Python interpreter and says, hey, Python, what's the daughter, remember, in that python object gives you back another python object. [00:50:54] And I say parentheses for the call and the arguments are going to pass. And so then it says, hey, a python object that is the result of expeditor call with these arguments again, calling into the python interpreter to do that work. [00:51:07] And so right now, this is all really simple. And if you if you dive into the code, what you'll see is that the the python module and swift is something like twelve hundred lines of code or something is written in pure swift. [00:51:19] It's super simple and it's and it's built on top of the C interoperability because it just talks to the Python interpreter. [00:51:26] But making that possible requires us to add two major language features to Swift to be able to express these dynamic calls and the dynamic member lookups. And so what we've done over the last year is we've proposed implement standardized and contributed new language features to the swift language in order to make it so it is really trivial. Right. And this is one of the things about Swift that is critical to the 10th floor work, which is that we can actually add new language features and the bar for adding those is higher. [00:51:56] But it's it's what makes it possible. [00:51:59] So, you know, Google doing incredible work on several things, including of so terms of Flow 2.0 or whatever, leading up to 2.0 has by default in 2.0, has Eger execution in. [00:52:14] Yet in order to make code optimized for GPU, ATP or some of these systems, computation needs to be converted to a graph. So what's that process like? What are the challenges there? [00:52:26] Yeah, so I'm tangentially involved in this, but the, the way that it works with autograph is that. [00:52:34] You mark your of your function with the decorator, and when Python calls it, that decorator is invoked and then it says, before I call this function, you can transform it. And so the way autograph works is, as far as I understand, is it actually uses the python parser to go past that, turn into a syntax tree and apply compiler techniques to again transform this down into sensible graphs. And so it you can think of it as saying, hey, I have an if statement I'm going to create and if node in the graph like you say, can't you have a multiply or I'll turn that into a multiplying node in the graph and that becomes this tree transformation. [00:53:14] So where does the swift photons come in, which is you know, Perla's for one Swifties interface, like Python is an interface. It has a flow, but it seems like there's a lot more going on and just a different language interface. There's optimization methodology. [00:53:32] So so the center of the world has a couple of different what I'd call front end technologies and so Swift and Python and Go and Rostand Julia and all these things share the tenth of the graphs and all the runtime and everything that's later on. [00:53:49] And so photons flow is merely another front end for tensor flow, just like any of these other systems. Or there's a major difference between, I would say, three camps of technology. There's Python, which is a special case because the vast majority of the community effort is going to the Python interface and Python has its own approaches for automatic differentiation, has its own APIs and all this kind of stuff there. Swift, which I'll talk about in a second, and then there's kind of everything else. [00:54:19] And so the everything else in our effectively language bindings. So they call into the tense flow runtime, but they're not they usually don't have automatic differentiation or they usually don't provide anything other than APIs that call the KPIs intense workflow. [00:54:33] And so they're kind of wrappers for that Swifties really kind of special. And it's a very different approach. [00:54:39] Swift returns for that is this is a very different approach because they're saying, let's look at all the problems that need to be solved in the full stack of the Tenterfield compilation process. [00:54:51] If you think about it that way, because Tenzer Flow is fundamentally a compiler, it takes models and then it makes them go fast on hardware. [00:54:59] Mm hmm. That's what a compiler does. And it has a front end, has an optimizer, and it has many backhands. And so if you think about it the right way or in and if you look at it in a particular way, like it is a compiler. [00:55:14] And and so swift is merely another front end. [00:55:19] But it's saying in the design principle saying, let's look at all the problems that we face as machine learning practitioners and what is the best possible way we can do that, given the fact that we can change literally anything in this entire stack in Python, for example, where the vast majority of the engineering and an effort has gone into is constrained by being the best possible thing you can do with the Python library, like there are no Python language features that are added because of machine learning that I'm aware of. [00:55:49] They added a matrix multiplication operator with that, but that's as close as you get. [00:55:55] And so with Swift, you can you it's hard, but you can add language features to the language and there's a community process for that. And so we look at these things and say, well, what is the right division of labor between the human programmer and the compiler? And Swift has a number of things that shift that balance. So because it's a because it has a type system, for example, it makes certain things possible for analysis of the code and the compiler can automatically build graphs for you without you thinking about them. [00:56:25] Like that's that's a big deal for a programmer. You just get free performance, you get clustering, infusion and optimization, things like that, without you as a programmer having to manually do it because the compiler can do it for you. UniMac differentiation. There's another big deal. And I think one of the key contributions of the Swift Flow Project. [00:56:45] Is that there's this entire body of work on automatic differentiation that dates back to the fourth trend is people doing a tremendous amount of numerical computing in Fortran used to rate these what they call source to source translators, where you where you take a bunch of code and shove it into a mini compiler and push out more Fortran code. But it would generate the bakradze passes for your functions, for you, the derivatives. Right. [00:57:10] And so in that work in the 70s, a tremendous number of optimizations, a tremendous number of techniques for fixing numerical instability and other other kinds of problems were developed. [00:57:22] But they're very difficult to port into a world where in your execution you get an update at a time like you need to be able to look at entire function and be able to reason about what's going on. And so when you have a language integrated automatic differentiation, which is one of the things that the Swift Project is focusing on, you can open up and all these techniques and reuse them in familiar ways. [00:57:45] But the language integration piece has a bunch of design room and it's also complicated. [00:57:50] The other piece of the puzzle here that's kind of interesting is to use a Google. Yes. So, you know, we're in a new world with deep learning. It's constantly changing. And I imagine without disclosing anything, I imagine, you know, you're still innovating. And the CPU front, too. Indeed. [00:58:06] So how much sort of interplay between software and hardware and trying to figure out how to together move towards an optimal solution? There's an incredible amount. So we're on our third generation of TVs, which are now 100 per teraflops and a very large liquid called box virtual box with no color. And as you might imagine, we're not out of ideas yet. The great thing about TPS is that they're a perfect example of hardware, software, code design. And so it's about it's about saying what hardware do we build to solve certain classes of machine learning problems? [00:58:41] Well, the algorithms are changing, like the hardware takes, you know, some cases years to produce. Right. And so you have to make bets and decide what is going to happen. And so and what is the best way to spend the transistors to get the maximum, you know, performance per watt or area cost or whatever it is that you're optimizing for. And so one of the amazing things about TV is, is this numeric format called Beefalo 16 before 16 is a compressed 16 bit floating point format. [00:59:11] But it puts the bits in different places in numeric terms. It has a smaller amount and a larger exponent. That means that it's less precise, but it can represent larger ranges values, which in the machine learning context is really important and useful because sometimes you have very small gradients you want to accumulate and very, very small numbers that are important to to move things as you're learning. But sometimes you have very large magnitude numbers as well. And before 16 is not as precise the a small but it turns out the machine learning algorithms actually want to generalize. [00:59:48] And so there's, you know, theories that this actually increases the ability for the network to generalize across datasets. And regardless of whether it's good or bad, it's much cheaper at the hardware level to implement because the area and time of a multiplier is and squared in the number of bits in the mantis. But it's linear with size of the exponent connected to solid efforts here, both on the hardware and software side. [01:00:14] Yeah, and so that was a breakthrough coming from the research side and people working on optimizing network transport of weights across the network originally and try and find ways to compress that. But then it got burned into silicon. And it's a key part of what makes Tipu performance so amazing and and and great now to so many different aspects of it that are important. [01:00:37] But the the code design between the low level compiler bits and the software bits and the algorithms is all super important. And it's this amazing trifecta that only Google can do. [01:00:50] Yeah, that's super exciting. [01:00:51] So can you tell me about LIRR project previously? This the secretive one? [01:00:58] Yeah, so am I. Is a project that we announced at a compiler conference. Three weeks ago or something, the Compilers for Machine Learning Conference, basically, again, if you look at tensor flow as a compiler stack has a number of compiler algorithms within it. [01:01:13] It also has a number of compilers that get embedded into it, and they're made by different vendors. For example, Google has Xolair, which is a great compiler system and video has 10 30. Intel has an Grauwe. There's a number of these different compiler systems and they're very specific and they're trying to solve different parts of the problems. But they're all kind of similar in a sense of they want to integrate with Tenzer. So that has an optimizer and it has these different cogeneration technologies built in the idea. [01:01:43] Volaris to build a common infrastructure to support all these different subsystems. And initially it's to be able to make it so that they all plug in together and they can share a lot more code and can be reusable. But over time, we hope that the industry will start collaborating and sharing code. And instead of reinventing the same things over and over again, that we can actually foster some of that, that, you know, working together to solve common problem energy that has been useful in the compiler field before. [01:02:11] Beyond that, I mean, there is some people have joked that it's kind of LVM to learn a lot about what LVM has been good and what LVM has done wrong. And it's a chance to fix that. [01:02:23] And also, there are challenges in the LVM ecosystem as well, where LVM is very good at the thing it was designed to do. But, you know, 20 years later, the world has changed and people try to solve higher level problems. And we need we need some new technology. [01:02:37] And what's the future of opensource in this context? Very soon. [01:02:42] So it is not yet open source, but it will be hopefully you will believe in the value of open source and because. Oh, yeah, absolutely. [01:02:49] And I think that the sense of the community at large fully believes in open source. [01:02:54] I mean, that there is a difference between Apple, where you were previously, and Google now in spirit and culture. And I would say the open source in terms of flow was a seminal moment in the history of software, because here's this large company releasing a very large code base, open sourcing. What are your thoughts on that? How happy or not were you to see that kind of degree of open sourcing between the two? [01:03:20] I prefer the Google approach, if that's what you're saying. The Apple approach makes sense given the historical context that Apple came from, but that's been 35 years ago and I think the apple is definitely adapting. And the way I look at it is that there's different kinds of concerns in the space. Right. [01:03:40] It is very rational for a business to to care about making money like that fundamentally is what the business is about. Right. But I think it's also incredibly realistic to say it's not your string library. That's the thing that's going to make you money. It's going to be the amazing UI product, differentiating features and other things like that that you built on top of your string library. And so keeping your string library proprietary and secret and things like that, maybe not the the important thing anymore. [01:04:11] Right. Well, before platforms where we're different. [01:04:14] Right. And even 15 years ago, things were a little bit different. But the world is changing. So Google strikes a very good balance, I think. And I think the potential for being open source really changed the entire machine learning field and it caused a revolution in its own right. [01:04:31] And so I think it's amazing for amazingly forward looking, because I could have imagined and I was at Google at the time, but I could imagine a different context in a different world where a company says machine learning is critical to what we're doing. We're not going to give it to other people. Right. And so that decision is a profound, profoundly brilliant insight that I think has really led to the world being better and better for Google as well and has all kinds of ripple effects. [01:04:59] I think it is really I mean, you can't understate Google deciding that how profound that is for software. It's awesome. Well, it has been. [01:05:08] And again, I can understand the the concern about if we release our machine learning software are our competitors could go faster. [01:05:17] But on the other hand, I think that open sourcing has been fantastic for Google. And I'm sure that obviously that that that decision was very non obvious at the time. But I think it's worked out very well. [01:05:28] So let's try this real quick. You were at Teszler for five months as the VP of Autopilot Software, you led the team during the transition from each hardware one hardware to have a couple questions or one. First of all, to me, that's one of the bravest engineering decisions undertaking sort of like undertaken undertaking really ever in the automotive industry. To me, software wise, starting from scratch. It's a really brave decision. So my one question is there is what was that like? [01:05:59] What was the challenge of that? [01:06:01] Do you mean the career decision of jumping from a comfortable, good job into the unknown or that combined with the at the individual level, you making that decision and then when you show up, you know, it's a really hard engineering process. [01:06:15] So you could just stay maybe slow down the hardware one or those kinds of decisions are just taking a full on. Let's do this from scratch. What was that like? [01:06:28] Well, so, I mean, I think Tesla has a culture of taking things slow and seeing how it goes. So, I mean, one of the things that attracted me about Tesla is it's very much a gung ho. Let's change the world, let's figure it out kind of a place. And so I have a huge amount of respect for that. Tesla has done very smart things with hardware, one in particular, and the hardware one design was originally designed to be very simple automation features in the car for like traffic workers, cruise control and things like that. [01:06:57] And the fact that they were able to effectively feature creep into landholding and a very useful driver assistance feature is pretty astounding, particularly given the details of the hardware hardware to build on that a lot of ways. [01:07:11] And the challenge there was that they were transitioning from a third party provided vision stack to an in-house built vision stack. And so for the first step, which I mostly helped with, was getting onto that new vision stack. And that was very challenging. And there it was, Time-critical for various reasons, and it was a big leap. But it was fortunate that it built on a lot of the knowledge and expertise and the team that had built hardware ones, driver's license features. [01:07:40] So you spoke in a collected and kind way about your time in Tesla, but it was ultimately not a good fit. Elon Musk, we've talked on this podcast several guess the course. Elon Musk continues to do some of the most bold and innovative engineering work in the world at times at the cost some of the members of the Tesla team. What did you learn about this working in this chaotic world that John? [01:08:03] Yeah, so I guess I would say that when I was at Tesla, I experienced and saw the highest degree of turnover I'd ever seen in a company, which was a bit of a shock. [01:08:15] But one of the things I learned when I came to respect is that he was able to attract amazing talent because he has a very clear vision of the future and he can get people to buy into it because they want that future to happen. [01:08:26] Right. And the power of vision is something that I have a tremendous amount of respect for. And I think that Elon is fairly singular in the world in terms of the things he's able to get people to believe in. And it's it's a very it's very there may be people that down the street corner and say, oh, we're going to go to Mars. [01:08:46] Right. [01:08:46] But then but then there are a few people that can get other others to buy into it and believe in build the path and make it happen. And so I respect that. [01:08:56] I don't respect all of his methods, but but I have a huge amount of respect for that. [01:09:02] You've mentioned a few places, including in this context, working hard. What does it mean to work hard? And when you look back at your life, what were some of the most brutal periods of having to really sort of put everything you have into something? [01:09:20] Yeah, good question. So working hard can be defined a lot of different ways, so a lot of hours. And so that's that is true. The thing to me that's the hardest is both being short term, focused on delivering and executing and making a thing happen while also thinking about the longer term. [01:09:40] And trying to balance that right, because if you are myopically focused on solving a task and getting that done and only think about that incremental next step, you will miss the next big hurdle you should jump over to. [01:09:52] Right. And so I've been really fortunate that. [01:09:56] I've been able to kind of oscillate between the two and historically at Apple, for example, that was made possible because I was able to work some really amazing people and build up teams and leadership structures and and allow them to grow in their careers and take on responsibility, thereby freeing up me to be a little bit crazy and thinking about the next thing. [01:10:20] And so it's a lot of that, but it's also about, you know, with experience, you make connections that other people don't necessarily make. And so I think that is that's a big part as well. But. The bedrock is just a lot of hours and. You know, that's OK with me. There's different theories on work life balance and my theory for myself, which I do not project under the team, but my theory for myself is that, you know, I want to love what I'm doing and work really hard and my purpose. [01:10:49] I feel like my goal is to change the world and make it a better place. And that's that's what I'm really motivated to do. [01:10:57] So last question, LVM logo is a dragon, yeah, you explain that this is because dragons have connotations of power, speed intelligence can also be sleek, elegant and modular. So you move the modular part. What is your favorite Dragon related character from fiction video or movies? So those are all very kind of ways of explaining that, you know, the real reason is the dragon. Yeah. So there is a seminal book on compiler design called The Dragon Book. [01:11:29] And so this is a really old now book on compilers. [01:11:33] And so the Dragon logo for LVM came about because at Apple we kept talking about LVM related technologies and there was no logo to put on a slide and sort of like, what do we do? [01:11:45] And somebody is like, well, what kind of logo should a compiler technology have? And I'm like, I don't know. I mean, the dragons are the dragon is the best thing that that we've got. [01:11:54] And, you know, Apple somehow magically came up with the logo and and it was a great thing and the whole community rallied around it. And then it got better as other graphic designers got involved. But that's that's originally where it came from. [01:12:06] The story is there dragons from fiction that you connect with for that Game of Thrones, Lord of the Rings, that kind of thing? [01:12:15] A lot of the rings is great. I also like roleplaying games and things like computer playing games. And some dragons often show up in there, but but really comes back to to to the book. [01:12:24] Oh, no, we need we need a thing. Yeah. And hilariously, one of the one of the the funny things about LVM is that my wife, who's amazing, runs the the LVM Foundation and she goes to Grace Hopper and is trying to get more women involved in the she's also a compiler engineer. [01:12:41] So she's trying to get other other women to get interested in compilers and things like this. And so she hands out the stickers and people like the Altium sticker because of Game of Thrones. And so sometimes culture has this helpful effect to get the next generation of engineers engaged with the cars. OK, awesome. Chris, thanks so much for your time. [01:13:01] It's been great talking with you.