JOSH: Hello, and welcome to Clearer Thinking with Spencer Greenberg, the podcast about ideas that matter. I'm Josh Castle, the producer of the podcast, and I'm so glad you've joined us today. In this episode, Spencer speaks with Danny Hernandez about the future landscape of AI, the growing capability of large language models, and the impact of AI on labor.

SPENCER: Danny, welcome.

DANNY: Hey, Spencer, excited to chat.

SPENCER: Yeah, me, too. The place I want to start with you in this conversation is the ways that artificial intelligence is going to shape the future, and why this might be an incredibly big deal, why this might be the biggest deal right now. Let's start the conversation talking about AI progress. I know you have some interesting views on this, about the predictability and exponential trends in AI.

DANNY: Yeah. I think there are some ways, some dimensions in which AI progress is straightforward and predictable, not exact capabilities, but there are a few underlying exponentials. There's Moore's law, which is an exponential most people are familiar with — 2x increases in efficiency every two years since the 60s — and that was the main thing driving the amount of computation available to AI experiments up until around 2012. And then people started pouring money into scaling up AI systems, and then the amount of computation — the amount of additions and subtractions and multiplications going into training an AI system — that was growing at 10x per year. And so this exponential in this space of the effective computation of AI systems, they're just these long-running clear exponentials that have been going on for a while. There's hardware efficiency, there's people spending more money which, maybe more recently, is growing at two to 4x per year, not 10x per year. And then there's algorithmic progress, which is a bit more difficult to measure. But I can talk about that one, too.

SPENCER: Got it. So essentially, the idea here is that you've got these three different exponential trends, all of which are driving towards improvement in AI. You've got the compute from Moore's law, you've got the companies paying more and more money as excitement builds around AI, and then you have these algorithms improving. So yeah, let's jump into the algorithmic improvements side.

DANNY: The thing that I measured there was that I showed that you could train something as good as AlexNet with two times less computation every 16 months if you just looked at a bunch of well-known image models that were open source, and then just train them, that it kept being cheaper and cheaper to get to a certain capability level. AlexNet was the thing in 2012 that got everybody excited about machine learning and deep learning. And there's been some more recent work that's shown that this is an underestimate in some way — that's a bit more complicated — but it generated an estimate that was more like 2x every (maybe) 12 months or nine months. I think this was an underestimate. That's in the image world and right now people are more interested in the language world. But an example of algorithmic progress in the language world is, it seems the transformer was something like a 50x increase in efficiency in a single year for a translation benchmark. And that's also in this algorithmic progress blog post that...

SPENCER: Wow, 50 times improvement in one year from algorithms, that's insane. What are some of the other examples of algorithmic progress that made a big difference?

DANNY: Well, I think a lot of the time, I would call the transformer a breakthrough, the most important thing that happened in language model architectures since 2015, or something like that. I give that as just an example of what breakthrough algorithmic progress could look like. I think other times, it looks more straightforward. One example for image generation was residual connections, which is sometimes just passing forward what the current layer sees directly to the next layer as an input, which is a simple idea.

SPENCER: Just to unpack that for a second, so the idea is that normally each layer in your network just feeds forward into the next layer. So each layer is only connected to the one behind it, but the idea is that you can actually skip layers — you can send data from earlier layers directly to later layers — and that this may have certain advantages.

DANNY: Yeah, that's right.

SPENCER: Got it. And then switching from other types of activation units to rectified linear units, was that another example where we got one of these algorithmic speedups?

DANNY: Maybe a better example — one I'm more familiar with or better able to speak to — would be Chinchilla, which showed that people weren't training models for long enough. There's this original scaling laws paper that Jared and Sam led, that gave an estimate on the correct length of time, the correct amount of data or compute to put into a model of a given size. And if you just train the model for longer, those models have something that's maybe a three or 4x boost in algorithmic efficiency. Or you could think of it as algorithmic progress. This really large hyperparameter was tuned incorrectly in this way for several years. That'd be another example of large amounts of algorithmic progress,

SPENCER: Just to make sure I understand that one, is the idea there that, if you have a certain amount of computation, or let's say you fix the amount of computation — because you only have so much money or whatever and you can choose how to spend that — do you throw more data at the model but then train for less time per amount of data or do you use less data and train for longer? And then it's the trade-off between the two and people were getting it wrong and then they figured out how to do it better? Am I describing that accurately?

DANNY: It's about right, the initial posing was right. You have a certain computation or dollar budget, but then the trade-off is, how big do you make that model versus how long do you train it for. How long you train it for and how much data you train it for, you could pose those as equivalent. Ideally, you don't repeat the data.

SPENCER: Got it. So it's like, do you use a billion parameters in your network or do you use 10 billion? If you use 10 billion, it's gonna take more computation per gradient step, so you're not going to be able to do as many gradient steps. It's like a bigger model that's trained less, essentially. Is that the trade- off we're talking about?

DANNY: Yeah, that's the trade off.

SPENCER: Got it. Okay, cool. So they basically figured out a more optimal way to do that trade-off, and that's another form of algorithmic progress. Okay, stepping back, we've got these three different exponentials, and when you combine exponentials, I assume you just get another exponential, but it's a faster exponential, so it's a really, really, really fast exponential. What does that look like if you think about the progress in AI of combining these three trends?

DANNY: If we look between 2012 and 2018, maybe we see around 40,000x growth from people spending more money, which they can do because machine learning is parallelizable effectively. You can spend more money to train this bigger model. Maybe 8x of the growth is Moore's law. You just get more out of each dollar, and an estimate I had for algorithmic efficiency was something like 44x for algorithms. And so you could combine those things, they would all multiply together, and now that 37,000x, that exponent would be slower. But when I try to get an intuition for it, I'll do something like think of the difference between GPT-2 and GPT-3, might be 300 to 600x in terms of effective compute, and how often do we get something like that? Or you could try to put it in years. The difference between GPT-3 and ChatGPT or Claude is two years and, if the exponents are constant, then we should expect the same jumps in some space. In this effective compute space, we expect about the same jumps in the same amount of time. You could try to have the intuition for like, if that's what two years of progress looks like, what would be the next jump that's equivalent, and expect that to happen every two years, as long as all the exponents are the same.

SPENCER: Are there signs any of these are slowing or do these all seem to still be going pretty fast?

DANNY: I think the main thing that slowed was the spending. It slowed from maybe 8x more spending per year to maybe two to 4x more spending on the largest models. It's a lot of private information so it's hard to estimate.

SPENCER: Got it. But there's not necessarily signs that Moore's law has been slowing in the way we care about, in this parallelizable way? And there's not really signs that algorithmic progress has been slowing as far as you're aware?

DANNY: Yeah, I don't think there's really signs of either of those slowing. Besides the spending one, I think it's a safe bet they'll keep going at their current rate for maybe ten or 20 years because they've been going for a long time. So I think it's the kind of thing where you can bet a big career jump on (or something) safely.

SPENCER: It's like one of those things where, the longer a phenomenon has been happening, the more confident you can be that it will continue happening for a while. If it's been happening for one year, well, you don't know how long it will last. But if it's happening for 100 years, it's probably gonna go at least one more year. That kind of idea?

DANNY: That kind of idea. And if we're trying to predict the spending one, it keeps getting more and more difficult because maybe currently large models, if you tried to find estimates of what they cost to train, maybe you'd find estimates on there, and it's something between 20 and 100 million dollars for the most expensive models. If that rate is going up at two to 4x per year, it quickly becomes billions of dollars and tens of billions of dollars. It's really hard for that exponential to keep going indefinitely and not become an S curve. But the more financial value people see in models, the more they're willing to keep going to the next steps. Right now, I think there's just enough commercial excitement that you wouldn't expect that exponent to die down, at least for a little bit, at least for a few years.

SPENCER: Right. It feels there might be some kind of spending cap. There's a certain amount of spending that a startup can do, and there's a certain amount of spending that a midsize company can do and so on. And there's a certain amount of spending that even a large company can't do, and eventually, you hit that size.

DANNY: Yeah, like a brand new fab, maybe for TSMC, might cost 20 to 40 billion dollars or something. This is an example of what really, really large capital expenditures look like. I'm just saying, if you're trying to project forward, the exponents, I think that one's the most difficult one and that you expect it to, at some point, taper off, whereas it's less obvious that the other ones have to.

SPENCER: Now, of course, increase exponentially in training of these models or ability to train these models, doesn't necessarily mean exponential improvement in the effectiveness of the models, which is a much harder thing to think about, how good is the model. You can think about accuracy on a particular benchmark but that doesn't always necessarily map perfectly into our qualitative assessment of how good the model is. I'm wondering, how do you think about the rate at which the models are getting better in the ways that we care about?

DANNY: It is pretty useful to look at the last step or something, the last step to ChatGPT or GPT-4 or Claude, which is Anthropic's model. I think that step is this zero-to-one in terms of, it's obviously economically useful enough for people to want to play with, to explore, to build companies on top of. I guess I agree; it's really hard to map between these spaces. But we've just seen this jump to this interesting part of the space where there's clear market excitement about using it.

SPENCER: Yeah, it wasn't clear to me that GPT-2 was good enough to do things that you really wanted to do. GPT-4 is clearly good enough for the things that you want to do [laughs], lots and lots of things, so somewhere in between that, there was a huge phase shift.

DANNY: Yeah. And I like the analogy of when voice recognition got to 95% accuracy, which was close to around Alexa, that Alexa was useful and broadly useful. And before that, there was Dragon and there were earlier versions of Siri that people found frustrating. And then you get to a certain capability level, which you were just making constant progress on on some graph. And then there was some threshold at which people were like, "This is good enough. I want to use it," and you hit that threshold. But now you do that with this AI system and then, if you're going to make another jump that's the size of GPT-2 to GPT-3 but the thing is already useful, or GPT-3 to ChatGPT but the thing is already useful, then I guess you should just expect that jump to be quite interesting. I think people will keep having the feeling that things are accelerating besides this zero-to-one effect of update, from not paying attention to paying attention. It's confusing to think about, but that's the way to build intuition and turn numbers like 1,000,000x more effective compute into intuition that people care about.

SPENCER: I can see at least two ways that more effective compute, even growing exponentially, wouldn't lead to these impressive model improvements that we care about. One would be if this curve that maps effective compute into the things we care about, was changing shapes. If that was flattening off: yes, you can go from GPT-2 to GPT-3, that's much better, GPT-3 to GPT-4, that's much better, but GPT-5 only feels incrementally better for some reason. There's that possibility. And then another possibility I could see is if it turned out, we just ran out of data to train these things. You've put the whole internet in it and then, well, what do you do after that? The internet is growing pretty fast, but maybe it's not growing fast enough with new information that you have more stuff to throw at it. So you could imagine running out of data. I'm wondering, on those two fronts, is there any evidence that there's a diminishing return in terms of throwing compute at it into turning it into stuff we care about?

DANNY: I think there isn't really evidence that we have this diminishing return in terms of things we care about. I do like to make the biological analogies even though they're a bit strange. If we look at the exponential for mammalian intelligence, it jumps from mouse to cat to human, and at some point, it just becomes vastly more interesting. And I feel we saw a similar trend here where it's boring, boring, boring, and then all of a sudden, this is quite interesting. My guess is that, once you've hit this interesting threshold, it should keep just getting more and more interesting. We'd be very interested in a species that was more intelligent than humans. And I think we'll be very interested in things that are more intelligent than Claude.

SPENCER: What about on the data side? Could there be a world where we run out of new data to train these things?

DANNY: I think that's a world people talk about. I think it is a plausible bottleneck, but maybe not the bottleneck that feels most likely to me. I think it's more likely that maybe Moore's law will peter out at some point, or we won't be able to invest. I think it takes exponential investment from scientists and engineers to keep the algorithmic progress thing going, and a bunch of people are moving careers to work on making large language models more efficient and more capable. But that trend will also end at some point, because they'll just run out of new, super talented people that can make that transition. I guess I put the bets there. I also think you could describe it as, do we have environments that the models can keep learning from rather than data? I think you do run out of internet data; it's just not the thing that I think is the most likely reason progress stalls.

SPENCER: Yeah, maybe you could help me understand that a bit. As I understand it, already, they're trying to train these models on a large swath of the internet, maybe trying to avoid really spammy websites and garbage kind of input. But I imagine there's not that much left of the internet that's high quality that they're not training on, or am I wrong about that?

DANNY: I think people are already leveraging close to the whole internet that's useful. I feel it's hard to talk about. There's this way in which there are aspects of projecting parts of this forward that can be a bit difficult to talk about. If there is a data bottleneck, then trying to solve it is just worth a lot in terms of capabilities, so it's hard to speculate about it.

[promo]

SPENCER: Okay, so you talk about these long-term exponential trends and that suggests that, well, things were maybe predictable in a certain way. And yet, we're seeing people really shocked by the behavior of these models. So I'm curious, how surprised were you about the models coming out? Was it what you expected, or were you shocked like many other people were?

DANNY: The models that have come out since GPT-2 were not that surprising in this view. I think they were close to 'as expected.' Maybe we have to start at GPT-3 for them to start to not be surprising in this view, given scaling laws, where scaling laws show you got predictable gains and performance based on scaling up computation and data. People that didn't have this exponential view around AI progress have been mostly disoriented and surprised. Sometimes it was a bit more impressive than the median expectation but it was never a shock to me. And I think this is common for exponentials in general. I think for exponentials in general — like COVID is an exponential that just a lot of people are like, "Oh my God, this is a thing we all have to pay attention to!" — and a lot of other people were just not paying attention until the exponential hit and did something. Actually, it had grown already to the point where it obviously mattered, rather than you could just project it forward and see that it would obviously be really important.

SPENCER: Yeah, to me, it seemed that most people were really shocked, which suggests that they weren't carefully tracking this exponential view that you have. Do you see, among your colleagues who are more AI savvy, that a lot of them were appropriately projecting things out?

DANNY: I think a lot of people either weren't surprised, or at least this was a viewpoint that they had been considering, if they were familiar with this viewpoint and had meaningfully considered it. So it might not have been their bet but it wasn't a shock.

SPENCER: How does it look if we look at specific benchmarks? You can take GPT-2, GPT-3, GPT-4, and throw it at lots of different tasks, hundreds of different tasks, and you can look at how it's improving as you go up different models. Does that look like linear improvement as we throw exponentially more effective compute at it? What are the kinds of curves there?

DANNY: Let's see. A few things that happen there are generally, you need a new benchmark every one to two years because you just solve it. That's the thing that may have shocked the ML community overall. You make a benchmark, and you'd make something that you thought was just really difficult, that maybe you wouldn't be able to do for five or ten years. I think Winograd schemas, which are disentangling a sentence where it's ambiguous what the object is. So say I was like, "My drink is on the table. Can you pass me it?" I mean my drink; I don't mean the table. I'm not asking you to pass me the table. Early AI systems would be confused by that; in 2016 or 2017, you'd maybe get 70% or 80% on that benchmark. And then I think GPT-2 or GPT-3 effectively solved that. And then you'd generate a new thing that to you seemed really difficult, and then the thing would just get solved within a year or two by a later generation. So you can try to extrapolate these benchmarks and what the progress will be, and they're usually quite a bit noisier than trying to think about what will happen with capabilities in general. Sometimes new surprising capabilities emerge pretty quickly, and so I think trying to extrapolate what will happen on any specific capability is quite a bit harder than trying to think about just how much more impressive will... It's a different kind of problem than trying to think about, will impressiveness go up? Overall, it's a bit noisier, I think.

SPENCER: Got it. What about with really hard benchmarks? Let's say a benchmark like translating languages where it can be extremely, extremely difficult. As we get our models better and better — I don't even know if you can talk about it this way — but do they tend to make linear progress on these for exponential compute, or is it hard to even say?

DANNY: It can be hard to say. A lot of times, you're looking at progress in accuracy. With accuracy, you have to stop making linear progress at some point because it can't go above 100%. One way I would just describe it is if you graph those things — things like loss or a big average of benchmarks — then it's beautiful and straight. And when you look at these individual things, it's noisy and those curves, you're much less able to extrapolate. And maybe I'll say one more thing that I think is useful about thinking about whether or not the progress is useful or not useful. One thing that you do get as the models get more capable is, they generalize better. The main finding of GPT-3 was showing that One, Two or Three, besides just that scaling up, the system got you capabilities that you really cared about and thought were worthwhile, that if you gave one to five examples that this meaningfully improved GPT-3's performance, and that this improvement in generalization with examples grew with model size. And then I also had the scaling laws paper where I showed that there was this predictable exponent that you could think about using to show that generalization predictably improved as you scaled up the model and learn on one distribution and evaluate it on a different distribution to see how transfer scaled up.

SPENCER: This is related to the idea of few-shot learning?

DANNY: Yeah, this is related to few-shot learning. Models with more effective compute have better few- shot learning, which is a capability that everybody cares about. If you think about a model as a person that you're interacting with, and you have to explain it to them, and you have to give the person ten examples instead of one or three, it's very frustrating, the more examples you have to give a person to have them understand what you're talking about.

SPENCER: With older models, we used to have to train them for whatever we wanted them to do. You want them to do some really specific task, well, you trained on that specific task. And then with these very general powerful large language models, we can give it just a few examples and it somehow figures out how to do the thing a lot of times, which is kind of a new paradigm. You just give it a few examples and it somehow figures it out, which suggests that somewhere deep in the neural net, it has a latent ability to do the thing, and you just have to point it to the right part of the neural net somehow. Is that how you think about it?

DANNY: Yeah, I think about it that way.

SPENCER: Cool. Okay, so we've talked about these powerful exponential trends and how they seem to actually be converting to better and better performance in things we care about in AI. Let's talk about the implications of this. I think you're gonna suggest that AI might be the biggest thing going on now in the world [laughs]. So let's get into that. Where should we start?

DANNY: Yeah, maybe we'll start with more of what I might call a median outcome, more like a safe argument or something. One meme that I've heard is that AI is the biggest thing since the internet. I've heard some entrepreneurs say this. A month or two ago, I'd heard people say the iPhone, they think of the iPhone as smaller. And yeah, I think the internet was the most technologically interesting thing that happened in the last 20 or 30 years. It had this exponential behind it. Moore's law, maybe you could have thought of it as actually predictable in the 80s, if the exponent kept going, which you had enough information to make that a reasonable bet at the time. People just showing up and just looking at this fresh, that are in this entrepreneurial mindset, often say something like this. I think this is still underselling AI here. I think that to be a bit more strange or something, I think this exponential view is the main basis for an argument of putting at least a 10% chance that we'll develop general systems that could do at least my job and everybody at my company's job, as good as or better than us. That's one definition I would use for human level general intelligence.

SPENCER: And this is 10% over how long?

DANNY: Over the next ten years or something.

SPENCER: So you think basically as good as you at all the different things you can do?

DANNY: Yeah, something like that. Or every person at my company elite. Seems like you could have the whole company or something. And that's a view that Anthropic defends, puts out in a 'core views on AI safety' post. And I think it's hard to argue against it. I think most people thought of that as a crazy or weird view five years ago or three years ago. And now people would put higher probabilities than that. I framed it as 'at least justifies just a huge amount of attention from a lot of people.' I like Holden's most important century framing, that you're making this argument that this is the most important thing going on. And in hindsight, this will be the only thing that we cared about how well it went in 50 years or 100 years.

SPENCER: One of the things that's really struck me about this is how people will try these AI models, they'll be really impressed, often a little freaked out. They'll find something that the models can't do, and they'll be like, "Ah, well, it can't do that. It's not actually that smart." And they don't seem to be projecting forward two years or five years. They're somehow stuck on the exact version that they're looking at. And I wonder if this is maybe more of an allergy to other technologies. Maybe with normal technologies, we're not necessarily used to them being so radically different where they can't do something at all, and then you do the exact same technique and then suddenly, in two years, they're able to do it really well, or something like this. I'm not sure. Have you observed this as well with people?

DANNY: I have definitely observed this. I think the online discourse here is really noisy and hard to get much out of. I don't look at it a lot. The thing that I think people are often arguing about is whether or not current capabilities are... There's overclaims about how good models are, and other people are saying that other people are not giving models enough credit for what they're good at now. But they're often talking past each other, where I mostly want to extrapolate progress. I think the uncertainty and extrapolating progress between people, for even a year or two years from now, is often 100 times or ten or 100 times the difference in what they're arguing about, which is what the current models are capable of, and whether or not people are giving them too much or too little credit. I also think there's this other problem with the discourse, which is that science is a bunch of norms for slowly converging to truth. Science isn't really built around forecasting, or trying to predict what happens two to five years from now. Even though scientists and engineers have the best information, they're not used to being in that mindset or discussion and the internet doesn't really reward them for using norms that would be productive in that sense.

SPENCER: Reminds me of this quote (I think I remember it), which was something like, "You should trust an old scientist when they tell you what's possible, but not when they tell you what's not possible." That's something about, a lot of times, people come with some new thing that blows everything out of the water that was unexpected in science.

DANNY: Yeah, I think that's pretty common. It's not exactly the same thing, but I also like stories of old scientists who said something was impossible, and that just motivated the scientist an incredible amount to show them that they were wrong. I'm thinking of Szilard and the atomic bomb. I think he heard a talk from Rutherford where Rutherford claimed fission was impossible and then he went and worked on it.

SPENCER: That's a very unfortunate motivation to make a weapon that could destroy the world, but...

DANNY: I don't know if his motivation was even making a bomb then, but yeah.

SPENCER: All right, so we've got this framing that these exponential trends are going to continue. If they do, it's likely to lead to these really huge qualitative improvements and these algorithms, and even plausibly, making them so good that they can replace someone like yourself. And if they could replace someone like yourself, they could probably replace a lot of people on the entire planet, doing whatever people are doing. So let's just start talking about what are some implications of that if, let's say, these models really just do keep getting better at the rate that you expect, based on extrapolating trend lines.

DANNY: It depends. We could go between short or medium term implications or something.

SPENCER: Just start with short, yeah.

DANNY: With short, I think there's some kind of job disruption and moving who has power, some change in the balance of power. I think you could look at self-driving cars and AI that does radiology, and those feel like jobs that you shouldn't recommend people start training for, given the horizon upon which they look replaceable. And I think there will just be more and more jobs like that, and I think some of those jobs might look really high-skill. It could be that some of those jobs look like certain kinds of lawyers or certain kinds of doctors. I think that's disruptive and will lead to people having to reskill, and there'll be winners and losers, and there'll be companies that benefit a lot, countries that are better at it. I think there's something that's inherently increasing inequality about it, where you could think these AI systems are able to turn capital into labor, and that puts capital in an overall better position and reduces laborers' bargaining power.

SPENCER: Right, so if a company can basically replace people with software, then the company gets the benefit, and they're not paying wages to the people anymore. That clearly can create a huge concentration of power among companies, perhaps even at a level that's unprecedented eventually. You could imagine companies that have automated a billion laborers in some sense.

DANNY: And you could think of there being some new class of managers that are good at managing AI systems, which might be the people building them, it could be prompt engineering as a way of managing AI systems. But some people will ride this wave and have their own skills and ability to create value amplified a lot, and other people will be faced with replacement and have to reskill. I start with those as things people are broadly scared about. That's a thing that politicians ask about a lot, that I think, if you take this trend seriously, you expect to run into eventually. I don't think we've really seen it yet in some meaningful way. But I think it's a thing to think about. There's also this very broad way this shift of power, where people that can make AI systems and make new AI systems and have AI systems do useful things, can have just a lot of influence on the world that can be positive, it can be neutral, could be negative, it could be positive in the normal capitalistic way. And so I think we should just be looking for ways in which AI systems will... Maybe it could be like the internet remade how a lot of the world works. We should expect AI systems to do something similar, and we should look at how that's being done, and whether or not we expect that it's mostly going well, or going kind of medium.

SPENCER: One thing I wonder about: are there certain types of jobs that are more amenable to replacement from the newer class of models? You mentioned doctors and lawyers. Is there something about a certain kind of doctor, a certain kind of lawyer that you'd expect to be more likely to be automated first?

DANNY: I think the more economic value there is on the task, and the more well-defined it is, and the more there's data — well-defined examples of the task being done well, in a form that a model can ingest — the better off you are. On the opposite end, I would go something like nurse, where it involves a bunch of physically moving things around and emotional interaction; that seems very far from automatable. But there is something about a doctor or a lawyer where one of the things that they do is memorize a lot of things, they'll have exams that involve a lot of recall. AI systems are just really good at remembering lots and lots of things and, if what you need is to find the best practice in this very specific medical situation, that seems something very amenable to models. Models are better at that than they are at, say, solving math problems that a similarly smart person would solve if that were novel and created. The AI systems are currently worse at reasoning, or worse at reasoning than they are at remembering a lot of things.

SPENCER: So we can take people who view radiological X-rays as a nice clean example here because, as I understand it, they look at one of these x-rays, they do an analysis of it, write up notes. And then there's existing somewhere, probably millions of examples of this done by real humans that they can, in theory, be trained on. So it's a well-defined task. It's very concrete. We've got lots of good training data. I imagine that would be one of the first things that would be likely to be automated? Do you agree with that?

DANNY: Yeah, I think that's automated pretty well already. There might be some kind of regulatory thing that keeps the radiologists at work in their jobs because maybe they have to sign off on it. But maybe they are now five or ten times more efficient. Maybe that job stops existing in other places with different regulatory regimes. But yeah, I think the radiology example is pretty good. But that's also just an example of something that you normally think this very highly skilled, very smart person has to do that job. Those jobs aren't, in some way, safer. It depends. You have to think through what AI is good at and how strong of an incentive is there to build an AI system that can do that well.

SPENCER: Well, one thing that's confusing to me about those examples, people have been talking about radiology in particular being automated for a long time. And yet, as far as I can tell, it hasn't happened, or maybe it's begun to happen. I'm not sure. But it makes me wonder whether there are hidden barriers to these things, because the tech to at least automate parts of radiology seem a little bit old.

DANNY: Yeah, I think there's some kind of regulatory barrier in the US where there's liability and other things like that. You already have the people there, and you don't lay them off or something. There's some kind of inertia in the system. But I think, if you're rebuilding the system now, you would have a lot fewer radiologists,

SPENCER: I imagine if you broke down a job like radiology, there's probably a bunch of components to it. We're focusing on one aspect of it, but I suspect, in practice, what would happen with these things is not that the entire job would be replaced with a big piece of software, but more like some chunk of the job — there might be a core aspect of that job — that is now just an algorithm running. And then okay, maybe you still have a human that's doing the other little bits and pieces around that. Does that seem right to you?

DANNY: Yeah, I think at first, most of the time, it looks like amplification. And then there's this question of what the elasticity for demand was in that job market, that we don't actually want ten times more radiologists. And so if you start amplifying them, at some point, you need fewer. But that's not true for all jobs, I don't think.

SPENCER: Yeah. Could you explain how that works? Because I think there's an interesting counterintuitive thing here. Let's say you take people who are doing something for money, and you make them twice as good at doing that thing because now they have a piece of software that helps them. The software can't do it alone and needs the person amplified. So you could imagine, in some cases, that now, many fewer of those people get hired because they're twice as good at the thing and so you only need half as many of them. But you could also in some cases argue, well, maybe hiring that person is really, really good now, because you get so much out of hiring them that you actually want to hire more of them than before. What determines which of the two paths it ends up going down?

DANNY: I think maybe I would just generate an example where I think it goes the other way, something like entrepreneurs. If the US could just have ten times more entrepreneurs and had to pay them the same amount that entrepreneurs currently get paid, I think they would quickly do that. I think that there is some kind of supply and demand curve for that skill, and that we're at this place where there's insufficient supply. There's still a lot of demand at the current way the market would pay those people and so that's different. Whereas, if you imagine us literally having 100 times as many radiologists, then a lot of them are not doing anything. People aren't going and getting scans of their whole bodies 100 times more often in that world because you mostly just want those when you have issues.

SPENCER: One example I think about: imagine someone works at a company and let's imagine we give them a software assistant that amplifies their behavior. You can imagine that, in some roles, that might mean that person makes more money for the company, like maybe in a sales role. Maybe that person is now bringing in 20% more money than they brought in before. And then you could imagine that that might cause the company to wanna hire more salespeople. "Oh, wow, our sales is really effective. Let's scale up our sales." On the other hand, you can imagine this other type of role where the amplification means the person gets the work done twice as fast or something. But you don't actually want any more of the people because you have a finite amount of that work to do because it's in a chain of other work, and doing more of it than needs to get done doesn't actually help you at all; you just want to get all of it done. And there you imagine the company might fire a bunch of people, because they're like, "Oh, well, we can get the thing done with half as many people." This just gives some ways of thinking about which situations you might want more of the people or fewer.

DANNY: Yeah, I agree with that. I started off with these shorter-term things, in part because they just are reasonable or something. I think what will happen is that this exponential, if it doesn't stop, will make things that are smarter than ourselves. Hopefully, they'll be aligned with our values, and they'll start just progressing technology for us, including themselves, and they'll just make the world unrecognizably different. You could imagine as different as things are after the Industrial Revolution, say in the 1900s versus 1400 or something like that, just a jump that's as big as that in how different the world is, but have that happen in five or ten years. And I think that is the kind of thing that AI systems that are as capable as us would lead to or AI systems that are doing AI research themselves. And so I think it's mostly this argument that it's the highest expected value thing to do, if you can effectively work on making that go better. I think that's what the main implication is once people start to take that view seriously.

[promo]

SPENCER: I know that you spend your time with the goal of trying to make these transitions into AI go better, build these systems in a safer way. Tell us a little bit about your own story of how you got into working on AI?

DANNY: Cool, yeah, let's see. It was around 2016 and I had been doing some forecasting consulting with Open Philanthropy. Actually, I think the first time we traded emails was around that consulting. And I talked to Holden, who's the CEO of Open Philanthropy, and he pitched me on, if I was excited about forecasting, that he thought the thing that somebody could do that was the most good in forecasting was to try and forecast AI progress. And I remember thinking of that as very interesting, maybe the most interesting thing I could think about. It's something I was suited to. And then also this person that I respected the most on the topic of how to effectively do good, was what told me that this is the way that he thought I could do the most good. And I think that was when I first started seriously considering it. I think I told them I would work two days a week on it and he said I should work three, and he convinced me of that. And I started doing that. A few months after that was when I did AI and compute work, which is the best known work I've been a part of. That was me starting to work on it and I think what was special about it was, I quickly felt a lot of meaning. When I worked at Twitch, I think I was mostly excited about making something big in the world. It was less clear to me how good it was. I didn't think it was negative but I was mostly excited about making something big. I remember when I first found a bunch of the feeling of meaning was, I was at an EA global conference, and Holden was giving a fireside chat. And I think what was moving was, he's up on stage, he's in this powerful high-status position, and he's talking to everybody, He's repitching this thing he pitched to me and I felt if he just convinced me that that was what mattered — that I could do this job, I believed I could do this job effectively — I think it's a novel or meaningful feeling to feel the person on stage is just talking to you. And I guess this felt like being called to work on AI because the person that I deferred to most on how I could do good, was talking to me in this way that was extremely emotionally salient. I guess I never really expect the President or something to tell me that he really needs me to do something. This felt the closest that I was going to get, just a hero talking to you in this way. I think that that is the closest to this movie scene call to work on something that I'll get. And I think that that's often how it looks and a lot of people might just miss it. I was really moved by a credit scene at the end of the Chernobyl series, where they explain that this character, the scientist character who I resonated with, who obviously took a bunch of meaningful risks to their safety by telling the KGB and other people that people have messed up and it's just really hard to acknowledge failure in that system, and trying to push forward scientific discourse in 70s or 60s Russia, and they had this picture of this whole bus of scientists. "This character represented all of these scientists, but for this story, we made them a single scientist." And so in Chernobyl, it felt that that scientist saw this crazy, large, important opportunity to basically save some meaningful portion of Russia. But actually, it was really just a large group that probably all saw something a bit weaker, and they all stepped up. That's the kind of thing where there first started to be meaning for me in working in AI, working on trying to make it go better. And I would argue that people work on it from that point of view. There's just this self-interested point of view to work on something that you find deeply meaningful, and I think lots of people could find a lot of deep meaning in this, in taking that goal really seriously. And any things that they might think of as sacrifices, if they have to work on things they're slightly less interested in, or take maybe a modest pay cut, when they're already generally making a lot of money and have had a lot of financial success, that I think that the meaning is just worth ten or 100 times any kinds of sacrifices that I have had to make, for me personally.

SPENCER: So it sounds your pitch is essentially: one, you think this is going to be a huge deal even if you just extrapolate current trends, even without any kind of really complicated reasoning. And two, it seems like you think it can be deeply meaningful work. It's not like you have to be like, "I'm gonna do something really boring or uninteresting." There's something really compelling about it as a subject. And also maybe it pays well, too, so it has a lot going for it. Is there anything you would add to that?

DANNY: Yeah, I guess just to reiterate, I just feel I haven't really made sacrifices, it's just all good. I think a lot of the times when people pitch working on this because of the chance of AGI and the chance that AGI ends the world or humans go extinct or something like that, they're coming from this fear or obligation framing, where it's like, "Hey, you've learned about this thing, where there's this big problem that you could work on. Now you have this obligation to work on it," and people throw up walls and think really poorly, and try to just do a bunch of mental gymnastics and not have to feel like they have some kind of new obligation. I never really felt obligation and would frame it in this other way.

SPENCER: It's just basically like, if you're telling someone they have to do something, they're obligated to do it, they're gonna be really resistant to that. Whereas you're saying, no, there's just a lot of awesome things about this as your focus area. You might be able to have a huge impact on the world while also having these other benefits, too.

DANNY: Yeah. And I like to try to put it into this story form. I think that the other stories are just much bigger. And I think it's interesting because it used to be that, if you talk to people about, "Yeah, I think there's some chance of human level general intelligence and this is my motivation," that you would get laughed at or you wouldn't be taken seriously, say, three years ago or five years ago by researchers. But now if you ask people why people in AI are not working on trying to make AI go well, why they're not motivated by it, I think they're much more confused in some way because a lot of them do put meaningful weight on that. The majority of them still do not seem like they're primarily working towards that goal. And I think they're just confused about the trade-off between meaning versus the trade-off for themself. They're like, "Why shouldn't I get to pursue my own capitalistic interests like everybody else?" And I think I would just argue that pursuing the thing going well and getting meaning from that is more in their interest, and they're just failing to forecast what thing they'd actually value.

SPENCER: So you're saying if people lived more in line with their values, they probably would be more likely to work on it.

DANNY: Yeah, I think actually, people often don't really think about the question as to how much they live in line with their own values. I think if they did, they often would. But I think that question is deeply uncomfortable. You can imagine asking people at a party or asking your good friends, "To what degree do you think you live up to your values?" It's an incredibly personal uncomfortable question. So I think most people just don't think about it. But I think one way that I would frame that here, if we put it in that way is, if the President shows up, and is like, "Hey, we need five people — and you're one of them — to stop this asteroid," then everybody's like, of course, they would, they would go. Only the total old, broken miser in the story doesn't do that. But it doesn't really look like that. It looks more like you need 500 people, and there's 10,000 people that could do it, but each other one helps. And you just happen to be one of the 10,000 people that would be most useful and, if you are living in line with your values, that's still your call towards meaning or towards heroism or whatever you want to call it. And I think almost everybody who thinks they're living in line with their values, thinks that they would respond to a call to heroism, and I think they just don't recognize what actual calls to that look like.

SPENCER: Are you saying that we needed a new call to adventure cause area? Do you take the red pill or the blue pill? Mysteriously appear out of nowhere? [Danny laughs.]

DANNY: I don't know if it's a cause area. I'm just trying a different framing that I hope resonates with people, partially because I don't like to talk about AI progress without pitching to work on it going well differentially. I try to differentially accelerate safety versus capabilities overall. I could have spent a bunch more time just getting people as excited about AI as possible, but didn't do that.

SPENCER: Yeah, let's talk about that. Because you suggested that people should consider focusing on AI, but I imagine you care a great deal about what that means, to focus on AI. Typically, focusing on AI means doing cool stuff with it, applying it to other things, or trying to make the models more capable.

DANNY: Yeah. Some kinds of things you could do, some aspects of this, there's this broad research area you could call AI safety, and we could split that up some. One aspect of it is to make it so that the models have values that we like. You'd call that the alignment problem. You could think of that as, is this model a good egg? Does it have good character? That's a well-defined technical problem you could work on. There's another thing you could call alignment science where you're more interested in measuring to what degree you've succeeded in that problem, or generally in other benchmarks or things that make you more convinced that this model is, in fact, a safe model in some way that you cared about. You could split off into mechanistic interpretability which is, do we think we understand exactly why. We looked at this model under a microscope, and we understand what this mechanism is and why this thing we care about happened, at lots of different levels of abstractions, so we're not worried that we're just going to be totally surprised. And so those are all kinds of things that somebody could do. You could also work on AI policy. I'm not going to split that up, but it's one thing that happens. Let's see, you could work on security for labs. There's also, I think, one thing about this that people don't understand, is that any sufficiently competent software engineer or security person (as examples), could just already be useful. They don't need to have any AI expertise. If you're just the engineer who can be hired almost anywhere that you want to work already, then you can also just work on these AI problems, because there's lots of large distributed systems in the works.

SPENCER: By security, you mean a security researcher who studies vulnerabilities in systems and things like that?

DANNY: Anything in security. I don't actually know what a good hierarchy of what people in security is, but it could be a security researcher. You could combine ML and security. You could be the kind of person who just helps an organization have really good security in any way. I think security researcher would be too narrow.

SPENCER: You mentioned AI policy, which is something that less technical people could go into. Are there other areas? For someone who's really interested in trying to make AI go well in the world, but is less technically inclined, what other kinds of things would you point them to?

DANNY: I would point them towards maybe policy, but often, if they already had some policy background, I would also point them towards just ops or management. A lot of the times, I'll make my pitch more targeted towards more senior people that obviously could be really high impact, if they wanted to, and I leave it to 80K (80,000 Hours) and other people to give advice to people that would have a harder time making this transition where people who might expect better arguments, because it's more costly for them.I think now is a reasonable time where I feel like I have lots of friends who have had large exits or been the CEOs of companies, and I would still try to convince them to just find something useful to do either in management that people might think is a demotion, or to become ICs, just because they're extremely capable people. And so I think I have less guidance for what people should do, and more just want to make sure people understand that these are large organizations with lots of roles now. And if they're the person that thinks they are just sufficiently capable to be useful, they should at least go check if they can, and I think many of them could be.

SPENCER: Before we wrap up, let's just do a quick rapid-fire round. I'll ask you some, not easy questions, but quick questions, and get your quick takes on them.

DANNY: Sure. Sounds great.

SPENCER: I understand that you have a take on how we should take more ownership of friendship. So what does that mean, 'taking more ownership of friendship,' and how does one do that?

DANNY: Yeah, so if two people meet at a party, romantically, generally the man takes ownership of pursuing the other. There's kind of a clear person who you expect to drive that forward. I think it's similar in business where, if I'm recruiting someone to work at Anthropic, then it's on me to send them the next email, send them the next text, figure out what the next step is. With friendship, people just expect it to be natural and easy, and often, I think neither of them takes ownership. And I think that's why my understanding is, there's been some research that it takes several unplanned interactions for people to become friends. And I think if you take ownership, that it can just be the same as other situations, like a business or a romantic situation where you have one interaction and you realize, "Oh, there's a lot of potential friendship here," and you don't need another unplanned interaction in order to make the friendship happen.

SPENCER: So what concretely should someone do if they want to take more ownership?

DANNY: When they meet someone that they feel they are interested in becoming closer to, they should get that person's contact information, they should figure out the next thing that they should do, they should invite that person to do something (an event or one-on-one), they should feel a bit nervous, they should feel they are getting some risk of rejection, the same as they would in other situations where they're the person pushing the relationship forward.

SPENCER: Alright, so I know, as an AI person, you probably tend to think in AI terms, but you're also someone who's invested in learning about meditation. I'm wondering, what did you learn about meditation by thinking in an AI perspective?

DANNY: I think that AI is helpful in meditation because it gives you helpful mental models. You have a bunch of simple models as to what might be happening. I think it's more helpful than neuroscience probably, even though I think there's more collaboration between neuroscientists and meditation people currently. One example of this is if you go on a meditation retreat, I think hour per hour of meditation, you just get way more than you would if you did that 20 minutes a day. I think that comes from this signal-to-noise ratio, where you're at this retreat, and after two days, you've gotten adjusted and you're no longer thinking about who you should text and what you should eat for dinner, and about video games or who you should pursue romantically. All these things that are just more large salient sources of reward are not around and the only reward that's around is moving towards joy or peace or healing. And I think those rewards are much smaller and quieter in a moment-to-moment way, at least at the start, and you have to remove all of this noise before those rewards even show up. And another thing is, the most positive meditative state that I have felt, it felt what happened was, I turned my preferences or my goal orientation down to close to zero, and this made things feel like they were perfect. And it's easier to do that than to actually construct a perfect scenario. You can see how that would make sense from a math point of view, that there's some level at which you turn your preferences down, and then that's equivalent to making the things as good as possible in some way. I think this makes more sense as a model of why that would be this extremely desirable state by definition. This feeling that things are perfect, sounds really good. It's a thing that lots of people would say, "Yeah, I do want to feel like everyday moments are perfect." Those are some examples of connections I think I made between meditation and AI.

SPENCER: Our final question for you: what is your approach to making difficult decisions, and how does it differ from how a lot of people think about it?

DANNY: I think a lot of what I try to do is to find one to three considerations that I can make the decision while I'm only holding those in my head. I think the starting insight is like the importance of concerns isn't normally distributed. I remember when I heard people talk about making bullet points of pros and cons, and there'd be five or seven on each one — I guess I just remember that visual from when I was younger — and I think that's a bad way to make decisions. I think that'll lead you to hold a bunch in your head that's not the most relevant stuff. And so what I'll try and do is, I'll try and make it so that the decision's clear in terms of one or two things, one or two top considerations. And it might be that, in order to do that, I have to roll up or redistribute those considerations. So an example might be like, what job should I take? And somebody might bring up, "Oh, I find this job more interesting, but at this other job, I like the people better. And this job is closer to my house." They might bring up a bunch of considerations like this. And I think what I might ask about is just, okay, at which of these jobs do you think your day-to-day wellbeing will be better? All of those concerns had to do with your day-to-day wellbeing. So if your day-to-day wellbeing at this one job was higher and you thought it had, say, more impact in the thing you cared about, if you were motivated by impact, and you got paid more, then you might be, "This job just dominates the other job." Whereas if we were trying to hold all those other things together, it's just less clear that that was the case. And it might be hard to think about this other abstraction as to where my day-to-day experience will be? But I try to see if I can find something like that where, if I just explained it to somebody else, they're just like, "Well, if that's what you think, obviously you're going to do that." And I just don't have to hold very much in my head.

SPENCER: Danny, thanks so much for coming on. This was a really interesting conversation.

DANNY: Yeah, thanks so much, Spencer, super interesting.

[outro]

JOSH: A listener asks, "What's something that people take as a given, preferably something about what makes a good life, that you wish everyone would instead take the time to examine and investigate for themselves and come to their own conclusions about?"

SPENCER: I think that many people make assumptions about what will make them happy that they haven't scrutinized very carefully. And I think a way to scrutinize that more carefully is to really think about the things that do that make you happy, and not just make you happy momentarily, but make you happy in a more sustained way. Thinking about what are the times in my life I was happiest? What are the situations in my life when I was happiest? And so really reflecting on what brings you personal happiness and trying to avoid the narratives about what makes someone happy or what a good life looks like according to society, which I think is often not the best guide for our own happiness.