SPENCER: Kristian, welcome.

KRISTIAN: Thanks a lot for having me on the podcast, Spencer.

SPENCER: What inspired you to work in climate change? And then, why did you decide to leave working in climate change to work on AI?

KRISTIAN: I think what inspired me originally was the idea of utilitarianism and caring about all beings, including beings in the future. In order to actually reduce global risks, we need to coordinate globally. A lot of issues, like climate, AI, and nuclear, are all global issues, and we need to learn how to govern globally. Climate change was the one domain that I first felt passionate about, but where there was actually momentum around global governance more than 10 years ago. I felt like I wanted to crack the formula of global governance. That's how I went into climate to begin with.

SPENCER: Did you feel at the time it was sort of the highest impact thing you could do, or was it kind of a balancing act between impact, passion, and learning?

KRISTIAN: That's a great question. I never felt like climate, in and of itself, was the most impactful cause area. But I did feel like global coordination and global governance was the most impactful cause area. However, that has sort of never been a standalone cause area, and climate was the domain where there was most momentum around governing it globally.

SPENCER: Yeah, I can see why climate would be a really good use case, because it's so important for large countries to be able to coordinate in order to combat climate change. If the US were to do it, but they couldn't get China and India on board, or if Europe were going to do it, but they couldn't get the US on board, it's just not going to be very effective.

KRISTIAN: Exactly. And it's this massively hairy problem, because fossil fuel is everywhere, and it's so entangled in our world economy. I felt like it was an area where longtermism was sort of almost the default; there are not very many areas within policy where people are actually passionate about the far future.

SPENCER: Although many climate change activists talk about climate change as sort of an incoming disaster that is going to create catastrophes in our own lifetimes, and it's not just about the future.

KRISTIAN: Yeah, that's true, but it's still the future from a policy perspective. I think politicians are sort of incentivized to win the next election and have a good election cycle, which might be four years long, and climate change is still several decades away. So it's more long term than most policies, on average.

SPENCER: And before we get into your transition into AI, do you feel that climate as a field has made progress on governance and how to get governance cooperation and achieve real goals there?

KRISTIAN: Absolutely. I think it has made a lot of progress since I started working on climate. For instance, having the Paris Agreement many years ago, but also more recently, all countries agreeing to phase out fossil fuels. It's monumental to get all countries in the world to agree on something. I'm especially passionate about building standards — specifically, those for calculating carbon emissions in enterprises — and making them the default and a legal requirement across multiple jurisdictions.

SPENCER: So what is this agreement where countries have all said they would phase out fossil fuels?

KRISTIAN: That's essentially an extension of the Paris Agreement. That was signed back, I think, two years ago, in the United Arab Emirates. I've actually been a part of the Swedish government delegation to COP for multiple years. Some years are quite slow; not a lot happens. But that year, they actually agreed to phase out fossil fuels. Obviously, the timelines and the level of ambition for different countries are completely different, but they all agree that fossil fuels must be phased out, and that includes the big oil-producing nations of the planet that would be less incentivized to do so.

SPENCER: So, what do countries actually agree to? Did some of them set timelines? And is there anything that actually makes them follow that?

KRISTIAN: So there are these nationally determined contributions. All countries have their own individual timelines for when to phase out fossil fuels. It could be 2050, for instance, which is very common. However, some less ambitious countries have 2060 and 2070. When it comes to enforcement of these NDCs, there are no real enforcement mechanisms. I think that is generally a problem in global governance. We can write the international legal frameworks all day we want, but it's very hard to enforce those frameworks in general because there is anarchy in the international arena. I'm sure we're going to talk more about that later on in the conversation.

SPENCER: But do you think the agreement will actually cause positive effects, or do you think it will end up not mattering because countries that want to do it will do it anyway, regardless of the agreement, and those that didn't want to do it are just going to renege on the agreement?

KRISTIAN: I think it will cause positive effects, and it's already causing positive effects. It sends signals for capital flows to be invested in green technologies. It sends signals that even countries like Saudi Arabia are massively trying to decouple from fossil fuels because they understand that it will not be sustainable in the future. The interesting part that I'm very passionate about is the signals it sends in value chains. If you have big countries like the European and U.S. economies that are committed to net zero, then the companies within the jurisdiction of those countries need to go to net zero as well. The way for them to go to net zero is that, on average, 90% of their emissions are located in their supply chains and value chains. The producers in countries like India or China, or mostly Asian countries, need to push those suppliers to go net zero as well. It becomes a network effect from the top down that has been quite impactful. Obviously, we're massively behind the climate targets, but if you imagine a world where none of those treaties existed, I'm pretty sure we would be even more behind.

SPENCER: Are you optimistic on climate? Do you think that, assuming that nothing else destroys the world, humans will be able to avoid major climate disasters?

KRISTIAN: No. I have to admit that I'm not particularly optimistic. I think the main way to actually solve it would be through technology. Let's say, for instance, that we succeed in building aligned artificial intelligence; then I think solving climate change will be a lot easier. A lot of interventions that are incredibly hard to implement have a quite bad cost curve in general. For instance, planting trees will be massively easier with artificial intelligence. You could imagine a whole swarm of autonomous robots more or less planting trees in deserts, and that way, get a massive amount of carbon emissions being sucked out of the atmosphere. To just mention one example, surely there will be breakthroughs in fusion, with the help of artificial intelligence and other alternative types of energy.

SPENCER: I've felt for quite a while that there are only three realistic strategies for climate. I'm just curious if you disagree with me. The first being government collaboration. If you could really get the major countries' governments all collaborating, it's within the power of the governments to greatly reduce carbon emissions and turn things around if there's enough political will. So that's strategy one. Strategy two is technology, which is basically if you can make it so it's in people's self-interest not to pollute, or the kind of technologies you're talking about where it just becomes much easier to do carbon capture. And then strategy three would be if there could somehow be enough pressure on corporations, because my understanding is there's a relatively small number of corporations, in the hundreds, that do most of the polluting. If there could be enough pressure put on them, then maybe that could make a huge difference. Other than that, I don't really see any viable strategy. What do you think about that?

KRISTIAN: I would agree with that, and I think they're sort of interdependent on one another. For instance, if countries can cooperate and double down on net zero, that often means doubling down on investing in new technologies that companies within those countries can develop. The reason why solar is so cheap right now is that Germany decided many decades ago to pour billions into solar R&D and massively subsidize it. Eventually, you reached a scale of innovation where it became more cost-effective, and then China actually turned out to do the same thing. Now, solar is cheaper than fossil fuels in most places around the world. That was a result of the government doubling down to invest in innovation. If you pressure corporations, you can also get a bit of an incentive to adopt innovations early on. For instance, a lot of trucking companies or car manufacturers were pressured to develop electric models, both from the public and eventually by their investors, because it became the new hype. I think all of those three interventions are somewhat interconnected.

SPENCER: Yeah, that makes a lot of sense. So did you then sour on climate as a cause, or did you just think that over time AI was a better use of your time? Can you tell us about that story or how you transitioned?

KRISTIAN: So I've been interested in artificial intelligence, safety, and governance for a really long time. I first got into it back in 2008, and my first real job after graduating was working at the Future of Humanity Institute back in 2013-2014. That was when Nick worked on Superintelligence. One of the reasons why I left the AI safety domain is that whenever we talked about it, the media published articles with big pictures of terminators. It was almost like a joke to people. It was this sci-fi thing that people didn't take seriously. I felt like it was very hard to get traction on global coordination around AI because, again, it's one of those problems that you need global coordination on. If one country defects and develops dangerous artificial superintelligence anyways, then it really doesn't matter. There needs to be a global governance regime. I felt like it was impossible to lobby for that back in those days. I wanted to crack the code on global coordination and governance in climate, but then when the "ChatGPT moment" happened, I started to look at AI again because it's a lot easier to convince people to take it seriously when there are examples out there of AI clearly being very intelligent.

SPENCER: Shifting topics a bit, and then we'll bring it back to AI in a few minutes, what is the Darwinian Trap?

KRISTIAN: The book that I published is called The Darwinian Trap, and within that book, I have this concept called a Darwinian demon that the book revolves around. A Darwinian demon is essentially defined as a selection pressure for an agent to behave in a way that is bad for others, essentially.

SPENCER: Could you give an example of that?

KRISTIAN: Absolutely. There are so many examples out there. The most obvious in nature is being a predator. That's a strategy that maximizes your own survival, but at the expense of maybe eating someone else alive. For that other animal, it's probably extremely painful. Also in nature, cancer is another example. Cancer cells, in the short run, might adapt by replicating as quickly as possible, but they end up destroying the host organism. Viruses are examples of that as well. We see the same thing in our societies. Corporations might be incentivized to seek short-term profit at the expense of the environment or people's health. Right now, when we talk about artificial intelligence, it can be at the expense of all future life on this planet.

SPENCER: Just to get really clear in this concept, if we imagine evolution and think of it for a moment as having the goal of species survival, or gene survival, maybe is an even more accurate way to say it. One way to do that is to increase the fitness of a creature at the expense of other creatures. A predator becomes better at hunting and then kills more creatures, but it survives better. Another way to do that is to do it in a way that's either neutral or even helpful to other creatures. For example, making a mutation in a gene that makes it more symbiotic with other creatures or makes it cooperate better with other creatures could benefit both it and other creatures. Is that right?

KRISTIAN: Yes, absolutely. In the book, I call this selection pressure for mutually beneficial cooperation. I call that Darwinian angels. Those examples exist all in our natural world. As a matter of fact, everything that is beautiful in this world is layers upon layers of cooperation. Cooperating molecules create a metabolism, cooperating genes create a genome, cooperating cells create you and me and us, cooperating creates societies and economies, and so on. It's layers upon layers of cooperation. You can see all of life as this tug of war between the forces of defection, the demons, and the forces of cooperation, the angels.

SPENCER: You also brought in this idea a moment ago of short term versus long term. It seems to me that's not quite the same thing, because you could have a Darwinian demon, where a creature gains fitness by harming other creatures, but it could still be beneficial for the long term survival of its own group. Maybe they're taking advantage of prey, and it's worse for the prey, but maybe their own group actually benefits. And so it's not necessarily a short term strategy; as long as it doesn't wipe the prey out to extinction, then, of course, it would be short term, because then maybe that whole group would eventually go extinct.

KRISTIAN: Absolutely, you're absolutely right about that. And I think this actually goes to the crux of the problem, because natural selection is sort of an optimization algorithm that can't make long term predictions. So it essentially only knows that a particular mutation will be adaptive from one generation to another. So it can almost only sense the slope or the gradient of the fitness landscape, but it will not be able to predict the long term consequences of a mutation. In biology, there is this concept of kamikaze genes or evolutionary suicide. One example of that would be predators that might have a mutation where they become so successful at hunting prey, maybe they become super fast or something, but they become so successful that they hunt all of the prey to extinction, and then they go extinct themselves. But there is no way for evolution through natural selection to know that this gene that is super successful to begin with will be less successful in the long run. I think to some degree, that's what is going on with our societies as well, where, in the short term, it might be adaptive for particular countries to be very exploitative in terms of resource extraction or building more and more dangerous weapons, but in the long term, it might actually cause us to go extinct.

SPENCER: Obviously, it's tempting to personify evolution. It's easier to talk about it having a goal of optimizing fitness, but really, essentially, all it's doing is there's random mutations, then there's natural selection, which causes some creatures to die and some to live, and then those that die off, or their genes get replicated less, and those that live long enough to breed, they get replicated more, and that kind of spreads their genes. I think your argument is that at the very short term, this is optimizing for survival into the next generation. It's not taking into account whether the whole species gets wiped out. But if you actually step back and take a bird's eye view, you could say, "Well, maybe it is also optimizing for long term, because those groups that end up getting genes that cause them to wipe themselves out, on the longer time scale, they do get wiped out through the process of natural selection; eventually, they do get wiped out of the gene pool." There's this whole concept of group selection, which is controversial in the field, but I think that a lot of people in evolutionary biology, as far as I understand, do believe in group selection, that there is some kind of selection pressure at the level of groups and not just at the individual.

KRISTIAN: Absolutely, and I think you're right about that. All of the examples that I gave before of major evolutionary transitions, going from cooperating molecules to a cell, and groups of cells outcompeting lonely cells, and then eventually creating organisms, are all examples of some sort of cooperative selection. There are five categories of them: group selection is one, but there's also kin selection, direct reciprocity, indirect reciprocity, and network selection. They all work in slightly similar ways. The problem here is that now, especially, but maybe even in our evolutionary past, there might have been mutations that led to the extinction of everything on the planet. If you look at past mass extinctions, arguably four out of six of them were caused by algae or bacteria replicating super aggressively and then causing climate change that almost killed everyone. From a cosmic perspective, you might be right; there might be one Earth-like planet where everything went right, and all learn how to cooperate, but it doesn't necessarily mean that it will happen on this planet, if that makes sense.

SPENCER: So to break down what you're saying, if there are group selection forces at an intermediate level, you have a bunch of groups that are competing. There are forces that cause one group to survive and the other groups to die. That does create an evolutionary pressure. But if the consequences are so great that all groups die, then in fact, there isn't any pressure, because it just ends everything. You can't sustain events like that.

KRISTIAN: Exactly. I think the most mind-blowing thing that changed my thinking around this was 15 years ago when I started to play around with evolutionary simulations. At the time, I just wanted to become a better Python programmer and played around with these evolutionary simulations of predator-prey dynamics and virus-host dynamics. What tends to happen in all of those simulations is that life died out because the predators became too successful in a way, and they killed all prey and then died as a result. I got obsessed with this idea of why that happens, and then I found these concepts of evolutionary suicide. If you look throughout our evolutionary history, there are a lot of points where we could have gone extinct. Imagine when life was confined to thermal vents on the ocean floor; a very successful predator back then, before life had a chance to diversify, could have potentially ended all life. After life gets diversified, all life has some sort of metabolism, meaning that we take in material, convert it to energy, and then create waste. That waste might become toxic for others. We have examples of very successful bacteria and algae creating climate change because they toxified the ocean, almost destroying all life. I think there might be a mental model of this where you could imagine life on this Earth as a random walk throughout a fitness landscape. In that fitness landscape, there might be landmines, but we will never be born on a planet where we stepped on a landmine. I think life is incredibly fragile, and I call that the fragility of life hypothesis. We might have been very lucky for 4 billion years or so.

SPENCER: You might be proposing a kind of anthropic principle, where complex intelligent life will never find itself on a planet where early life got wiped out. If it had gotten wiped out, they just wouldn't be there. Any intelligent life looking back is going to say, "Look, life survived all this way. It never got wiped out. Maybe it came close a few times to getting wiped out, and maybe it had these major resets, but it never got wiped out." But then it's like, what can you really conclude from that? Because if it hadn't been wiped out, you wouldn't be there to reason about it.

KRISTIAN: That's exactly it. In thinking about alien life, there's this concept of a great filter. Is there a filter throughout our evolutionary history that wipes everything out or makes it incredibly hard to create complex life to begin with? It could be that creating a eukaryotic cell that enables multicellular organisms is just incredibly hard, and that's why we don't see aliens. When we look out in the night sky, it might be that later on, when they become intelligent, they tend to destroy themselves. The idea of Darwinian demons offers an alternative where the whole thing is just a great filter because evolution, through natural selection, can't think of long-term consequences. It might end up mutating in a way that is essentially like a landmine, like the examples I provided earlier.

SPENCER: So it acts as a short-term filter at the level of species or genes, and maybe it acts as a medium-term filter at the level of groups, but not a long-term filter at the level of life or most species, or something like that. So when we apply this to society, I think, in part, you want to use it as a kind of motivating framework to think about society today. How do you make that application?

KRISTIAN: So you can sort of do it in your everyday life, in a sense. You can look at it from a framework of natural selection when you look at why some colleagues that are really bad get promoted. Well, it might be because there is not necessarily a selection pressure for competency. One way to sort of hack the system might be to just nag your boss and become friends with your boss, and therefore you get promoted, but not because of skills. So there are everyday situations that I talk about, and they cover everything from fake news to corporate greed on Wall Street. But then there are also the bigger picture things, and maybe we can talk about both of those categories. The bigger picture things are like the arms races between corporations and nation-states, where, in order to win as a civilization, you need to extract resources quicker than anyone else, because then you can have a more powerful economy. But with finite resources, it creates a selection pressure for more and more dangerous weapons. In order to create more dangerous weapons and more advanced economies, there is also this selection pressure for intelligence that enabled public education but is now powering the artificial intelligence arms race, because with more intelligence, you can invent better strategies to compete on the international arena.

SPENCER: In order to apply these ideas from evolution to something in society, my understanding is you need at least three components. You need some unit of information that gets copied. You need some form of mutation or changes to the copies, and you need some kind of selection where some copies survive and some do not. If you have those three things, you can really use the evolutionary thinking that we use for genes and species. You mentioned companies, and you can see how this could apply to companies, because companies have different kinds of information that might get copied. What kind of business they're in could get copied. Maybe their culture could get copied. You can see that there is some kind of replication mechanism, like other companies can crop up that try to copy them, and then there's a filtering mechanism. Some of them go out of business, and some do not. Also, people can leave the company but bring aspects of it to a new company or start a new company. You can kind of see all three of those forces. Do you think we can literally apply evolutionary thinking to companies?

KRISTIAN: I think we can, because of the reasons that you mentioned, and I'm really glad that you asked that question to get a bit more rigor here. The unit here is almost a unit of culture, or what Richard Dawkins would call a meme, and that meme can spread orally, like a business culture. I think the most common way when it comes to countries and companies is that in companies, you have these strategy documents. How are you supposed to behave? Look at the social media landscape, for instance. One strategy might be pay-per-click. That's how you make money. Other companies decide to copy that model because it turns out to be successful. Similarly, when it comes to countries, laws and the legal system can be such examples. All countries on the planet recognize legal persons, like corporations as these entities, and also recognize shares within these entities as some sort of capital. You can have this evolution happening on a corporate and company level through these strategic or legal documents, more or less.

SPENCER: If we apply those same ideas to academia, you could also see how you might have those three things. You can have a unit of information, which could be a culture of how you do research, or what topics you study, or the methods you use. Then you could see a copying, because advisors are going to train the next generation, and that's going to spread those ideas. Then you have selection, which is who gets to stay in the field, who gets tenure, who gets published in top journals. It seems like a natural fit there as well.

KRISTIAN: Exactly. This is where I think a concept called Goodhart's Law can be quite useful, because in many fields and occupations, there's a definition of success or a way to measure success. In academia, for instance, a way to measure success might be the h-index or number of citations. The h-index is essentially a way of calculating how impactful a researcher has been based on their publications and the number of citations from those publications. What Goodhart's Law states is that whenever a metric becomes widely used, there are ways to hack that metric. It might actually be adaptive, from a natural selection perspective, to find new and creative ways to hack that metric. In academia, for instance, we're seeing a replication crisis, and a lot of that is people engaging in so-called p-value hacking. You essentially slice the data in a way to yield a statistically significant result, even though it's useless. We have also seen many cases of fraud where people actually use Photoshop on images and so on. Once that spreads, and you have to compete with others that don't follow the rules, it's almost like you're forced into fudging the numbers and hacking the metric yourself if you're a researcher and you want that limited grant money.

SPENCER: If the way that people were promoted or got tenure was more of a broad measure of how good the science they're really doing is, how high quality it is, then it would be a lot harder to hack. But if it's literally counting the number of papers in top journals, then it becomes a game that you can optimize for, and you can find ways to technically get those papers, even if you're not producing high-quality science.

KRISTIAN: Exactly.

SPENCER: Okay, so let's apply this to AI in particular. What do you see as the clearest way of applying it?

KRISTIAN: There are multiple ways of applying it. One that I'm fascinated by is the arms race between AI companies because we're in this incredibly unique situation where the CEOs of all the big AI labs, such as Google, DeepMind, Anthropic, and OpenAI have all talked about the existential risks of the technologies they're developing. Approximately two years ago, they all signed a statement saying that AI is an existential risk, equivalent to nuclear weapons and pandemics. Around the same time, there was another open statement suggesting we should pause AI development, but none of the CEOs signed that statement. It's a weird situation where they all realize that the technologies they're developing might pose an existential risk, but they're not willing to pause. I think it's back to the arms race dynamic, where if you decide to pause, then the others will win the arms race. If you're incentivized not to pause, there is a level of rationalization as well, where everyone thinks they are the good guys. If they have to do less safety testing after releasing new models, it's actually for good, because if they do less safety testing, they are more likely to win. Obviously, they're the good guys, so they should win.

SPENCER: Can you just tie that together with what we were discussing in terms of applying evolutionary thinking?

KRISTIAN: How I would apply the concept of Darwinian demons to the AI arms race is that there are financial selection pressures. In order to win, you need access to capital and talent, and if you decide to pause, then you're less likely to get access to talent and capital at the end of the day. So there is a selection pressure for creating more appealing products that users will use.

SPENCER: Right. If you had a company that was focused on making AI safe and not on making money or growing, it might be harder for them to get capital and then actually get outcompeted.

KRISTIAN: Yes, exactly. I think we have seen a little bit of that in action, right? With OpenAI deciding to create the for-profit entity to raise a bunch of money, and then we had the whole drama around the board firing the CEO of OpenAI, Sam Altman, because they were concerned that he apparently wasn't completely honest with the board, which had implications for safety. The board was assigned to safeguard the mission of OpenAI, which is for AI to benefit all of humanity. But in the end, they need money to run operations, and all of their funding came from Microsoft. So when Sam Altman got fired, he ended up going to Microsoft and told them and all of the employees of OpenAI that either they let him stay or he would bring all of the best employees with him to Microsoft, because that's where all of the money exists. I think it's an example of how money wins at the end of the day.

SPENCER: So do you see AI as a more important or more pressing cause area than climate?

KRISTIAN: I do see AI as more pressing, and I think I've always seen AI as more pressing, at least sort of intellectually, because it will be one of those transformative technologies. But what I didn't expect was that it would be so fast. My predictions for when we would have AI that is as smart as we have now is maybe more than 10 years in the future, so 2035. I think AI is more important.

SPENCER: And can you elaborate on why you think it's more important?

KRISTIAN: I think AI is actually the most important thing because it is, in a sense, the ultimate dual-use technology, meaning that it can be used for both good and bad. Artificial intelligence, just like human intelligence, can be used to create nuclear energy that is cheap and clean, but it can also be used to create nuclear weapons. I think what AI will enable is a whole host of new technologies being unlocked, and all of those new technologies might create an arms race in and of themselves. One example that I bring up in the book is the technology of engineered pandemics, or weaponized pandemics. You could, for instance, imagine that with the help of artificial intelligence, you could take a virus like rabies, which has a lethality of 100%. If you get it, then it spreads to your body, you're more or less guaranteed to die. But what if you can make rabies airborne, and what if you can make it have the incubation period of hepatitis B, which is 10 months, meaning that it can spread throughout the population for 10 months, and you will not know that you have it until people start dying. Furthermore, what if you can engineer it to only target specific ethnicities? All of these things could be possible in principle with the help of artificial intelligence. As we know from the race towards nuclear weapons, as it's brilliantly depicted in the movie Oppenheimer, once you have this arms race dynamic, you almost don't have a choice. If you're concerned that China is going to develop this technology, for instance, the US will be incentivized to raise that technology first. I think AI will unlock a whole host of such technologies in the technology tree.

SPENCER: What work is AI doing there? If we're saying AI is going to make this more dangerous, people might think it's already getting more dangerous every year as biological knowledge increases and scientific understanding of gene editing increases. The kinds of risks you're talking about are already becoming more likely. So why does AI matter in that equation?

KRISTIAN: I think AI matters in that equation because it will be a lot easier with the help of AI technology to figure out what genes are doing what specifically. Of course, it's not AI alone. You need to have some sort of information feedback loop where, for instance, you can, with the help of CRISPR, try different genetic combinations and see how the specific genetic combinations affect what proteins are being produced, the morphology or behavior of a particular virus. But I think that loop can more or less be automated with AI, so we can have very quick feedback loops in terms of what genes are doing what specific things, and then AI is this pattern-matching machine. If you just have enough data, AI can figure out what genes and what DNA sequences are responsible for what.

SPENCER: So then is the concern that AI will accelerate these kinds of arms races, and then one of these arms races will end the species or destroy human civilization?

KRISTIAN: That's exactly it. It will create not just one of those arms races, but a ton of those arms races simultaneously. I mean, similar to how our increased understanding of the quantum realm created the arms race of atomic weapons.

SPENCER: Yeah. So give us another example of an arms race that AI might create that's not biological.

KRISTIAN: So there are, of course, other domains of weapons manufacturing that could be enabled by AI. One thing that some people are talking about is the worry of atomic precise manufacturing. This is not my domain of expertise, so I wouldn't be able to tell you exactly how AI will help with atomic precise manufacturing, but I'm sure that others who are experts would be able to go into the details. The idea here is that, with the help of artificial intelligence, we could be better at mastering very small scale production. For instance, you could create almost this mist of small nanobots that could be spread and enter into people's bloodstream when you breathe them in, creating all sorts of problems. That's another weapons example. But I think there are also a lot of examples in the non-weapons or non-misuse domain that I'm happy to talk about as well.

SPENCER: Sure, what's one of those?

KRISTIAN: I think one of those would be the gradual disempowerment of humans. It wouldn't be one weapon that wipes us all out, but rather a gradual replacement of humans. The whole idea here is that, in general, there are very few examples of a more intelligent species being dominated by a less intelligent species. If you look from the perspective of Darwinian demons, there will be a selection pressure for countries and corporations to outsource more and more of their labor, but also decision-making to artificial intelligence. You could imagine a future where you will have AI CEOs, or you might still have human CEOs, but you will have AI advisors that are the real decision makers behind all decisions. Similarly, you will have AI advisors to politicians telling them exactly what to do, or AI advisors to military leaders. It's almost like this co-evolution where we're voluntarily giving up more and more power to the artificial intelligence systems, and we're building a society around the incentives of corporations and countries, instead of the incentives of human beings. The main point here is that if you look at the most oppressive dictatorial regimes, they at least need to care about human welfare and well-being minimally, because they're dependent on human labor to stay in power and to be profitable. But with artificial intelligence, they might not be dependent on human labor anymore. They can directly convert capital into compute that are used to run AI models that create the economic value they desire, essentially, and humans will not need to be around anymore. We will sort of be a little bit like rats in a big city. There might be some wealth trickling down to us, but a cityscape to the rats is entirely foreign to them, and they have no control over it. Similarly, we might live in a society that is not controlled by us and utterly alien to us.

SPENCER: So is the key to that argument that if companies and countries increasingly depend on AI labor rather than human labor, that essentially allows them to be more detached from producing human values and less bound to doing what people want?

KRISTIAN: Yes, that's exactly it. There are fewer incentives for them to care about what humans want at the end of the day.

SPENCER: So are you more concerned about these kinds of scenarios you just described, of companies and countries being less tied down to actually caring about people's values than you are about loss of control scenarios where eventually people build a kind of super intelligent AI and they lose the ability to actually control it?

KRISTIAN: Yeah, I'm more concerned about those slow takeoff evolution scenarios that I just mentioned than the loss of control scenario. I might change my mind about that in the future, but right now it seems we need exponentially more compute to make the AI models smarter, which seems to indicate more of a slow takeoff. But also, these things can move really fast. I would not be too surprised if we have some sort of algorithmic breakthrough where all of a sudden, an AI model can recursively self-improve and make itself a lot more effective. For instance, some paradigms of AI safety, such as mechanistic interpretability, where you actually try to figure out on an artificial neuron level what is going on, could lead to an AI essentially knowing how its own brain works and then upgrading that brain rapidly. I think both those scenarios are really scary, but right now, I'm worrying about the slow takeoff gradual disempowerment scenario a little bit more.

SPENCER: Is it just because it's more likely to occur?

KRISTIAN: Exactly, because I think it's more likely to occur. I think it's important that regardless of which scenario is true, we need to govern this technology globally, and the mechanisms for governance are quite similar. We need to keep track of where all of the big AI chips are and who has access to this technology. We need to be careful about our approach to artificial intelligence, ensuring we don't build AI models where it's impossible for us to know exactly what they will do. We might have more of a symbolic approach to AI, where we can use mathematical proofs to figure out what actions it's likely to take and so on. I think the solutions are roughly the same for all of those worries, and we should strive towards those technical solutions and governance regardless of which scenario we happen to worry about the most.

SPENCER: And what would you say to a listener that thinks, AI is just another technology. We've seen lots of technologies that have made things faster, easier, and cheaper to do, have enabled us to do more and more things, and have even made weapons more viable. What's so special about AI?

KRISTIAN: I think AI is more special because it's essentially a meta technology. What enabled all previous technologies is our intelligence, but our intelligence is fundamentally bounded. To create the atomic bomb, we had to gather the most brilliant physicists alive in the last century, and the result was the atomic bomb. But if you can build AI that is smarter than humans, it's not just a specific technology. It's not like a car or a vacuum cleaner that only has one particular purpose. It is really this sort of meta technology that can enable innovations across all possible fields, which fundamentally means it can be weaponized.

SPENCER: You might argue that electricity and the internet were both meta technologies of the same form. Do you think that the traits that AI has make it fundamentally different from those as well?

KRISTIAN: So I think both the internet and electricity were tools. Electricity, of course, can power motors that make our labor a lot more efficient, like manual labor, and the internet made communication a lot faster. But intelligence is a completely different thing because then we're talking about fundamental cognition and intelligence. My favorite definition of intelligence is that it is an agent's capability of achieving whatever their goal is in multiple different environments, which essentially means that if you're more intelligent, you're not just better in chess or better in physics; you're better across the board at everything. Obviously, you can use that to develop new drugs and solve climate change and all of these different problems, or you could use it to create new and dangerous weapons, and which one it is depends on the incentives we have in our society. I think right now, at least to me, it's clear that we're sort of driving towards the cliff, but we're driving in this old sedan that is really slow, and with artificial intelligence, we're upgrading that old sedan to a Ferrari.

SPENCER: What would global governance look like for AI? Suppose that you're trying to make the world safer from AI. What can you actually govern?

KRISTIAN: A lot of people have said, and I think rightfully so, that artificial intelligence might be incredibly hard to govern because it's software at the end of the day. Software can easily be copied, which is a huge problem. However, in order to train these hugely intelligent AI models, you need a whole data center of AI chips, or so-called GPUs. For instance, look at X AI's latest model that required 100,000 of these really expensive NVIDIA GPUs. That could actually be a tractable way of governing artificial intelligence because it's more or less only two companies that design these chips that you need to create AI, and they're all manufactured in one factory in Taiwan called TSMC. They all use lithography machines from this Dutch company, so you have this enormous concentration in the AI value chain, and each point in that value chain is a potential governance node. I think the most compelling one is compute. You could actually build into the hardware, the chips, mechanisms whereby you're not allowed to train an AI model without certain evaluations that it's not going to be dangerous. For instance, you can build the chips in such a way that they will reject the training of the AI in the first place unless you can prove that the AI will be safe.

SPENCER: How would that work in practice? What would the chips actually be checking and how would they get approved to continue training?

KRISTIAN: So that's a great question. This is something that my current company, Lucid Computing, is working on, and it is a tough problem. I think it's especially tough with the current paradigm of artificial intelligence because when you train a big neural network, it's more or less impossible to know exactly what it will do. So I think fundamentally we would need to change our approach to artificial intelligence, where instead of building these general intelligences that are brilliant in all domains, we build tools that are brilliant in specific domains. For instance, a chess engine being brilliant at chess, but nothing else, or AlphaFold being brilliant at protein folding, meaning that we can discover a bunch of new drugs as a result. So I think we need to change our approach to AI development, and you could sort of build these tiny computer programs that you could run inside of the chip in something called the Trusted Execution Environment that can essentially test that the AI is being trained using a particular paradigm, for instance. So at least in theory, that would be possible.

SPENCER: Is that really realistic at this point, though? The genie is kind of out of the bottle. All these companies are making large language models or open-source large language models. These are fundamentally generalist technologies where you can chat with them about your medical problems, or you can ask them to solve a math problem, or you can get them to write an essay.

KRISTIAN: I don't think it is realistic at this moment in time, and that is why, with my current company, we're focusing on a specific subproblem, which is actually where the chips are located. To make sure that the chips are not in China or Iran or any dictatorial regimes or that no terrorist groups have access to them. So that is a more tractable problem at the moment. However, I think these things will change over time, and unfortunately, I think the only way to make incentives aligned for a completely different approach to artificial intelligence would be us being close to a major catastrophe. For instance, when the Soviet Union and the US had their power struggle, there were incidents such as the Cuban Missile Crisis, where we were very close to World War III. When you're really close to World War III, you have no choice; either you cooperate or you enter into the war, and a bunch of people will die in that war. In the Cuban Missile Crisis, the US and the Soviet Union decided that the US would withdraw a lot of their nuclear weapons from Turkey, and the Soviets would not put nuclear weapons in Cuba. So they reached an agreement. I think there might be a point in time where the great power conflict between China and the US reaches that tipping point where either we cooperate around the governance of this technology or there is a hell to pay in the next World War.

SPENCER: And suppose that we don't get to that point where war is imminent, and therefore suddenly there's a will to control things a lot more. How do we get the world to come around, or the governments to come around, to actually put in place some kind of regulation that helps protect things?

KRISTIAN: So I think, for instance, the US at the moment has a lot of power to use export controls on other countries across the world to incentivize these types of investments. So for instance, the US can say, You're not going to get access to all of these chips in your big data center in the United Arab Emirates or Saudi Arabia unless you can track where the chips are, and unless you can track that the chips are not being used to train an artificial superintelligence that is unlicensed in some way." So that could be a more tractable short way of doing things. But for instance, that will not be applicable to China. China is already cut off from our advanced AI chips, and as a result, they are now racing to create their completely independent semiconductor value chain. So I think at some point it has to become international. But I think you could do it step by step, starting with export controls, and then eventually, when the world is ready, and we have taken the arms race too far, the right technologies will exist in the world because they have been used for export controls, and then they can be used for more global governance.

SPENCER: Many of these challenges you're talking about seem quite daunting, because it seems like there are local incentives to do things that make things riskier, rather than incentives to make things safer. If every company makes more money by making the best, most effective AIs, then that just pushes everything forward faster and faster. And if it really requires a lot of global coordination to monitor things more carefully. Well, we know coordination is difficult, and it's often very hard to do this. So I'm wondering overall, do you feel pessimistic about the situation, or do you think it's optimistic that we will actually be able to make these technologies safer?

KRISTIAN: Yeah, honestly, it fluctuates for me day to day whether I feel pessimistic or optimistic, and I think one part of me feels incredibly optimistic. I feel like we're making a lot of progress on this problem. More politicians than ever before understand the problem. People are starting to understand the problem. Whenever I go abroad and take a cab, I tend to ask the cab driver about artificial intelligence. It's clear that everyday people are getting more and more educated. So personally, I actually don't feel afraid. I don't feel doom and gloom on an emotional level. I'm incredibly optimistic, but then if I take a step back and try to look at things more objectively, it is sometimes hard not to be a pessimist, because going back to what we talked about earlier, survivor bias and the anthropic principle, I think life has this tendency to self-destruct. If you look throughout the universe, despite there being probably millions of potentially habitable planets in our galaxy, we don't seem to see life anywhere. So it seems to be a very unstable thing, so maybe we will self-destruct. But certainly on an emotional level, I feel more optimistic than pessimistic.

SPENCER: So your solution to why, when we look at the universe, we don't see it teeming with technologically advanced life is just that life wipes itself out too quickly.

KRISTIAN: That's what I think. As I said earlier about the great filter, I think it's just a continuous filter. It's not just one filter. I think throughout evolution, through natural selection, there could be mutations that are beneficial in the short run but will wipe us out in the long run. Such a "mutation" could be the invention of artificial intelligence, but it could also be the invention of a super virus or a bunch of other things. Nick Bostrom has this analogy in his paper, The Vulnerable World Hypothesis, that every technology is almost like taking this random ball from an urn. A black ball might be a civilization-destroying technology. Whenever there is a mutation, it is almost like playing that game of taking that ball from the urn, and with AI, there will be a lot of new balls. It's almost like AI will create this entirely new urn of technologies that we could have never dreamed of.

SPENCER: If the main dangers from AI are the literal destruction of civilization or the death of all life on Earth and the AI ceases to exist, then that could be a partial explanation for the Fermi Paradox. But if the AI would then continue to exist and continue to develop and expand, it doesn't really explain things, if that makes sense. So even if AI wipes all life on Earth out, if the AI continues to build, then there is another question: why don't we see AI out there in the universe doing lots of stuff?

KRISTIAN: Yeah, and I think the answer there is probably that none of the other planets that had life got far enough to develop artificial intelligence; they self-destructed way earlier than that. So it could be the examples I had earlier with bacteria replicating so rapidly that it causes climate change or acidification of oceans that kills all life, for instance, or super predators when life was confined to these thermal vents in the ocean floor. I think life probably self-destructed before we had AI to begin with. Because, as you rightly point out, if AI is successful, it can spread throughout the galaxy; actually colonizing the whole galaxy would only take a couple of million years. Life on Earth has existed for 4 billion years, so we should probably see a bunch of AI alien structures out there in the whole of the Milky Way galaxy. But the reason why we don't see that is probably that life self-destructed before that point on all of those other planets.

SPENCER: Suppose if there were many civilizations out there in the universe that created AI and got wiped out, there could, for some reason that we don't understand, be a reason why AI tends to not explore. It just kind of sits on that planet, does its thing, rather than spreading throughout the galaxy. I don't know why that would be, but perhaps it's another explanation.

KRISTIAN: Yeah, it could be. And I think also what I suspect is that in general, there will be some almost cosmic selection pressure to evolve really slowly. So, if you sort of have this mental model, again, of life as this random walk throughout the fitness landscape, in some places in the fitness landscape, there will be life-ending mutations that you could illustrate as a landmine in the landscape. Then you will actually exist for longer if you move really slowly throughout that landscape, and you're very careful about technological innovation. So I think if we sort of invent AI in the future, they might realize that we need to put a lot of resources into future predictions about the implications of technologies before we put those technologies in the world. For instance, really advanced civilizations might realize that researching this particular quantum thing or whatever, maybe there could be very dangerous, life-threatening technologies coming out of that. Therefore, we want to make a bunch of predictions and spend a lot of resources on running advanced simulations on the consequences of potential technologies before we put them out there in the world.

SPENCER: It seems that doing that kind of thing would require extremely good approaches to solving collective action problems and other kinds of multipolar traps. Because, okay, maybe some group decides, let's not build this technology because it's too risky. But then another group just builds it, because they estimate the risk as being less or they're just less safety conscious, or there's just some local gain that they want to get, and so it's worth it for them to take the risk.

KRISTIAN: Yes, and that gets me to the other part of my book where I talk about solutions to these multipolar traps, or Darwinian demons, as I call them. One such solution would be just a very strong world government that can penalize any potential defectors, so sort of more of a global police state. But then, obviously, with such a solution, you have the risk of some sort of totalitarian lock-in, or it actually becomes this very dystopian totalitarian state that just monitors everything that you do in order to make sure that you don't create any existential risks, and your freedoms are being taken away and so on. In the book, I also discuss more decentralized options to make that happen.

SPENCER: Yeah. So what are some of the decentralized options?

KRISTIAN: So I think that decentralized options are mainly based on the fact that our economies are incredibly interconnected. I have this example in the book of a guy on the internet who tried to create a chicken sandwich from scratch. You might think that creating a chicken sandwich is easy. You just take the chicken from the fridge and the sandwich, and then you make it. But doing it from scratch means growing grain, milling the grain, and doing all of that using more or less Stone Age technology, because, again, you're not allowed to trade with anyone else. So you can't buy a fancy milling machine or whatever. It took that guy $1,500 and six months to do a chicken sandwich, and that just exemplifies how incredibly interconnected everything is. When you talk about something like the semiconductor value chain and the AI value chain, there will be cooperation happening across so many different actors in the value chain. If one of those actors decides to say, "Hey, we think that this is a little bit dangerous, we are going to pull out from this," then the whole value chain would collapse. I discuss some ways in which you could, through the help of reputation and something called reputational markets, leverage that interconnectedness in value chains for more safety.

SPENCER: Because I suppose that if there's a kind of chain where a bunch of things have to happen to use this technology, then you could avoid the kind of multipolar trap by, say, putting a bottleneck at some part of the chain and saying, "Well, everything has to go through this bottleneck anyway. So you can't really act unilaterally unless you use this part of the chain that we can control."

KRISTIAN: Exactly. One way you could see it is that every essential part in the chain has a vote, in a sense, like, "Do we want to do this, or do we not want to do this?" When it comes to creating dangerous artificial intelligence, you could leverage that vote to say, "Hey, I think this is dangerous. I don't want to do it." But that would require a lot more supply chain transparency than we have today, where actually, the creators of the chips will know that, upstream, when we sell those chips, they might be used for dangerous things. That is actually something that I discuss at length in the book: "How could we create more transparent value chains," where every actor in the value chain could actually leverage their vote for good?

SPENCER: Before we wrap up, if there's one idea you want to leave the listener with, what would it be?

KRISTIAN: I think the one idea that I would leave the listener with is we need to change the narrative around our core problems. I think we have a name, shame, and blame type narrative where all of our problems are due to this particular corrupt politician or this particularly greedy CEO. I think that narrative is not serving us because it's not getting to the root cause of the problem. So instead of hating the players, we need to hate the game, or perhaps even better, change the game.

SPENCER: Kristian, thanks for coming out.

KRISTIAN: Thanks a lot for having me, Spencer.