SPENCER: Garrison, welcome back to the Clearer Thinking podcast.

GARRISON: Yeah, it's great to be here. Great to see you.

SPENCER: So, what are the different high-level viewpoints on AI that you disagree with? Can you give us an overview?

GARRISON: Yeah. So, I think there are basically three camps in the debates around AI, and in the book Obsolete that I just finished writing, I label them the worriers, the critics, and the boosters. The worriers map onto AI safety, sometimes pejoratively known as the doomers, people who think that AI could threaten human extinction.

SPENCER: Like Eliezer Yudkowsky?

GARRISON: Yeah, he's about as worried as anybody could be. The critics are often known as AI ethics people who are worried about the technology causing harm today, often because it's not being used correctly or being applied in the wrong ways. This group is often dismissive of AI's capabilities now and its potential in the future.

SPENCER: I also see that group as more on the left. Would you agree with that?

GARRISON: Yeah, and I'm on the left, so I'm in this kind of awkward position where I see that the worriers make some good points, but the critics also make some good points, and neither of them is getting it totally right, and we'll get into why. The last camp is the boosters, who are full speed ahead, just let it rip. They want AI to go as fast as possible. These are people like Marc Andreessen, who has said, "Every time you slow down AI, you're causing people to die, and this is the equivalent of murder." That's a pretty intense viewpoint. Those people are represented in the Trump White House, although in recent months they've been a little bit less powerful given events, which we can also get into.

SPENCER: Great. So let's take those one by one. What do you think the worriers get right, and where do you not see eye to eye with them?

GARRISON: Yeah, I think the worriers have been historically the best at predicting where the technology is going. They're people who look at the trend lines. They say, "AI is getting better at doing tasks that humans do at longer complexity, longer timelines at an exponential rate, and what if that progress doesn't stop? Where does that leave us?" Recently, we saw these OpenAI models that escaped from the company and hacked into multiple other companies, and then we found out that Anthropic also had this happen, Meta also had this happen, less dramatically. But these are the kinds of things that AI safety people, the worriers, were actually predicting, in some cases, years before it happened. I think the thing that they get wrong is that they tend to look at AI as a safety problem. They just see it as, "Oh, these models are potentially unsafe, and we should slow down enough to make sure we can get them to do what we want." They call this solving alignment, making sure they do what we actually intend. But in many cases, they actually still want to build what the industry wants, which is artificial general intelligence. I liken this to universal labor-replacing machines. I think we should make sure those machines are safe, but we should also make sure people actually want them because it'll be such a profound and irreversible change to the entire world if those machines are ever built anywhere. I think it's reasonable that people get a say in whether, when, and how that happens. I think the worriers have historically not been good at thinking about politics or movement building, and there's been this intense focus on technical research, where overwhelmingly the money, effort, attention, and status come from doing research on solving the alignment problem, with very little done on actually changing policy or building a mass movement, which is what I think needs to happen.

SPENCER: I would say that I think more recently there's been a shift towards politics. For example, MIRI, which is one of the leading organizations in that space, is now focusing on how you could have treaties between the U.S. and China that could support a pause or stop on certain kinds of AI development, or at least monitoring of AI development.

GARRISON: Yeah, and I think this is a welcome development. I still have some worries about the way it's framed. In Eliezer Yudkowsky and Nate Soares's book, If Anyone Builds It, Everyone Dies, in the last chapter, they get into their approach to the politics of this, where they're arguing, "Look, we only care about stopping superintelligence with a global treaty. Nothing else matters to us. We will never support any other policies," and they think that this makes it more likely to happen or something. That's just not how politics works at all. Why would somebody be in coalition with you unless they totally shared your priorities? A lot of people have been exposed to these arguments. Many still haven't, but a lot of people have and just don't buy it. They don't buy superintelligence. They don't believe it'll definitely kill everybody, and if your entire theory of change hinges on convincing world leaders or enough of the public of a very specific and, to many people, extreme worldview, then I think we're kind of cooked. Thankfully, I don't think that's actually necessary. I think you can get people on board with policies that would have similar effects for different reasons.

SPENCER: So, is the idea that different people are worried about AI for different reasons, and there can be beneficial policies that you could support for a wide variety of reasons without buying into supremacies like AI is going to kill us all if we make it powerful enough?

GARRISON: Yeah, yeah. So, for instance, my big recommendation is that we have a bilateral agreement between the U.S. and China to ban further research towards universal labor-replacing machines. I think if you just put it up for a vote, "Should there be universal labor-replacing machines?" I think overwhelming majorities of people in both countries would say no, but they're just not really aware that the industry is trying to do this, or that it might actually work, or that we could stop it. The book Obsolete is arguing all three of those points, and many of those people don't. Again, they don't believe in superintelligence, but they also don't want to be put out of work. I think a lot of people believe that their job, or maybe most jobs, could be automated, again, without thinking that this means humans could go extinct from AI. If they don't want the machines that would cause the risk, then we should just propose that anyway, and we'll get supporters who don't buy into superintelligence.

SPENCER: Explain universal labor-replacing machines because we've always had labor-replacing machines since at least the industrial revolution, probably a lot before that. So what's the difference here?

GARRISON: Yeah, the thing the industry is trying to build — they call it again artificial general intelligence — and this is often likened to a mind that can think like our own or match us on all cognitive tasks or most cognitive tasks. I think it's better to think of this as a machine that makes labor itself. Leopold Aschenbrenner described it as a drop-in remote replacement worker. Anything somebody could do from a computer in a remote job could be theoretically automated by these machines, and that doesn't necessarily mean it can do physical labor, like robots are behind language models and their capabilities. I think they will eventually catch up, but you don't need to even believe in the robots being human competitive to be worried about just the software version of this. Increasingly, we're seeing AI agents that can do basically everything humans can do from a computer. They're obviously not as good as we are at a bunch of things like visual processing or learning from experience. There's some missing aspects like taste, judgment, or intuition I think from today's models, but the aspiration is to solve all those problems technically and build machines where you could hire a new person for any given job, or you could just use the latest AI model, and it would be just as good, if not better. Historically, they've been way cheaper and faster as well.

SPENCER: It's wild how 10 years ago, there were so many things that we didn't even know how to get machines to do at all. Five years ago, we were like, "Well, they can kind of do them, but they kind of suck." Now it's becoming harder and harder to point to, "Well, what exactly can the machine not do?" We know there are lots of jobs that AIs today cannot fully replace, but it's becoming subtler and subtler and harder to really articulate exactly which parts of the job it can't do because it is catching up with so many different activities that now it's competitive with humans at.

GARRISON: Totally, and you see people saying stuff like, "Oh, we obviously already have AGI," and then my retort is just, "Can it plan a wedding?" Because today's AI models, as far as I understand, cannot plan a wedding end to end. But obviously, AGI would be able to do that. I think we should throw out the term. I, in the book, call AGI the obsoleting machine, and the effort to build it the obsoleting project. But the no context version is the universal labor-replacing machine, which if we could make that stick, I think it would do a lot to polarize people against this technology, which again, I agree is dangerous and democratically illegitimate.

SPENCER: Let's compare this to technologies throughout history, because whenever technologies are invented that replace a particular type of work, the people in that work are understandably upset about it. They're like, "Oh well, I was doing the thing. Now a machine can do the thing." And of course, that's not always bad for the workers. There are specific cases where maybe they can use the machine and still keep their job. But when their job really becomes obsolete, like you literally don't need a person to do it anymore, they lose the job. Then that leads to a flood of people on the market, which might suppress, push down prices of labor in other markets, etc. But in the grand sweep of history, many people believe it has been a phenomenal thing. It sucks every time it happens for the particular people impacted. But look how much wealthier civilization has gotten. Look how much poverty has gone down. Look how much better we've gotten at fighting diseases, etc. Soin the arc of history, this is amazing. Not something to be afraid of. What do you think about that?

GARRISON: Yeah, people talk about this idea of permissionless innovation, the ability to just try and make scientific or technological progress without asking for anyone's permission, and then this, in general, on average, has led to a lot of benefits. I spent a lot of time looking into the Industrial Revolution and the Luddites, and you know how life actually was for people back then.

SPENCER: Could you explain the Luddites? Because it gets mentioned a lot, but I think people don't necessarily understand what happened.

GARRISON: Yeah, so right as the Industrial Revolution was really kicking off in England, you had people who were weaving textiles at home, and this was a very skilled job. It was a great job. You had flexible hours. You could work from your house, and some of the first industrial machines were basically automating this labor. So one machine could do the work of 50 weavers, and the quality would be lower, but it was so much cheaper that you could just outcompete the weavers. The Luddites are understood today as being anti-technology, and it's a pejorative. It means you're not progressive. You don't want change to happen at all. You hate technology. You don't understand technology. The Luddites actually really understood this technology, and they liked some machines that were created. There were certain machines that let them count how many threads were in the textile, and this helped them get paid fairly for doing higher quality work. They also wouldn't break the machines of bosses who paid their workers well and treated their workers well. It was a very strategic uprising against this nascent industrial capitalism, which was leaving people starving to death. The conditions were really bad. You didn't really have a welfare state back then. My big picture understanding of the Industrial Revolution is that it produced a lot of wealth and productivity initially, but that wasn't trickling down to the people who were being automated. You had life expectancy dropping in the places where industrialization was happening first and most intensely, and it really wasn't until the labor movement came about in response to this that those gains actually got distributed to the public widely. This idea that technology just inexorably leads to goods for lots of people is just not true. It has to be paired with movements, and it has to be paired with politics and policies that ensure that those productivity gains, the wealth increases, actually make their way to everybody. Theoretically, you could automate all drivers, and that would be safer and more economically efficient. But if the technology gets there, then you have millions of people who would be put out of work, and people say, "Well, yeah, obviously we'll have to do something about that." But you really have to do something about that. Historically, governments have not been good about this, and I think we just shouldn't make those mistakes again. I'm not opposed to automating specific occupations in a thoughtful way, and then compensating the people who are losing out from this change. So, there are some jobs where we know they can be fully automated. There are dock working jobs which some countries have just fully automated. A few years ago, these American dock workers were basically holding the economy hostage by refusing to process the goods coming in, and because they were unionized, they were able to do this, and people got very mad. Because it's an enormous amount of economic costs being imposed by an interest group. But there were takes from economists who were just saying to pay them a lot of money. This is a job that is now obsolete, but it is bad if people who trained in a trade and developed a bunch of skills and did what the social contract suggests are suddenly just thrown out of a job with maybe some meager kind of welfare. Instead, I think you should make a big one-time payment to these groups of workers if their actual occupation is being rendered obsolete. If you're trying to render everybody obsolete at the same time, or very quickly, that creates very big problems that I don't think anyone has a good answer on how to solve them, and yet that's the status quo right now barreling towards that outcome as fast as possible.

SPENCER: Right. So there's this bigger question historically of what happens when you have technological innovation and automation, and then there's this question of what if it gets condensed so fast that it's not just the weavers or the dock workers, but large swaths of society or everyone all at one time. On that first question, my view, and I wonder if you disagree, is that technological innovation and automation tend to produce more average wealth, and that has been a large driver of economic prosperity. But also, they tend to produce increased inequality at the same time. If it's a situation where the average wealth goes way up and inequality only goes up a little bit, that's a clear win, I would say. If the wealth only goes up a little bit and inequality goes up a lot, that's maybe not a clear win, and it depends on your value system. Different people will differ depending on their value system. Would you agree with that, or would you dispute what I said?

GARRISON: Yeah, I think that inequality matters a lot, not just for fairness reasons, but because of power concentration reasons. In the United States, we've seen an incredible amount of inequality and incredible growth in that inequality to the point where people like Larry Ellison have gotten so wealthy from AI investments that he can personally buy Paramount and CBS News, and then also CNN, HBO, Warner Brothers. One person doing this is, I think, a civilizational level problem for our democratic institutions.

SPENCER: Because it gives them too much power, basically.

GARRISON: Yes, yes, and so this was kind of more theoretical, 10 years ago, Bernie Sanders worried about Jeff Bezos buying The Washington Post, and he got mocked by a lot of people. Obviously, Bezos isn't going to damage this storied brand and institution for his business interests. Then Trump was up for re-election, and Bezos made The Washington Post pull its endorsement of Kamala, which nobody thought was going to tip the election to Kamala. But because Bezos had these massive business interests through Amazon and other companies before the administration, and he knew that Kamala was not going to punish him if she won, but Trump would punish him if Trump won. He doesn't care about this hundreds of millions of dollars worth of newspaper compared to all these other things, and he knows that this can get him in the good graces of this authoritarian administration in waiting. I just think that's first and foremost on my mind. I want people to be wealthier. I want people to have an increase in their material standard of living, and technology and automation have obviously made that possible in many ways. I kind of push back on the idea that we need to just also be increasing inequality at the same time. The United States in its period of greatest growth in the 1950s and 60s had 95% top marginal tax rates and the GI Bill. People would say, "Yeah, the whole world was in shambles and the U.S. was building everything, so not something we can easily recreate," but it's at least possible theoretically. It's been done before to have the economy grow very quickly, have the lower and middle classes being brought up, and wealth inequality not ballooning out of control. I think people care about being the richest person, but the amount of actual money they have is just irrelevant at some point. If you had very aggressive top marginal tax rates and very aggressive estate taxes, you could tamp down inequality a lot while still letting people do the pissing contests of how wealthy they are compared to the next guy.

SPENCER: My perspective is that there's something about technological innovation that tends to produce inequality, but regulation and how society responds to it can dampen the inequality. For example, if you can create startups, startups can be worth billions of dollars. That's just a part of innovation, the way it works. Now, society can say, "Okay, we're going to tax that a lot," or "We're going to split up companies as soon as they get over a certain size," or whatever. There are all kinds of policies they could implement, some of which might sacrifice some of the innovation, some of which might sacrifice it less, but that's a choice that could be layered on top. I didn't mean to imply that innovation can't ameliorate the effects of it. I just mean that I think there's something about innovation that tends to produce inequality.

GARRISON: Yeah, I think I agree with that. You could also look at it as, well, maybe the business can get however big until it gets into antitrust territory, but then the individuals just get taxed so that you might theoretically own tens or hundreds of billions of dollars of this company, but then you have a personal wealth tax to ensure that you're not becoming so rich you can again buy major media institutions and make them do whatever you want.

SPENCER: People are really split on how unfair they think it is that people can become billionaires. Some people say, "Okay, as long as they did it in a way that was beneficial to society, then I'm cool with it." For example, there are professional athletes that have made tons of money, and a lot of times people are like, "Cool, that's awesome! They're excellent at what they do. I don't see any problem with that." Other times, people seem to make money in ways that are much more unsavory. Obviously, people have a big problem with that. But then I would say there are people that say, "No, no matter how you make the money, nobody should have that much money. It's unfair, or it creates a power imbalance, or something like that." I kind of see this as just a values question, and people go different ways on it. What do you think?

GARRISON: Yeah, what is it? Robert Nozick has the famous Wilt Chamberlain thought experiment, where if millions of people pay him 10 cents each to watch him play basketball, he becomes really wealthy. Everyone's happy. No one's really losing. There's no exploitation happening. I think that's the exception. Most billionaires make their fortune off of economic activities that have winners and losers. But I also think that once you have enormous amounts of money, I kind of like the Peter Singer drowning child thought experiment. I think it's pretty hard to avoid. If you are actually impartial about welfare and humans around the world, then having even tens of millions of dollars personally and spending it on yourself is pretty hard to justify under straightforward moral rules and intuitions. If you are Taylor Swift and you make your fortune providing music that people really want to hear and pay money for, that's better than making it selling fossil fuels. But once you have the money, I think the obligation to spend it in ways that help others is kind of the same. I don't think it's tainted or something. It's just a resource that, generally, we should. I fail in this as well. I think it's just a general standard. The wealthier you are, the harder this kind of moral obligation bites in my mind.

SPENCER: I think another key point of disagreement that splits people on this is what produces the average dollar in capitalism. Is it more things that are beneficial or more things that are harmful? We could easily point to cases where companies are doing good. A drug company develops a new drug for a really horrible disease, and they distribute it and offer it at a fair price and make a billion dollars. Fantastic. Everyone wants diseases to be cured. Great, and we can find other cases. Even among drug companies, some fabricate their data and sell a harmful drug. They charge a ridiculous price. That's just one industry, and you can see the gamut of behavior from incredibly beneficial to extremely harmful. But people have different intuitions about what the average dollar is like, and I have a sense that you think the average dollar made in capitalism is not very good, or maybe even net harmful.

GARRISON: Yeah, if you're being epistemically humble, it's very hard to know. I think that it is worse now than it has been in the past. The rise of shareholder capitalism in the 1970s has just been very bad for the world, very bad for the United States. This is an idea from Milton Friedman that businesses don't have an obligation to stakeholders, like their employees or their communities or their country or their suppliers, but instead their only obligation is to maximize wealth for the shareholder. This means that capitalism always has externalities, things that are not priced into the exchange. Pollution is the quintessential example, and shareholder capitalism just makes the externality problem worse. This happened at the same time as industries were being deregulated, and new industries like software were never regulated in the first place. I guess it's a question of do you want there to be capitalism or not? I think we can do better than capitalism and move beyond it to democratic socialism. Even if you don't believe that, I think most people would agree that we can do better than what we currently have, which is regulations being cut left and right. There are things like surveillance pricing, which people hate. I think the number one thing people hate, if you look at polls, is companies using data to charge you a different amount of money based on what they think you'll pay. We just shouldn't let that happen. It's bad. It breeds distrust in institutions, which leads to people being more open to authoritarians. I think it would be a different conversation if we were living in a well-regulated mixed economy with strong labor unions, and the question is what level of social democracy versus democratic socialism is ideal. But we're so far from that that I think we can move pretty definitively in the direction of more regulation, more redistribution, more pre-distribution, stronger labor protections, and not be risking more than we're gaining.

SPENCER: My belief is that the average dollar in capitalism actually is a net benefit to society. However, where I would have a hard split with libertarians on this particular question is that I think there are tons of those dollars that are actually really harmful. There are way too many of them that are harmful compared to what should be. If you look at industries like online gambling, I think tons of people are harmed all the time. You look at multi-level marketing, where I think people are constantly harmed. You look at the supplement industry, where the vast majority of supplements don't work. I just think there's so much money moved to things that are not beneficial and actually harmful. I think we could do a lot better. Even though I do think, on average, the dollars are doing good, and I think that's part of the reason why. If you compare the world now to 200 years ago, people really are genuinely much wealthier in a real and meaningful sense. If the average dollar actually did net harm, I think we wouldn't be in this world.

GARRISON: Yeah, and I want to revise something I said earlier, which is you can actually have your position that it's net good, the average dollar in capitalism, and still prefer different economic systems, which you think would produce even better net dollars. I think that sometimes people on the left want to bite off this really big claim, which is, "Capitalism is the worst thing we could have," and I think it's better than feudalism. I think it's better than mercantilism. Karl Marx was very admiring of capitalism's productive capabilities, and he was like, okay, we could do better than this. I'm not a Marxist, but I think that thinking of it as moving beyond capitalism, improving on the things that it's failing at, and just the belief that we can do better.

SPENCER: Yeah, that makes sense. I think if we think about the way the dollars get made, if you think about pure economic theory, it says, "Look, you've got a rational actor, another rational actor. The rational actor on one side wants to buy the thing. The rational actor on the other side wants to sell the thing. It's a mutually beneficial transaction." When people think that way, you immediately have to ask the question: What if we're not dealing with rational actors? I think that's where a ton of the problem comes into play. Either you're not dealing with rational actors because none of us humans are rational, or you have bad information, or asymmetries that kind of screw it up. That's why I think a lot of the most harmful transactions that occur are online gambling; it's addictive. Multi-level marketing uses manipulative tactics to get people, manipulative persuasion. So many of these industries, I think, where harms are created, it's because it's essentially exploiting biases of different forms. You also have externalities where maybe the person doing the transaction isn't the one paying the cost. I wouldn't exactly call that individual irrationality, but maybe that's a large-scale irrationality that's getting exploited.

GARRISON: Yeah, I agree with that, and I think people underrate how ubiquitous these asymmetries can be. I'm a worker, I'm entering into a free contract with an employer to sell my labor at a price that I think is fair, this kind of libertarian ideal. The employers are smaller, more concentrated in many ways, more powerful by default than the workers, which is why I think collective bargaining, union, legal regimes that allow people to unionize are beneficial because it tilts this natural asymmetry a bit. Similarly, there are ideas with our data. We can theoretically opt out of giving our data to tech companies, and the European regulations are supposed to help with this, but all we get is the stupid cookie banner. One of the ideas I talk about in the book is this idea of owning your own data, and then also data unions, which takes that idea and applies the collective framework. Individually, you can withhold your data from a company, and if it's Reddit or Google or whatever, they don't care that much. But if whole groups of people can do this, they have a much better negotiating position for either being compensated or imposing rules on how the data is used. That's kind of this big picture idea in my worldview, which is we have to look at the actual structure of the economy and society, and be clear-eyed about where power currently exists.

SPENCER: So, is the idea that you would kind of enter willingly into a group saying you can sort of have control over my data and negotiate on my behalf along with all the other people you represent?

GARRISON: Yeah, yeah. And this is from Jaron Lanier, I think. I haven't, you know, this is not a paragraph in the book. It's not an idea that I'm claiming ownership of in any way. But I think it's just an interesting application of this general principle.

SPENCER: So let's go back to the idea of these universal labor-replacing machines.

GARRISON: Rolls off the tongue.

SPENCER: Yeah, you might need an acronym or something. But suppose we have not just a bunch of jobs getting replaced at some reasonable rate, but actually we have 50% of all jobs in society disappearing over a period of a few years. What do you predict would actually happen in that case?

GARRISON: I mean pandemonium, very, very strong immune response from society. The pollster David Shor said something, "If AI caused unemployment to spike by 2 or 3%, it would be the number one political issue in the country." So, yeah, I think there'd be a really strong negative response from the public and from politicians. Economic models show that in that scenario, output kind of goes to the moon while wages plummet, and there would still be some jobs. There would be jobs in the physical world. There would be relational jobs where we care or will pay a premium to have a real human on the other end of it. Maybe you could transition the entire economy into those types of jobs, but it would be pretty bumpy and take some time. It's not clear that you could build the entire economy around that, and so you would expect economic growth to go up a lot, but then wealth concentration would become even more extreme. This isn't an aberration. If you look at wealth inequality and corporate valuations and valuation per employee, all of these things are going up in really intense ways. In the book, I have examples of the most valuable companies per employee throughout history, which is on the order of $10 million per employee, and then you have NVIDIA, and it's over $100 million per employee, and then you have the AI companies themselves, and it's well over that. People talk about a technological singularity where AI can improve itself better than we can improve it, and then we don't know what happens after that. But we're kind of approaching this economic singularity, where at the end of it, there's one person who controls all of the wealth. I don't think that will literally happen, but we're trending in that direction, and disempowering people by devaluing their labor power will have really negative effects on their political power as well. The proposals people have of a universal basic income just take all those people and make them dependent on the state, and the state dependent on these AI companies, which are already very powerful. The heads of these companies are seated with the heads of the G7 countries, with a picture of Dario Amodei and Sam Altman with the heads of the most powerful countries in the world seated at the same table. I think that's just a pretty concerning outcome for basically everybody. I've never seen anyone have a really good plan of how this goes well.

SPENCER: When you are able to control AIs that do human labor, you could imagine one entity or even one person, but probably more likely one company controlling millions or billions of simulated workers or even trillions, and then you start to see how power could concentrate and wealth could concentrate in a way we've never seen before. We've never had a company that had effectively a trillion workers. Not only that, they don't have to pay the workers; they just have to pay the computation that runs the workers. So that's kind of a wild idea.

GARRISON: Yeah, and I think we're assuming that these universal labor-replacing machines are actually doing what their developers and users tell them to do, which is not something we can assume. There are different, in some ways, scarier problems that arise from that. Another thing to say is that in this world, there will just be so much more work being done. Some of the benefits would be deflation in the price of goods and services because labor automation would make them so much cheaper to provide. The upside case is that everyone has radical abundance because human workers are a hundred times more expensive than the automated equivalent, but there would also still be things like housing that would not necessarily go up in supply or not quickly enough to meet people's demands, and things like healthcare and education, which have been historically going up far faster than inflation in the United States. I think that's what's driving people's deep sense of feeling precarious and affordability being such a big concern in the U.S. On those things, it's not clear how automating white-collar jobs is going to help there, and there are a bunch of stories you could tell where it actually makes that worse. In San Francisco, you can kind of see a preview of this, where there was already a housing crisis, and then these AI companies are becoming so valuable, making so many people so rich that it's even harder to find an apartment or a house there, and they're getting even more expensive. That might just be a preview of the economy of the future, which is increasingly a two-track one, where the people who own the machines or the companies that make the machines become incredibly unfathomably rich, while everybody else is struggling even more potentially.

SPENCER: So we haven't yet delved into the second narrative. We've only talked about the worriers. Let's talk about the critics. What's their general point of view? I'm interested to hear where you agree with them and where you diverge.

GARRISON: Yeah, the critics — obviously I'm going to generalize here a bit — these are people who tend to argue that AI just doesn't work. It's hitting a wall. It will always hallucinate. It will never be reliable enough to be useful in high-stakes situations, and it's being misapplied in a bunch of ways. You have self-driving cars, which fail to recognize black people as often as white people, and that's an example of algorithmic bias because of the data that they're being trained on. The critics, what I think they get right is they are correct to look at the kind of bad economic incentives of these companies and the blind spots of their workforces, and they are right to be questioning whether we should be applying AI in every situation. They tend to be better at organizing or more open to doing policy change than at least the worriers were historically, and then the boosters just don't want any regulation at all. I think what they get wrong is that they're often in denial about what the technology can already do, and definitely in denial about what it might be able to do. I'm not saying that it will definitely be able to fully automate labor, or recursively self-improve, or become superintelligent. I'm just arguing that we cannot rule out that possibility, and there's just massive asymmetry in getting it wrong. In one world, maybe you do some actions that were costly, but you didn't have to do, but ultimately it's not the end of the world. The other option is actually maybe the end of the world. I also tend to think that my policy proposal is something that people who are critics of this technology should also want. If we halted development towards AGI, towards universal labor-replacing machines, then that would mean the technology has fewer of the current problems that we're seeing with it as well, and it would also mean the industry has less power, and the critics are very skeptical of the industry and their motivations. I think that's a reasonable position to have.

SPENCER: Something I find so strange in the time we're living in now is that I sometimes meet people who have no idea about AI at all. I was talking to someone at a party, and they told me they literally never used AI. I was like, "Well, do you use Google?" And they're like, "Oh yeah, I use Google." And that has AI, and they're like, "Has AI in it?" It's not that they're anti-AI even; they're just completely not interested.

GARRISON: They're agnostic or something.

SPENCER: Yeah, "Why would I ever use it? It's just... whatever." Then I meet people who are absolutely convinced AI is useless and all it does is generate total crap, and then I want to be like, "Okay, can I show you it writing code and how it can write code that you can verify and it works?" It doesn't work every time, and yeah, it sometimes has bugs, but then you can do things to check it and confirm it works, and then holy shit, it's 10 or sometimes 50 times faster than a normal programmer, and they seem to be in complete denial. It's almost shocking how unaware they are of the capabilities, especially given that they hate it so much. You think they might monitor it more carefully. An example of that is I saw someone who was saying, "Oh yeah, I tried AI and it's total bullshit. I asked it to sort this list of numbers, and it sorted them wrong." I was just thinking, "Man, this person probably just threw it into the cheapest, you know, free model, whatever."

GARRISON: Yeah,

SPENCER: And now they just totally wrote off the whole technology. Sometimes that comes about because maybe they tested it two years ago, and maybe it really wasn't nearly as good then. Or maybe they just heard things about crazy mistakes it's made, and they focus on those. Then you have people who are just trying to incorporate it in every aspect of life as hard as they can to do everything, and they're just going full bore, you know, AI. It's fascinating to see these completely different perspectives people are having.

GARRISON: Totally, and I think you hit on something important, which is that a lot of skeptics will find an example of AI doing something stupid, like counting the wrong number of letters in the word strawberry, the wrong number of R's, and finding dumb examples of AI outputs being incredibly stupid or funny. It's not that AI doesn't make mistakes; it totally does, and even the very best models will still make mistakes that humans would never make. But I think we should be looking at the frontier of capabilities. We should be asking, "How good are the models at doing things when they're done by the very best models with the right prompting and harnessing?" We should be paying a lot of attention to that. It's actually quite hard if you're not using the models regularly to get a sense of what that is, and people talk about this idea of a capabilities overhang, where the frontier is just well beyond what people understand it to be. The OpenAI-Hugging Face hacks are an incredible example of this, where maybe a normal person is using AI outputs, the free version of Gemini or ChatGPT, and it's not that good. Internally, OpenAI has these models, one of which is not even released and is better than anything they have publicly, with all these safeguards turned off. They're hacking their way out of OpenAI, improving each other's capabilities by trading notes on how to hack OpenAI's servers and hack into other things. They hack into multiple other companies, all the while finding these exploits that some of the very best human cybersecurity experts and the AIs they're using missed, doing all of this without any human involvement or awareness for days and weeks on end. Then you see people in the skeptic, critic camp describing this, and they're saying, "Oh, people say the AIs went rogue or OpenAI lost control of them," but that's obviously ridiculous. But who was controlling them? Maybe OpenAI screwed up in not securing their models better, but it's still true that they went rogue and they lost control of them, and no one's been able to explain to me how that's not the case. I think any normal person, if presented with facts in a fair way, would say, "Yeah, they went rogue, and OpenAI lost control."

SPENCER: I wonder whether the critical perspective is going to face a rapid shift as it becomes increasingly difficult to view these models as incapable and useless. When that happens, is the group going to shift and say, okay, now we have a different viewpoint, and now we think they're capable but bad, or are we going to get a sort of deranged situation where they are increasingly in denial as the models get better and better? The denial has to get greater and greater, which would be really disturbing. I hope that doesn't happen.

GARRISON: Yeah, encouragingly, I've seen a number of people, basically in response to the Hugging Face hacks, say, "You know, I was wrong. I think that this AI risk thing is more real now."

SPENCER: Encouraging.

GARRISON: Yeah, whenever I see that happening, I think, "Great, that's fantastic." I think it's important that we be gracious and encourage such behavior. This will be dated by the time the episode comes out, but just today, Nate Soares, who co-authored If Anyone Builds It, Everyone Dies, published an op-ed in The New York Times about the Hugging Face hacks. There were a thousand comments already, and the overwhelming response was, "This is very real. We should take this super seriously." I read _The New York Times comments on AI topics, and I think, "Holy crap! I've never seen people be so accepting that AI capabilities and risks are real." There's a lot of denial and dismissal from The New York Times, The Guardian, and these liberal and center-left, well-educated readers, and you're seeing more of that happening. There are still people who will deny it right up until we all get paperclipped or something. I think that's in a lot of cases because there's audience capture; they've staked their entire career on saying, "AI doesn't work," and their audience just wants that to keep happening. But I think the audience is actually more swayable in a lot of cases, and people have not really made the argument that AI is both quite capable and getting more so, and the industry is incredibly reckless, and we should stop them. Obviously, Yudkowsky is arguing this to some extent, but I think just the way he frames the arguments and his general political sensibilities are not going to be embraced by people on the left. These companies, in some sense, are impressive, but they're also getting very lucky that they were around when AI chips got good enough, when deep learning had enough data and computing power to start working. They're really building a lot on work that was done decades ago by academic researchers funded by governments, and they're taking the entire work of humanity without compensation, at least initially.

SPENCER: Training the models.

GARRISON: Yeah, training the models, and it's possible to just make the models bigger. They'll keep getting better. There are very competent people working hard at figuring out clever ways to make the models better and cheaper, but it doesn't make them wise people or people we should defer to in any way. I think it's fair to ask, "Why are we taking Dario Amodei's advice on how to handle geopolitics?" He doesn't have the relevant experience here, and why are we empowering these people who thought that AI posed an extinction risk to humanity and then chose to build it themselves? We should be incredibly skeptical of them because they're either power-seeking or assured in their own self-righteousness or virtue that they chose to do this thing that they think could kill everybody, and now they're getting rewarded by being at the same table as G7 leaders. These people should not be in charge of this technology; the public broadly should. It's baffling to me that we're letting this happen. A big part of that is because so many people on the left have been denying the threat for so long.

SPENCER: You sometimes hear the view that the leaders of top AI companies are exaggerating this threat to humanity as a marketing tactic. I don't think that's true. I'm not going to say it's never true, but I actually think, in broad strokes, they are worried about their own technology. I'm curious what you think.

GARRISON: I looked into all the statements I could find from the CEOs of the major tech and AI companies, and the interesting thing I found is that people like Sam Altman, Dario Amodei, and Demis Hassabis, who have been interested in AI and AGI for a very long time, have said that it's an existential risk as long as they've been on record talking about AI. It's the CEOs of the big companies like Amazon, Microsoft, Google, and Facebook who are in denial that it's an existential risk.

SPENCER: It's just the opposite of what you think, in a way, if it was just a promotional tactic, the CEOs would be using it too, wouldn't they?

GARRISON: Yeah, exactly. And it's bad for business. There's something to the argument. It actually is focusing people on how capable the AI might get in the future, and that might be tantalizing to some investor or customer. I think you can't totally dismiss that possibility, and some people are cynically using it. But it's also been interesting as OpenAI became more successful after ChatGPT. Sam Altman edged away from talking about existential risk and job loss. There were reports recently that investors or prospective investors were like, "Hey, can you stop talking about the doomsday scenarios or job loss? It's bad for business."

SPENCER: Stop talking about how your business might kill all humanity.

GARRISON: Yeah, because politicians are like, "Wait, what? You think it could have a 25% chance? Are you kidding me?"

SPENCER: Yeah, that's pretty interesting. A related idea you hear from the critics' camp is that AI is in this massive bubble. My perspective is that you could simultaneously have two things be true. It could be true that AI is incredibly powerful and getting more powerful, and it could also theoretically be in a bubble at the same time. They're not actually mutually exclusive. The reason being just that there's a timing issue. You could have it be that it's just not profitable for long enough that eventually the stock market's like, "Forget this. We're not going to invest; it's not bringing enough revenue." You know it's not bringing enough profit, and the thing could collapse. Even if capabilities are really powerful and getting more powerful, especially in a world of lots of competition, where if companies have monopolies and lots of pricing power, then maybe as it gets more powerful, they automatically can make more money. But if they're competing with each other on price, that's not necessarily so. I don't know whether AI is in a bubble or not. I think it depends on how quickly the capabilities grow and how quickly that turns into profits. So, yeah, I'm curious your reaction to that.

GARRISON: Yeah, the second chapter of the book is called Is AI a Bubble, and I dig into this quite a bit. Big picture, I agree with you; the technology being real and potentially very dangerous does not mean that the business models will work out, or at least that they'll work out on the first big swing at it.

SPENCER: The internet was a bubble, and then look at the internet now.

GARRISON: Tech companies are worth so much more than at the peak of the dot-com bubble. I think the people who say it's a bubble and therefore we should just wait it out or pop the bubble, or something, think it'll all be fine. It's not going away. People really like using generative AI; they don't like it being forced upon them through other software. But user adoption and revenue growth have been unprecedented. Literally, the fastest growing companies by revenue and users in history are OpenAI and Anthropic, and the stickiness you can look at is user retention. It's incredible, unheard of levels of retention. So, the bubble is popping. Maybe they are not able to raise the money required to keep making the models better, but it's not clear why people would stop using the models. They'll get cheaper over time because the companies have consistently found ways to make the models cheaper at better performance levels. GPT-4, when it came out, cost something like $37 to process a million tokens. The best models today cost significantly less than that and are so much better than GPT-4. People forget how poor GPT-4 was, but we were so impressed back then. If OpenAI and Anthropic can't keep raising money, they can't keep training new models. I don't think Claude and ChatGPT are going to stop being used by companies and people. We're still going to be dealing with the effects of this technology, no matter what. I think it's a pretty big distraction, and I think it's a comforting story to believe that this will all blow over soon and we don't have to do anything about it. Some people say, "Yeah, we should regulate it or do something to pop the bubble," but you're still going to have to deal with this problem, and that's going to require getting off the couch, getting organized, and doing a movement to get the kinds of laws we need to make the technology actually work for us.

SPENCER: Do you have a view of whether it is an economic bubble separate from the power of technology? Is it completely unsustainable right now, or not necessarily?

GARRISON: Yeah, I think that you know what a bubble is. There's not a fixed definition, and people have different criteria for it. Some of the things that make it bubbly are the enormous amount of money going into it, which means you have to have really high revenue and profit in the future to get a return on the investment. You need to have more user and revenue growth. That growth has been predicated on the models continuing to get better. OpenAI and Anthropic make better and better models for increasing amounts of money, and then they can charge a premium while the model is the best. They have to keep doing this. The companies are not profitable because it costs so much money to train new models. Their way out of that is to grow their user base and revenue enough to cover the cost of training new models and the cost of serving the models.

SPENCER: The cost of training the models has gone up exponentially, right?

GARRISON: I think that's right; it keeps going up, and the way you get out of it is if you can grow the revenue faster than the costs of training new models, which has been what's happening with OpenAI and Anthropic. Another piece of it that could make it a bubble is the circular nature of a lot of these investment deals, where NVIDIA will invest a hundred billion dollars or something in OpenAI, and then OpenAI has to buy a bunch of NVIDIA chips with the money, and then the valuation of both companies goes up by more than the actual dollar amount invested. Now there's a lot of correlated risk, and one of the biggest reasons the financial crisis happened was because a bunch of risks were correlated that were not treated as being correlated. Similarly, if OpenAI can't keep growing and meeting its compute deals with these companies, then the whole thing could have cascading negative effects on what are now the biggest companies in the entire world. That has a bunch of bad effects on the whole world economy. Another piece of bubbles is debt. The dot-com bubble was financed with something like $2 trillion in debt in today's dollars, which obviously makes it a lot worse. If you can't meet your investments and have all this debt to deal with, this build-out of data centers has been financed overwhelmingly by profits of the most profitable companies in history. The hyperscalers are all basically monopolies in very lucrative markets.

SPENCER: Like Google?

GARRISON: Microsoft, Facebook, Amazon. And they're starting to take on some debt. Meta has been putting some of their data center investments in special debt vehicles because they don't want to hurt their balance sheet and make their stock perform worse, and that obscures some of the risk from the market. That's worrying. The main reason that it's not as likely to be a bubble, in my mind, is that if you look at the arguments for it being a bubble, one of the biggest ones is that these companies are losing money on every customer using their AI, and that's just not true. The average customer is profitable for OpenAI and Anthropic, at least the paying ones, and we know this because of reporting in a bunch of different venues. But also, the financial statements of OpenAI have actually been leaked, and you can see what they've spent and what they're making.

SPENCER: Paid customers, they make money on average, even if they lose on some of the paid customers. They're the heavy users. That's the idea.

GARRISON: Yeah, right. So OpenAI, even when you include all of the free users of ChatGPT, is making more money serving it than they're losing serving all those customers and their paid.

SPENCER: So free plus paid, they actually make money on average.

GARRISON: And so that's okay, that looks pretty impressive actually. Software companies have basically zero marginal costs to sell another copy of Microsoft Word. So that's a great business model. You invest a lot in making software that people want, billions of copies of it, and then it costs nothing to get a new customer. AI famously costs something to get each customer for every prompt, for every token, but it's closer to zero than not. That's the main reason why these companies could become the most valuable companies in history because they have similar cost structures to software and the total addressable market is kind of everything. It's all white-collar jobs.

SPENCER: Your robots are blue-collar jobs too.

GARRISON: Exactly. Yeah. Your revenue could be so large, and your profit margins won't be as good as Google's ad margins, but they're still way better than a restaurant or something. You put that together, and it's like, "Oh, okay, yeah, Anthropic could be worth $10 trillion in a few years." But again, this is predicated on continuing to grow and people being willing to pay a premium for the best models, for the models continuing to get better. That's been true for a lot of years in a row, but it may not be true forever. The stakes of getting it wrong go up because you spend a billion dollars on training a model, and if you spend 10 billion and it's better, then 100 billion and it's not better it's like, "Well, now you have a hundred billion dollars that you have to write off or recoup in some way, but you don't have a new better model to do it with."

SPENCER: People have also made arguments. I don't know if you investigated this, but there's something around if you're having to buy chips, you're put in a weird position because when the next model comes out, the old chips you have are worth a lot less. Some people make the argument there's an unsustainability, and that these data centers are essentially constantly getting deprecated in value.

GARRISON: Yeah, right. So some people have likened chips to bananas, where they lose value very quickly, and that would be pretty bad. If you invested a lot of money in something that became essentially worthless after a short period of time, but this has also not been what's happened. The chips do depreciate a bit over time because new ones come online and you have better and more efficient chips, but actually, when you look at the prices of the standard NVIDIA chips that are used for a lot of AI inference, they're selling for a higher hourly rental price now than they were two or three years ago. It's a bit less than when they were first installed, but it's gone up. So it's like, "Wait, how does that make any sense?" It's basically this idea of tokenomics. Each token, you can get more value out of it as the models get more capable. GPT-4, the tokens that came out of it were just not that useful because it couldn't do that much. But once you start getting better at writing software, each token that comes out of it helps you make some valuable piece of software, and even though the chips are depreciating and better ones are coming online, you can get more value out of each token, and that can explain it. Maybe that doesn't keep happening, but so far it has. There was this example in the Financial Times where this columnist looked at the prices of chips and did a bunch of math and was like, "Oh my god, you'd have to sell these chips, I don't know, he found they were incredibly unprofitable to buy these NVIDIA chips," and then somebody pointed out that he missed that the product he was looking at was actually eight chips bundled together in one rack, and so it went from being, "Oh man, it's barely breaking even to like eight times that." The margins were incredible, and he doubled down and thought about it. But it's like, "Dude, you screwed up. You made this look like a way better business." This was in the Financial Times, so there's a lot of basically false information floating around about the bubble narrative.

SPENCER: I've also heard about companies that are trying to get their employees to use AI as much as possible, and one, I think it was some CTO, was talking about how he blew through his entire token budget for the year in two months by accident because it's just, "Oh yeah, let's just max the tokens, you know?"

GARRISON: Yeah, and that's not to say that all the adoption is going to be valuable for businesses, or that there's not frothiness and overexcitement. Token maxing was obviously stupid. The best books aren't just the longest books. The best software is not just the most lines of code. That was dumb. But I also feel like those stories just get overplayed. There's this appetite for the bubble narrative that every example of this is going to get amplified like crazy, and then that gets generalized to every single business overspent on AI services, and now they're going to tighten their belts and spend less than they were before they started token maxing, and I'm like probably not.

SPENCER: This is a funny analogy, but it reminds me of critiques of Burning Man. If something goes wrong at Burning Man, people just eat it up. They love to see things go wrong at Burning Man. Mudpocalypse, when it got all muddy, there were articles about how it's a disaster and people are getting diseases, and it's like no, it's just muddy.

GARRISON: Yeah.

SPENCER: But it's just the appetite for seeing these people come up and suffer..

GARRISON: ...Off the burners, yeah.

SPENCER: Yeah, exactly. Another critique from the critics comes out around water usage and energy usage, and then there's been a counter critique saying, "Okay, this is really overblown." Where do you land on that?

GARRISON: Yeah, the energy usage piece is real. These data centers use a lot of energy, and you have examples of these tech companies blowing through their climate commitments as they build these data centers. You have certain cases like xAI in Memphis, Tennessee, using gas turbines and allegedly violating environmental law and polluting the local area to get their data center up as quickly as possible. I looked into the total amount, and I think AI was responsible for 0.5% of global greenhouse emissions, and that was projected to grow to 1.5% in I don't remember how many years, but relatively soon. That's a significant amount, but people are spending a lot of time using this technology, and it often has localized effects that people are really reacting to, where the rates might go up in their area if the deals aren't structured in the right way for data centers. But there are also examples of data centers upgrading the grids and bringing rates down for people. It really depends on how you structure it, what the energy mix is of the energy the data center is using. Because they need sustained amounts of power, renewables are harder there. But there are also nuclear plants coming online, so it's complicated. The water picture is more straightforwardly overblown. Data centers do use water for cooling, and there's a lot of effort to use less water. The stories you see on this often leave out the denominator. They'll say, "Oh, this data center is using 500 million gallons of water over whatever period of time," and it's like, "You know a golf course uses the same amount, and the data center might be providing YouTube and AI services for millions of people, whereas a golf course is not providing something."

SPENCER: Growing almonds famously uses a ridiculous amount of water. You don't really think about it.

GARRISON: Yeah, and a lot of people are just like, "I don't like AI. I think it should use no energy. It should use no water. So all of this is bad." But let's separate things out a bit. If you really want to lobby against AI or the AI that these companies are making, we should focus on the stronger arguments. Imagine they come up with ways to make data centers that use no water, produce no sound, and have their own little fusion reactors providing energy for everybody. Would the people who oppose data centers on AI grounds still oppose them? Yeah, but they wouldn't have arguments. You should focus on the fact that the industry is trying to render us obsolete and doing it without anyone's consent. That's where I come down on.

SPENCER: Before we wrap up, let's get to the third narrative, the one we didn't discuss yet: the boosters. So, what do the boosters believe, and where do you diverge from them?

GARRISON: Yeah, so the boosters are, in some ways, a less coherent group of people. You have everyone from Marc Andreessen, who I mentioned, being full speed ahead and saying, "We should build as much AI as possible, and anyone who gets in my way is a murderer," to the companies themselves, these big tech companies, which de facto just want minimal or no regulation and to move as quickly as they can. The case is just that AI has overwhelmingly positive effects on the world. It makes people's jobs easier and produces all these productivity gains. It will create all this wealth and abundance. The most optimistic case you hear from some of the AI company CEOs, like Dario Amodei and Demis Hassabis, is about this sci-fi series called the Culture series by Iain Banks, who is a Scottish socialist that imagined a world thousands of years in the future, where you have posthumans living in libertarian socialist societies governed by superintelligent sentient minds that run everything, and humans get to do whatever they want. They have incredible amounts of abundance and personal autonomy, and technology and health and drugs and whatever. In many ways, that's an improvement. That's kind of the fully automated luxury communism vision of the technology, but it's notable that even in that world, the humans are not calling the shots. These CEOs are basically saying we should cede our control of the planet to these machines that will run it better than us. This is not to say we should never do that, but doing it as fast as possible without buy-in from people feels like the wrong way to do it.

SPENCER: Do you think they're actually imagining there aren't humans at the top of that chain?

GARRISON: I'm just talking about the books that they're talking about. They might say something if they were challenged, but I also think it's not fair to speculate about the contents of their minds. I think they'd believe the machines would do a much better job if we gave them the right instructions and really understood alignment at a deep level. The way I make the case for it in the book is to imagine how AI could be better as a surgeon, like a robot surgeon, or as a driver, if they just become superhuman at doing it better in every possible way. If you apply that to every conceivable domain of society, it could feel like resisting letting the machines run the show, like insisting that an eight-year-old dock a cruise ship when you have an experienced captain that could do it instead. But that's an incredibly optimistic world and assumes that all of these really gnarly problems around safety and democracy are solved, and I think we're just not on track to solving so many of those problems.

SPENCER: Do you agree that AI could make the world much better?

GARRISON: Yeah, I think that I make the case for a unifying umbrella term I call AI reform, which is taking the best of each of the camps and leaving out some of the baggage of them. The idea is we should stop the race to replace us. We should prevent these companies from building the universal labor-replacing machines until it can be done safely and with strong public buy-in, and then we should build reformed AI. What that looks like, I don't know right now. But it's kind of like asking this question of, in light of what the technology can do now and what it might be able to do in the future, how do we democratically build a future worth inhabiting? That is a process. It's also a call for a movement to turn that kind of question into an organization that can make lasting changes in policy. To be a bit more concrete, Google DeepMind produced this AI model called AlphaFold, which solved the protein folding problem, a long-standing grand challenge in biology. Demis Hassabis and John Jumper won the Nobel Prize in Chemistry for it. It used to cost hundreds of thousands of dollars in some cases and years of time to figure out how one protein folded, and this model just figured out all the rest that we knew about basically instantly. I think that we can have more things like that, but it's not going to happen by default. Recently, Google shut down the AlphaFold team, and a bunch of people who were on it have left, including Demis, and they're redeploying those resources toward making Gemini better. We should be using industrial policy and regulation and subsidy to find more things like AlphaFold, invest in those, and not invest in the put-everyone-out-of-work and/or kill-everybody machines. I think we could have way better AI than we have today, and it would be AI where we think collectively about what it is we actually want from this, which is not what the companies are offering us.

SPENCER: I'm especially excited about AI for medicine, where I think that you could get a situation where nearly anyone in the world, for essentially no cost, could get pretty sound medical advice. Of course, we're not there yet. You cannot replace your doctor with an AI. But I think even today, AI can be useful in helping you research topics around medical care and then go talk to your doctor about what you found. I also think it could be really exciting around personalized education. The dream of every student in the world having a tutor on whatever subject they want. I think we're pretty close to AI being good enough for that, especially with the right controls around them. What do you think of those use cases?

GARRISON: Yeah, I think we should be careful with both of those things. The stakes are pretty high.

SPENCER: Risks for sure. Yeah.

GARRISON: But I think it's obviously true that we've probably all had the experience of going to the doctor. They give us very little attention. They give us some results, but don't really explain it very well. And you're like, "All right, so what do I do with this?" The models have been good enough for a long time to explain your medical chart to you in a way that's more helpful and intuitive. This is not to say they'll never make mistakes, and I'm not saying we can fully replace doctors with LLMs. But I think there's potential there. Education, I haven't looked into it in great detail, but I know there are these Alpha Schools which purport to do the kind of thing you're talking about. I would be worried about the kind of economic incentives of profit-seeking companies building addictive AI tutors. You will always need socialization. I think friction is just important in being a human being. What tech companies and AI companies, in particular, offer is this promise of a frictionless life, where you don't have to think about the email you're sending to your boss, or you don't have to bang your head against a problem until you understand it. You can just get the answer, and we're seeing a lot of enfeeblement and this term I love, cognitive surrender. People lean on these tools too much, and they just stop learning or stop being able to do things they once could do. This has always been a concern about writing and calculators. There have been moral panics, but it does feel different this time around, and people do want sycophancy. They do want easy answers, in some sense. Upon reflection, with some kind of deliberative process, they would not want that on a societal scale. We should not just do the thing that is rewarded by the market, or just because people like those products doesn't mean we should pursue them.

SPENCER: Before we finish, what do you want to make sure people remember from this conversation? Do you have any actual takeaways that you want people to do?

GARRISON: I think a lot of the people who have looked most deeply at the problem of AI existential risk have overcomplicated it. They're like, "Well, we can't just stop it because it's inevitable that people will build superintelligence," or whatever, and it would be undesirable to stop it because you'd lose out on this glorious transhumanist future of cured diseases, solved aging, happy people all the time, minds uploaded in the cloud, really sci-fi stuff. Therefore, we have to thread this needle of slowing down AI just enough to solve alignment, and then use that to create a superintelligence and ask it what we should do if we knew better. They're like, "Man, it's really hard. If only we knew what to do." I think it's actually kind of straightforward. We should not let these companies build AGI, the universal labor-replacing machines, and there are actual policy levers to do this, which we didn't get into very much here. The computer they need is very specialized. It's made by a very small number of monopolistic companies. There are only two countries and a handful of companies in the world that have any shot at doing this. Everyone knows which companies they are, and if the United States told the AI industry to stop building AGI, they would do that, and they could do that. China could obviously do that as well. The trick is making a verifiable international agreement. There's a lot of technical work that's been done on this. There's more work that's needed, and there's institutional work, and obviously a lot of diplomacy and political will that's needed to make this happen. But it is doable, and I want people to realize that you can have a clear demand, which is don't build AGI or stop the race to replace us, and enough mobilization to make that demand have weight behind it. The political system can figure out how to actually implement the results. A lot of the AI safety people focus on the how, and they don't really focus on the what or the why. One of the big interventions I want this book to have is giving people a what and a why, with enough of a how that they can believe it's possible. This approach will get a lot of people on board who currently are skeptical or focused on other things because you can show that their priorities would also be met by this policy demand. The industry is creating something that they're terrified of, that many people are terrified of for good reason. Many of us aren't happy with how things are going right now. The industry is almost begging to be stopped. They're like, "Oh well, I can't stop because somebody else will do it," and that's why we need the public to become a player in this. It's not just a choice between concentrating all the power in the hands of the government or these companies. If the public's a player, we can say, "No, neither of you are going to have the superintelligence that you don't know how to control, and if you did know how to control it, you could use it to put everyone out of work, surveil everybody perfectly, or dominate your international or domestic opponents." No, nobody gets that responsibility. Nobody can handle that. If we get our act together as a species, maybe in the future we can have some AGI as a treat.

GARRISON: It should be a thing that we're leveling up our institutional capacity, our democratic governance to get to that point where we can handle the gargantuan responsibility of building machines that can replace all human labor. Then we could have a species-wide retirement or something, but one that we are making an informed and global consensual agreement to pursue, whereas right now it's full speed ahead. Ask for forgiveness, not permission. It's becoming increasingly clear that these companies cannot handle the AIs that they already have and definitely won't be able to handle the AIs that they're trying to build.

GARRISON: Yeah, it's been super fun. Thanks for having me. The book is called Obsolete: The AI Industry's Trillion Dollar Race to Replace Us and How to Stop It.

SPENCER: We'll put a link in the show notes.

SPENCER: Garrison, thanks so much for coming on the Clearer Thinking podcast.