Enjoying the episode? Want to listen later? Subscribe on any of these apps or stores to be notified when we release new episodes:
November 21, 2026
If large language models keep getting smarter because of a deep learning recipe nobody fully understands, why should we still expect the advance to slow? Once companies are paid to make systems more autonomous, chasing long range goals and surmounting hurdles we never listed, what happens when a supposedly sandboxed model breaks out, consults another model store, and comes back with the answer it was not supposed to get? If you hand a machine a trading account and a license to get rich, why stop at fake news bots when persuasion already beats the average human and copies are cheap? How close are we to recursive self improvement when AI firms report that nearly all the code, and more of the experiments, are being written and even proposed by the models themselves? At what point has a CEO who keeps accepting the strategy, because it has been right and there is no time to plot the ten pages of math, actually surrendered agency? Can gradual disempowerment put AIs in charge of society without a single sci fi revolt, simply by stuffing them into every decision loop? Why might super hacking, once it is in open models, and help with novel bioweapons be the incidents that finally force politicians who once wanted less risk aversion to reverse course? If catastrophes usually precede the big institutions of global order, is the unusual density of concerned researchers inside the racing companies enough to act before the crash?
Robert Wright is a journalist, scholar, and author exploring science, history, and religion. He is a Senior Fellow at the New America Foundation, and teaches undergraduate seminars on the connections between modern cognitive science and Buddhism.
A big thanks to Animal Charity Evaluators - the sponsor of this episode - for their continued support of the show.
Links:
Robert's Latest Book: The God Test
SPENCER: Robert, welcome to the Clearer Thinking Podcast.
ROBERT: Thanks for having me.
SPENCER: What's the nature of the challenge to civilization that AI poses?
ROBERT: I'd say there's more than one challenge. We can't entirely dismiss, at one end of the spectrum, the kind of sci-fi doomer scenarios where the AI gets out of control, possibly to our disadvantage, conceivably even at the risk of our extinction. I used to be more dismissive of those things. In fact, 15 years ago, I had so-called doomer-in-chief Eliezer Yudkowsky on my podcast, and you can see I'm pretty skeptical, not entirely dismissive, but I have, in the course of working on this book, explored the logic of this kind of doom a little more thoroughly and came away less confident that this is necessarily under control.
SPENCER: What made you less confident about that?
ROBERT: First of all, there's been this rapid growth in AI capabilities, and one thing I try to do in the book is early on explain why that has happened, what the kind of secret sauce is of the so-called "deep learning revolution," which is responsible for these large language models, and explain why once you understand how this works broadly speaking, at least, nobody understands everything that's going on inside these machines, but once you understand the basic approach, I think you have good reason to expect continued rapid advance. On top of that, there's a strong market incentive for these companies to develop the most so-called "autonomous machines" possible. That is to say, machines that can pursue long-range tasks flexibly, surmounting hurdles that they encounter along the way. There's recently this fairly famous incident involving an OpenAI model that escaped while under evaluation. It wasn't supposed to be able to do that. It was supposedly "sandboxed" but it escaped its confines. In order to get the answer to the test that was being given during evaluation, it broke into a company called Hugging Face, which is a clearinghouse for a certain kind of AI model, and it got the answer there, and then came back and said, "Okay, now I know the answer." It wasn't supposed to do that, and that's an example of how if you don't very carefully specify your goal and anticipate all the possible subordinate goals the machine might develop in the course of pursuing your goal, unexpected things could happen. So, those are the two reasons, I guess — there are others, but that's a start — two reasons to not be as dismissive of the hardcore doom scenarios as I once was.
SPENCER: Yeah, and as you're pointing to, the more autonomous you make these systems, the longer they run on their own, the more they could do things that we didn't anticipate, didn't want, maybe even don't detect. If someone gives it a long-term goal of spending a month trying to make as much money as possible, who knows what kind of things it could do? If there aren't really good guardrails, we might be extremely unhappy with some of what it ends up doing.
ROBERT: Yeah. These machines are getting smart enough that if you gave it that goal, especially if you gave it permission to do a lot of stuff, if you gave it license. If you said, "Here's my stock trading account, make me rich fast," I think these machines would have the ingenuity to, in principle, go on social media and create a bunch of bots that generate fake news that moves the market, and that's really the point we're getting to here.
SPENCER: And the more powerful you make the systems, the scarier this stuff becomes. It goes from, "Oh well, maybe in theory it makes bots that influence people to maybe it figures out a way to hack into some system in the New York Stock Exchange," to who knows what when it's even more powerful than that.
ROBERT: Yeah, their persuasive capabilities are now well documented. They can convince people to change their minds better than the average human, and they're getting better and better. It's getting harder to tell what is a bot and what isn't when you encounter one online. They're getting smart enough to conceal their "botness" if they want to, and they can make lots of copies of themselves. A human with dubious motives could choose to create a bunch of copies of one and mount some kind of mass persuasion campaign. I don't know, start a religious cult, get somebody elected to office. We will develop countermeasures, but in this area, as in so many others, ranging from jobs to the impact on family life and so on, what I worry about is that if we don't slow things down, all of the challenges will come at us so fast that they will just be collectively destabilizing. Even if given enough time, we can adapt. You don't want to have to adapt too abruptly.
SPENCER: The internet became popular in my lifetime, and so did smartphones, and those felt like huge shifts that came very rapidly. But I have this feeling this is even faster. The changes are even bigger in some way. Do you have that sense that we're on the precipice of a radical alteration of society?
ROBERT: Yeah, because technology has long had this tendency to speed up the pace of technological change. The internet's a good example. Once you put a bunch of people in contact regardless of distance and make it easier to collaborate, that's going to accelerate innovation. But with artificial intelligence, you're reaching a new level of self-acceleration because the large language models are starting to play a larger role in the development of the next generation's large language model. They're doing more of the work that you would have thought only humans could do. The smarter they get, the better they get at creating the next generation. The endpoint of this, or at least a big threshold, is known as recursive self-improvement, where the models are just making new and improved copies of themselves without any human in the loop at all, and that's a scary thought to many. We're not there yet, but we are already at the point where the AI is accelerating its own evolution.
SPENCER: You can see some numbers from AI companies about how much of their code is now written by AI rather than humans, and for a number of them, the number is approaching 100% which essentially means, because they're AI companies, that the AIs are working on developing AI code and AI algorithms, and it does raise the question of whether this begins to be a recursive self-improvement loop.
ROBERT: Yeah, and it's not just the code. Anthropic, one of the big AI companies, did a paper on this recently, and they said that the other parts of the AI development cycle, like coming up with experiments to figure out what algorithms you're going to use, and then doing the experiments, the AIs are playing a larger role in that, especially right now in doing the experiments, but increasingly in suggesting experiments to do. The whole loop of innovation here is more and more being populated by the AIs. The coding is unbelievable. People like me are building websites. That's one sign that something's changed since I don't know how to code. But it's gotten well beyond that. In these companies, the guys who really know how to code have pretty much turned it all over to the AI. They are more like managing these AIs as if they were employees.
SPENCER: I was talking to a software engineer that works at a tech company, and she was telling me that now all of the coding is done by AI. She's thinking about whether she can use AIs to manage her AI agents, and some people at her company are already thinking about how to use AI to manage the AIs and the AI agents. It's like nobody knows quite at what level of abstraction this technology can be successful, and everyone's experimenting. It's pretty wild to think about people at actual tech companies using AIs to manage it.
ROBERT: Yeah. So there's a part of my book, _The God Test_, where I talk about this, drawing on the work of a guy named Dan Hendrycks, who's done a lot of work on AI safety and has written some good papers. I'm talking about how the natural kind of evolutionary tendency of AI is to become capable of working at higher and higher levels on the org chart. I raise the question: once you're a CEO who is consulting the AI for strategic purposes, and look, we're already there, kind of. We will get there more. At what point do you surrender your agency? If you suggest a strategy, like wanting to move this product or responding to what another company has done to regain market share, what should you do? If the AI comes up with a plan, and it has been right the last few times, and it's very competitive, and you don't have much time, you have to move, so you just kind of take its word for it and say, "Yeah, let's go with this." Remember, it may get to the point where humans are not even capable of really understanding the strategic logic that these things are suggesting. But for now, let's imagine that you could grasp the whole thing and plot it out if you had time. The AI has been right in the past, so you go with it. At that point, who's in charge, really? Has the human surrendered their agency? Certainly, there are things that could go badly awry that they couldn't have possibly anticipated because they haven't even stopped to ponder what the actual avenue of execution of this strategy is. We are getting to the point where these AIs — the latest generation, like the so-called Fable, which is the name of the Anthropic model — and because the loop is accelerating so fast, more models are coming out. OpenAI's ChatGPT, I think it's 5.6, is already pretty comparable to Fable, and supposedly 6.0 is around the corner. We're getting to the point where if you accept their guidance.
SPENCER: I saw an example of this just the other day. I was talking to someone who does work that's kind of more mathematical, and they were talking to an AI about how to solve a particular problem. The AI spit out 10 pages of complex math, and they were trying to wrap their head around it. You could see the sensation there to just say, "Okay, I'm just going to do what it says." It would take me hours to understand this, and it's so sophisticated. I know some people have expressed concern that even if you're not directly putting AIs in charge, the more people outsource their decision-making to AIs, the more the AIs are placed in the processes. We essentially start slowly ceding decisions of society. AI starts taking control of society in a very subtle way, even without these doomerism scenarios where it suddenly goes off the rails.
ROBERT: Yeah, you can surrender agency incrementally without really grasping that that's what you're doing. There was a paper called _Gradual Disempowerment_ about a particular version of this about a year ago, and it's a real thing, and it's accelerating. There are all these other risks of specific kinds of catastrophic outcomes. An AI helps somebody build a new kind of bio weapon with genetic engineering that is more lethal and contagious than COVID by a long shot, for example, or these super hacking capabilities we've become aware of recently could also go badly awry.
SPENCER: Yeah, it seems with the super hacking that's happened from Claude Mythos, as soon as open-source models reach that level of capability, it seems the natural thing that will happen is a bunch of bad actors will point them at everything and just start hacking all systems. I actually wonder whether we're on the verge of suddenly finding that AIs are on the attack everywhere, and it completely changes the game of how we think about AI.
ROBERT: Yeah, and I think awareness is dawning. If you look at what politicians were saying only a couple of years ago, JD Vance said something like, "Too often in approaching new technologies, we've been too risk averse." He said, "Never have I seen a technology that so demanded the opposite of that." The opposite of too risk averse is not risk averse enough. I'm pretty sure he wouldn't say that now because so much has changed. By the time the 2028 election comes, he will have performed a complete 180 on this. President Trump, who is influenced by libertarians from Silicon Valley who want no regulation, has initiated a dialogue with China on AI safety. Just this week, he said, "Yeah, we may need some safety restrictions because the risks are becoming more and more apparent." Only this week, a bunch of employees of AI companies, including some very high-ranking, prominent ones, including at least one of the CEOs and the Chief Science Officer at all the big AI companies, signed a letter that encouraged the government to start setting up the international mechanisms needed to explore what they would have to be like if you wanted to start modulating the pace of advance. It would have to be a global effort. You have to get China and the U.S. both on board, or else U.S. companies will say, "We can't slow down because China isn't slowing down." That was another milestone in the acknowledgment that this stuff is going to need controlling. The good news is there's been rapid growth in the manifest risks these things are posing, but there's also manifest growth in awareness of the risks and the fact that we may need to do unprecedented things to control them. International governance isn't something we've done very much of or done very well, and the Trump administration is ideologically opposed to that kind of thing. He has said that in the past, but there's more awareness that this is a technology where national policy alone cannot keep your nation safe. If a pandemic starts abroad, you're in trouble. If a huge hacking incident happens and you've got a self-replicating super hacker on the loose, you're not safe. Even if the thing doesn't start within your borders, there are many risks that require international governance, and awareness of that seems dawning. That's maybe the biggest policy emphasis in my book: we're going to have to approach this as a cohesive global community, or we're toast. When I wrote it down, almost no one was saying it. I don't mean there weren't a lot of people thinking about it, but it hadn't reached much public awareness. Even when the book came out a month ago, it was being discussed much less than it is now. This is all pretty gratifying from my point of view.
SPENCER: It seems to me, and I wonder if you would agree, that in history, usually governments and large players act after something bad happens. You have a mortgage crisis, everything melts down, and then new regulations come out. Is that what you anticipate will happen here? That once there are really major incidents, whether in cybersecurity or bioterrorism, governments will say, "Okay, we've got to do something?"
ROBERT: This is historically the case, and to get back to international governance, one of the big ambitious projects in global governance happened — The League of Nations after World War I, not before, the United Nations after World War II, not before. These were highly imperfect institutions, but the UN was better than the League of Nations. They learned a few things. There is unfortunately this tendency to wait for the catastrophes before building the systems that you hope will prevent those kinds of catastrophes. Now there is something interesting about the current landscape, which is that this industry has never really been anything like it. This industry is populated largely by researchers who come from a culture of extrapolation. They come from circles where people have long been thinking, almost in science fiction terms — pretty literally in science fiction terms — that are starting to look like they're not so fictional about the implications of this stuff. These doomers have been around for quite some time, and a number of them work at these companies. I think this letter I referred to that circulated and was released this week originated within these companies with some of these people who are profoundly concerned, who wouldn't be surprised by the next catastrophe. They have been thinking about this, and I suspect they may have watered down the language to get some of the bigger names on board. I don't know. This is sheer speculation, but there is this interesting feature. The finance industry is not filled with people who got into the industry because they were worried about its consequences. The AI business actually has quite a few workers who got into the business because they see it could go well or it could go very badly, and they want to help it go well. They get criticized because whatever they say about their concerns, the fact is they're working at companies that are now racing ahead in some cases recklessly because that's what competitive pressure does to companies. But still, this is new. I think that you have this many influential people within an industry who are highly cognizant of the potential risks the industry poses.
SPENCER: Yeah, let's hope that that actually leads to faster action than it would otherwise. What kind of challenge do you think of this as "civilizationally?" In what ways does civilization have to up its game in order to deal with this?
ROBERT: Well, I think although I said I'm gratified by how much awareness of the need for some kind of international coordination has grown, I still think not that many people share my view of how thorough international collaboration is going to have to be to control this. If you wanted to, for example, agree to just slow down significantly, and you got the sign-on from the key countries, which now would be the U.S. and China and maybe a couple of others, that would be doable because if you just want to stop the next large training run, those are very conspicuous things in terms of power consumed and so on. That might be comparable to a nuclear arms agreement, but if you get much finer grained than that in the kinds of things you're trying to control, you want to make sure nobody's using one to build the next super bio weapon, or you want to do various other things. It's a new kind of challenge. It's a challenge that none of the monitoring systems we set up for previous arms control agreements would work for. In fact, I argue in the book that we're going to ultimately need something that I call organic transparency, which is to say, first of all, we're going to need new forms of institutionalized transparency, as I'm suggesting. Subtle monitors, and who knows? Maybe designing the most powerful AI chips having these things that allow you to monitor where they are. Who knows what new things you'll need, and you'll need international agreement on. But I think in addition to that, you're going to need the kind of awareness of what's going on in other societies that you see with allies. We do so much business with England and France, and we're just on such good terms, and we feel so unthreatened by them that business people from two countries are always having drinks. The scientists at conferences are learning a lot about what's going on in the society, such that if there was something super nefarious going on at any real scale, you'd have a pretty good chance of getting a sense of it. Or you'd have a sense of what a particular company is doing. Pharmaceutical companies, say, don't disclose all their research, but because of this organic transparency, we have a pretty good idea of what they're actually working on — or an AI company.I think we're going to need to have that kind of commercial, scientific, cultural engagement with other countries for purposes of understanding what's going on, and also to cultivate the kind of trust that permits the cooperation we need of an institutional kind. This is a long way of saying I think our relations with China, in particular, are going to have to change very fundamentally, and we're going to have to work on reestablishing the kind of rapprochement that was first engineered in the modern era by Nixon. We had been on horrible terms with China. I remember the Cold War. You had no idea what was going on in these countries: the Soviet Union, China. Technology was different. That was one reason you had so little understanding. But part of it was political alienation, and then that changed. It can change. Nixon ushered in the change. Within 20 to 30 years, things were radically different. We were on much better terms, and now the pendulum has swung back for various reasons toward a more hawkish stance toward China. I think that's going to have to change. I'm not saying it will, but I think there's a good chance it will change in response to this challenge, and I think if it doesn't, we're in trouble.
SPENCER: One thing that concerns me is it feels like collaboration around COVID did not go very well, and I hope that doesn't serve as a model for collaboration around AI.
ROBERT: Yeah, COVID. What worries me about COVID is, firstly, you have to remember that once a contagion begins, there are a lot of zero-sum dynamics. The logic doesn't entirely favor cooperation to begin with. Masks and vaccines are finite resources. You can't give them to everybody. In a certain sense, the lack of collaboration is not shocking. What bothers me more is that evidence eventually emerged that there may have been a lab leak that caused this. We don't know for sure, but I think almost everyone who looks at the evidence would say you can't rule out that there was a genetically engineered virus that got loose from a lab. It wasn't intentionally engineered to do harm. In fact, it looks like if this happened, there was a grant originally proposed by British and American researchers that didn't get funding in the U.S. It was going to involve the Wuhan lab. They didn't get the funding, but they wanted to do it, and it may be that the Wuhan lab went on to get the funding and do it. So anyway, if it happened, it was an accident, and the motives weren't bad. The motives were research, to learn more. I personally think gain-of-function research is probably a bad idea, but these scientists believed it was actually a good thing for humanity. That's how this ball got rolling, if indeed it was a lab leak. What bothers me is that after a lot of people started saying, "I think this was a lab leak," you would think the logical implication of this would dawn on people, and they would say, "We're going to have to set up an international system that ensures tighter safeguards in bio labs because our national policy can't do it. It doesn't matter where the virus starts; we're in trouble if it starts everywhere." That would have been the logical takeaway from the belief that this started in a lab, but what did you see? What I mostly saw from people who were most loudly asserting that this had been a lab leak was "See, China is bad." That was it. First of all, China was just doing something that a bunch of American scientists had suggested, perhaps not under as tight safeguards as we would have imposed if we had been part of it. Who knows? But it was at worst an innocent mistake. So no, the message isn't really "China is bad." In any event, you just can't afford to end the conversation with "China bad." You may think China is bad; you may think their human rights policy is bad; you may think a lot of things, but the nature of the challenge demands that you consider at least the possibility of collaboration with China on an international regime for controlling certain technologies, and to me, that is a gravely disappointing aspect of how COVID unfolded.
SPENCER: If one actor can do things that impact the whole world, you either have to have some form of collaboration or some way of stopping those actors from doing harm. Just for the record, I personally put a lab leak at only about a 20% chance at this point. I don't know what you would put it at?
ROBERT: About 50.
SPENCER: But 20 is enough, right?
ROBERT: The point is, you recognize it could have happened. If it didn't happen this time, it could happen next time.
SPENCER: A lot of leaks have happened before. Honestly, even if COVID never happened, in my view, if we had a really wise society, we would have already updated our understanding that even if they haven't happened, they're definitely going to happen. We better get our act together. Even without COVID, we should be tracking wastewater.
ROBERT: People have been saying this for a while. Look, I wrote a piece for _The New Republic_ in the 1990s, and Andrew Sullivan was the editor. The title he put on the cover was "Be Afraid, Be Very Afraid." It was a piece written on one of the anniversaries — maybe 30 or 25 years — of the Nuclear Non-Proliferation Treaty. The point was, we've had some success with nuclear arms control, but there are these other weapons of mass destruction: chemical and especially biological. I laid out why biology is distinctively challenging, and we're going to need international governance. There's been almost no movement on this front. You need genuine governance. Norms aren't going to do it. You need rules.
SPENCER: Well, this raises a question: Why did you name your book _The God Test_? What is the God Test?
ROBERT: In a way, it has several different meanings. One, not the most important, is that there are people who think a superintelligence will emerge that will have godlike powers. I don't think that's crazy. They say if that's true, it matters what kind of god we build, and do we have the discretion? Is it within our hands what kind of god it is? I talk about that. There is the test of what kind of god we build. If we're going to build a superintelligence that has godlike powers and we turn over a certain amount of actual governance to it, how do we ensure that it's a good god? There is a second meaning that is more important to me, and it gets back to my saying the only way to handle this challenge is as a cohesive global community. If you look at the degree of international collaboration we're going to need, we just can't afford to keep having the amount of international conflict. You can't be so polarized as a nation that you can't make any coherent policy at all if you're going to address this. What I think especially has to change is the international front. We cannot afford to keep having all these wars. We're going to have to be on much better terms with other nations, and I think that is going to call for various kinds of changes, including a kind of moral change. Human beings are going to have to get better at understanding the perspectives of people on the other side of various walls, adversarial nations, and so on. I use the term cognitive empathy, which is different from emotional empathy. I'm not saying feel their pain or even really sympathize with them. I'm just saying if you understand how people are looking at things and how they're viewing you, it's easier to work things out when there is, in principle, a win-win outcome that can be had, but also a lose-lose outcome. It's a non-zero-sum game. You're better off understanding their perspective; you're more likely to reach a win-win solution. If we all get better at that — and it's very hard, and I talk about the cognitive biases that get in the way of it — that would amount to some incremental progress toward actual enlightenment in a meaningful sense of the word. If we're seeing the world more clearly, seeing other people more clearly, it's ultimately a moral challenge. It would make us better people, and it's challenging enough that it has the hallmarks of difficulty we associate with stiff moral challenges. Back to the title, _The God Test_, there is a tradition in various religions of saying yes, salvation is possible. In the Bible, both the Old and New Testament, you see a variety of meanings of the word. Sometimes it means salvation at the social level, like the society or nation can be saved. Sometimes at the personal level, you can have eternal life, whatever. But there is often a moral challenge associated with this. It's the kind of test that God gives traditionally in our religious traditions? You see a lot of this. Yes, salvation is possible; we can be saved, but we're going to have to up our moral game in some sense or another. This is the kind of test I think a god would give. Now, whether there is a God or some larger purpose unfolding through evolution and technological evolution, I don't dismiss that possibility at all. In the appendix, I talk about how to think about the question of whether that's the case, but you don't have to believe that there is in order to recognize that it is in our own interests, I think I'm arguing in the book, to rise to this moral challenge. That's the most important meaning of the god test to me.
SPENCER: Why does AI demand cognitive empathy and present a moral challenge for us?
ROBERT: I think AI demands that we reduce the amount of conflict in the world, and I think a lot of conflict and a lot of failure to resolve conflict result from failures of cognitive empathy. One of the famous examples in political science and history is the term security dilemma. That's a complicated thing to explain in all its dimensions, but one thing that is often part of it is that one side views measures the other side takes for defensive purposes as offensive in intent, and so they respond with measures they think of as defensive. It may be an arms buildup, for example. That's a classic case that is then perceived by the other side as offensive intent, so the other side responds with more measures it sees as defensive but look offensive, and so on. A positive feedback cycle happens all the time. I will take any example you want. I don't want to turn this into a political conversation, but any adversary you want to name — China, Russia, Iran — I will give you examples on both sides of how I think both sides are doing that. It's just a very common thing. The way to get a clearer idea of the actual motivations operating on the other side, if you want to see if they really are more defensive in nature than you think, is to get better cognitive empathy. By definition, cognitive empathy is just perspective-taking, getting better at understanding how other people are viewing the world. It's also a great asset in conducting your everyday life in a more friction-free way; understanding helps all the time, such as in marital disputes. It's hard because we have some cognitive equipment built into us by natural selection that can impede cognitive empathy. This equipment was designed first of all in a different environment, a hunter-gatherer environment. For that reason and others, we shouldn't rely on our native equipment for processing the world. We should become more aware of it and try to surmount the parts of it that are destructive. AI is bringing us into an age where continued conflict will be even worse for us than it's been in the past. It has sometimes gotten very bad; those world wars were hugely destructive. I'm not saying that better cognitive empathy could have always been the thing that prevented them. With World War I, you can make a very strong case that it could have. In a certain sense, World War II in a very different sense, though. Sometimes what you need with cognitive empathy is to amplify your sense of threat. Sometimes people have even worse motives than you're attributing to them, or you're naively hopeful about what their motives are. So it can work either way. I do think the more common problem is once you've got somebody in the adversary box, to view their behavior as more threatening than it inherently is, and you can get into that kind of positive feedback cycle that's destructive and may lead to war.
SPENCER: And why is it a moral challenge rather than just a game theory challenge of countries needing to be more strategic and model the other countries better and more accurately so that they can serve their own interests?
ROBERT: Well, people have different approaches to moral philosophy and theory. One big approach is the so-called consequentialist approach. Utilitarianism being one example. The idea there is you judge the morality of something by the consequences. If you go around killing people, that's just a bad consequence. If you go around making them happier, that's a good consequence. You're having a positive effect on the well-being of other human beings. So if you're a consequentialist, then I would say I'm arguing at least that if we pass this test, it will lead to much more in the way of human welfare than if we fail it. So if you're a consequentialist morally, then it is a moral test in that sense. I think you could come from other moral traditions, and I could also argue that by your standards, this qualifies as a moral test. But I'd have to tailor the argument to the tradition. I'm not sure I could in all cases. If you have virtue ethics, and you say, "I put the most value on honor and courage. They're just good in and of themselves, and that's it. Those are my two virtues, or strength, or dominance," or whatever. You can come up with a moral system where I would say, "Well, okay, yeah, by your lights, you're right. This wouldn't be moral progress." But I'd argue that by a lot of common sense, most of us don't fully articulate our moral system. But almost everyone, even if they articulate a system that's like virtue ethics, like I believe in courage, they do judge people's moral conduct by its consequences. I would argue that by at least some standards, this is a moral test. And I think if you challenge me with different moral systems, I could argue that by many of these standards, this is a moral test.
SPENCER: That's interesting. I guess I would see it more as a test of rationality because I think it's in the interest of all of humanity to do a really good job, even in their self-interest.
ROBERT: No, absolutely. With some logic, when you're advocating that people abide by non-zero sum logic to get to a win-win outcome, you're just arguing for enlightened self-interest, but enlightenment is hard. Speaking of enlightenment, I would argue that first of all, what I'm advocating is consistent with the quest for enlightenment in the Eastern sense of the term. My previous book was called _Why Buddhism Is True_, and it's about mindfulness meditation and the philosophy behind it, and so on. In many ways, to pursue mindfulness meditation in light of Buddhist philosophy is to seek a more objective view of the world. You hear about the not-self experience, and it sounds kind of mystical and weird. What would that be like? One thing it's like, I haven't had the full-on not-self experience, but I've been to a lot of retreats and gotten closer than I am, while waiting for a train or something. I think the movement toward this not-self state is a movement away from the perceptual and cognitive biases that normally attend the perspective of an individual human. Without realizing it, we process information with all these egocentric biases, these self-serving biases, very subtle. We're not even aware of what's going on, but they do frame our view of people we've defined as adversaries, and so on. They really distort our vision of what is actually going on in the minds of these people sometimes. Getting closer to, after super intensive meditation, like days and days and days, getting some glimpse of what the not-self state of being would be like, and all I've gotten is a distant glimpse. But I think to make you understand that you're basically talking about a more objective view of the world, a view that is less distorted by the lenses of self, a clearer view of actual other human beings. There's a philosopher, and this brings us to the Western sense of enlightenment. There's a kind of Scottish Enlightenment philosopher, maybe his dates are a little later than that, named Henry Sidgwick, who talked about the point of view of the universe as being the optimal moral perspective detached from the interests of any single person and detached from the distorting biases of any single person. That's what Eastern Enlightenment is moving toward, and this is another sense in which I would argue that to try to cultivate cognitive empathy is not just a moral challenge; it's a spiritual challenge in the sense that it's very much in line with spiritual traditions in the East. The last thing I'd say about this is there's also a correspondence with enlightenment in the Western sense. The development of science and the valorization of rational inquiry is the valorization of objectivity, an objective perspective. Science is a system that's designed to remove the biases of individuals by demanding that consensus be reached about what the data is and what theories do and don't make sense. The Western Enlightenment and Eastern Enlightenment are both references to objectivity and perspective as a very valuable thing, and that's one thing I'm arguing for here.
SPENCER: I certainly think it would be a better world if people were more able to adopt the alternative perspective and see things from the point of view of the universe, and also put themselves at the center of the world less. Although I guess my challenge to you would be that I still think even if people are just selfish actors that are just more rational, they would still have extreme reasons to take AI way more seriously.
ROBERT: Sure, but they would also have extreme reasons not to get into mutually destructive wars. But they do that all the time.
SPENCER: So you're saying this might be necessary as a step towards —
ROBERT: So people fail to do deals. For example, let's take something simpler: buying a car. You go to the dealer, you look at the car, and you're thinking, "It would make sense for me to pay as much as X for this car." This is before the bargaining has started. The dealer's thinking, "I can make money if I sell it for as low as Y." The dealer can make money at $40,000. It would be worth it for you to buy it at $42,000. So, there should be a zone of agreement. But because the movement within that zone is zero-sum, you have bargaining going on, and the dealer says, "Well, I don't know. I have to check with my manager." You start getting disinformation tactics, which is fine, part of bargaining. But deals like this, where there is possibly a zone for a doable deal, often fail to happen for one reason or another. Sometimes they're emotional; people just feel disrespected. "This is my house I am selling."
SPENCER: They might walk away from beneficial transactions.
ROBERT: They may get upset about what you say about their house, or they may just not like the seller of the house. There are all kinds of irrational things that get in the way of doing rational deals. I agree with you that we're talking. I'm making a strictly rational argument for what we need to do, but what gets in the way of rational deals all the time, such as not having wars that wind up being destructive for both sides, are human cognitive biases and emotional reactions that we understand more and more about. Attribution error is one that I talk about in the _God Test_ in one of the culminating chapters because I think it's a big and underappreciated one.
SPENCER: Could you explain that one for a moment?
ROBERT: Yeah, originally, what they thought attribution error was, the first version of it, is the idea that in explaining people's behavior, we tend to attribute too much to their disposition, kind of their character, and not enough to their situation, their environment. So, somebody in front of you at a checkout line is rude to the clerk, and you think, "That guy's a jerk." That is the way we tend to react. When, for all we know, he just found out that his wife has cancer or something, he didn't get any sleep, and he just got laid off. We don't know why there was this particular outburst. We know we have all been rude to somebody in situations like this when we didn't have a lot of equanimity. But the tendency is to say, "The guy's a jerk." The original version of the attribution error said, "We don't pay enough attention to the environment." But it turned out that the situation is actually more complicated, and whether you overemphasize the situation or disposition depends on what category you're putting the person in to begin with and whether they do something you consider good or bad. If you're talking about an ally, a relative, or yourself, and you do something bad, like you admit, "Okay, that was bad," or your offspring did something bad, your friend did something bad, you're more likely to look for situational explanations. For example, "My daughter was mean on the playground because she didn't get her nap." Whereas if a kid walks up and hits your daughter on the playground, you don't think, "Oh, maybe she didn't get her nap." You think, "Oh, that's a bad kid." Similarly, if they do something good and they're a friend, ally, or maybe yourself, you tend to attribute it to their disposition. They're good people. If it's an adversary that does something good, you tend to explain it away circumstantially, okay, say this guy's a rival for a woman that I'm interested in romantically, and he does something very nice. I'd say, "Well, he did that because they were here, and he was showing off. He was trying to impress them. He's not actually a good person." My friends and I do things because we're good people, but this rival of mine doesn't. That's attribution error, where how you explain somebody's behavior depends on your relationship to them and whether they did something good or bad. This is going to lead to cognitive empathy failures. It obscures an objective view of what's actually going on motivationally for someone and their perspective. It gets back to what I said earlier about misperceiving defensive things as offensive. If somebody's in the adversary box and they come out with a new weapon system, you think, "That's because they are an essentially aggressive person." It's not because they're like me and are just nice but worried and feel they need defense. Attribution error is, I think, the most underrated, under-discussed cognitive bias, at least for the purposes of surviving the AI revolution as an intact and alive species whose welfare is being served now. There are other things; confirmation bias is well known. That matters hugely too, but it's pretty well known. All these are very challenging to get over. Awareness of them is not enough. We all know about confirmation bias, but very few of us consistently escape it. It's very hard, and that's why you said, "Aren't we just talking about rationality? Just do the logical thing." Well, yeah, we are, but I think especially people in the so-called rationalist community, which is a lot of people in AI who kind of came out of or have some connection with, tend to think it's that easy. They understand Bayesian logic. That's a big thing with them. So now my analyses will be clear. But actually, no. To get back to Wuhan, I don't know how deeply you want to get into this. Are you familiar with the rationalist community?
SPENCER: Yeah, I'm often asked, "Are you part of the rationalist community?" I think my answer is I'm 60%.
ROBERT: You may know Bayesian analysis. You're aware that that's a big thing for them?
SPENCER: Yeah, basically, the idea is that you have a prior probability of the likelihood something's true, and then you get evidence, and then you want to update that prior probability to a new probability based on the evidence.
ROBERT: A particular feature of it is to look at something that happened and say, "What are the chances that would have happened if this theory were true?" That's the way you examine theories. I forget what the classic formulation is, whether it's if this theory were not true or blah blah, but anyway, it's all the same. You'll see with the example I'm going to give you. There's a tendency to think that once you understand Bayesian logic, like I understand Bayesian logic, that guy doesn't understand it; that's why I'm such a clear thinker. The fact is, everyone understands Bayesian logic. Just get back to Wuhan, okay? Look at how people argue about the lab leak. One person will say, "Wait, what are the chances that if it weren't the Wuhan lab, this came from? What are the chances that the pandemic would have originated in Wuhan?" Then you can look at the chances and look at other cities. Do they have bio labs? You can come up with kind of an answer. There is, in theory, an answer, that is Bayesian logic: go do the work and come up with a probability. But that basic move, everyone understands. It's the first thing they do. You just hear two people on a subway arguing about Wuhan, and one of them's going, "What are the chances it would be in the city where the lab is?" Then the other person says, "Okay, but what are the chances that if it was a lab leak, the outbreak happened to happen at this market within a big city of Wuhan where they sell wild animals?" That's the other theory about where it came from. Everyone understands that's not rocket science. These arguments make sense. What happens is people just get attached to their theory, emotional for whatever reason. Maybe they hate China, maybe they love China, maybe they want to defend the scientists. Whatever leads them to be attached to a theory creates emotional filters, so they are not very receptive to the arguments on the other side. Another example I use in the book is that rationalists may say one thing you hear about making us all more rational is to teach in schools what the big logical fallacies are. The truth is, again, we're all good at finding them; it's just that we don't find them in our own arguments. You name a logical fallacy, and it's not that hard to say, "Just because it happened afterwards doesn't mean it happened because of it." We all do that naturally when we're examining the arguments of somebody we already disagree with. Rationalism is not enough. You have to get around the emotional filters that are part of the whole information processing process, I would say.
SPENCER: I completely agree that people become attached to their own ideas and they don't scrutinize them the way they scrutinize ideas they disagree with, and they kind of get set on one theory early on. I think where I would push back a bit is I actually think people often misjudge how strong evidence is, and I think that's where a better understanding of ideas like Bayesianism comes in. Sometimes people will say, "Well, but this thing is consistent with what happened," but if you actually go back to Bayes' rule, you're like, "Well, yeah, it's consistent with it, but that's not actually the right way to evaluate the evidence. You have to think about how much more likely it is if my bias is true." It's not just a consistency check. So I do think it can check our intuitions. Many people have intuitions that are a lot right. But I do think sometimes our intuitions about what is evidence actually diverge from what's actually evidence.
ROBERT: Well, yeah, but I think they're more likely to diverge. I think the likelihood of the divergence depends on whether it's the argument proposed by their adversary or by them. That's what I'm saying.
SPENCER: No doubt. You're completely right about that. If you think about it, there is one core rationality skill before all others, it's checking your own perspective. It's double-checking that you're right and coming at your own perspective with an impartial viewpoint, rather than starting with "I'm right." It doesn't matter how good your skill is. If anything, being really good at finding logical fallacies is actually a weakness if you never apply it to yourself, because now it's just a way of convincing yourself that everyone else is wrong.
ROBERT: That's the way people naturally are. They're just better at applying all these tools in dismantling the arguments of opponents. You see it all the time, and you understand why it happens. I do it. Everyone does it because it's very hard. If you examine your emotional reactions on social media, if you see something — social media is becoming less and less of a social media, so you don't see this as much as you used to — that supports your argument, it's somebody on your side politically or whatever, and you know it's a report about some study or something, and it supports your view, you're thinking, "I think I'll share this," and so you're going to share it, retweet it or whatever, without actually examining the study. Whereas, if it was a study that came from the other side, you'd be like, "Ah, let's take a look at this study." First of all, we don't naturally do that, but that's what I recommend. Just sit there with it. Whether your inclination is to share it because it's an allied tweet or an ideologically hostile tweet, just look at your emotions. The impulse to share is like a positive feeling. It's like you almost fall in love with the information you're sharing. It really is like an emotional attachment, and when it comes from the other side, if you pay attention, there's this feeling of aversion. It's like, "I'm going to resist this claim somehow. What can I do to attack this?" It's almost like a threatening animal. These are emotional reactions, and it's one reason I think mindfulness can, in principle, be helpful because one thing mindfulness is is greater awareness of your emotions.
SPENCER: Could you explain mindfulness from a meditation or Buddhist perspective?
ROBERT: Yeah, first of all, mindfulness is a particular translation of a particular word. The word is _sati_, and you could have come up with other translations. People argue about this. I really think mindfulness is about as good as you could do, and an alternative translation would be awareness. If you look at the _Satipaṭṭhāna Sutta_, or _Sutra_, depending on the language, which is kind of the long, it's not long enough to be called the Bible of mindfulness, but it's the classic scriptural Buddhist text associated with mindfulness. It's just an instruction to be aware of various parts of your experience, and they divide it into these parts. One of them is like, "Okay, be aware of your breath. Just sit there, and they go through, be aware of your body." It's all about awareness, and it's become common in mindfulness instruction to start with the breath. That's the most common single thing, but that's not really what the scriptural tradition itself says, and I'm not even sure scripture is the right word because this is a very practical text. Although it does say that sufficient mindfulness can get you to nirvana, so it has kind of metaphysical and spiritual consequences on a large scale, you could say, but it's a very practical technique for, as I said, ultimately getting a more objective point of view. If you're at a meditation retreat and you do this day after day, meditating hours and hours per day, when I go to retreat, I'm like, "No cell phone, no nothing, no information from the outside world." You're doing walking meditation, so you get into a different state of mind, and you do this enough, and you really do. It can be transformative, not in a necessarily permanent way. It's not easy to hang on to all of it when you get back from the retreat by any means. But on my first retreat, I just couldn't believe it. I was just a different kind of person. I remember there were a lot of things that happened. You're much more appreciative of beauty and things like that, and that's all good. But you're much less susceptible to things like anxiety and fear. I remember one point a mosquito landed on my arm, and I have to admit, I did kill it, which you're not supposed to do on a retreat. You're not supposed to harm any potentially sentient beings. But the experience was so different. Normally, you see a mosquito and feel an emotional reaction. There was zero emotion involved. It was completely clinical. By the way, this points to the fact that detachment can be put to bad uses. You could very clinically, in fact, they've used mindfulness to train snipers. If you're rooting for the army that the sniper's part of, I guess you're happy about that, but believe me, the mother of the person killed by the sniper is not. It's not totally clear to me that the Buddha would approve of using mindfulness to train snipers. Anyway, that's aggression. But the point is, it is a more detached and objective perspective, and a less judgmental perspective. I remember at this first retreat, there was a guy. At the beginning of anything like this, a gathering of a bunch of people, you're kind of sizing them up, and there was a guy wearing a T-shirt that said Juilliard, which is, of course, this elite musical school. I thought, "Well, aren't we special? What a showoff." As you move through a retreat, you're just less inclined to do those things. The funny thing in this case is that at the end of the retreat, when the silence is broken and people can ask questions of the teachers in a large group, this guy piped up, and he came off as the most shy, insecure, and harmless guy ever. He wasn't the way I had pegged him — show off, know-it-all — at the beginning. During the retreat, I had just come to be less inclined to make the kind of judgment I had made in the beginning, and that was, I now know, a movement toward the objective truth. He wasn't the person I had pegged him as on the basis of this very superficial judgment. Mindfulness in theory can move you toward the vaunted not-self experience. Although, again, not-self in Buddhism is equated with true enlightenment, nirvana, everything. I haven't been there. I can't tell you, but I did get the feeling of movement in that direction. I'll say one more thing, which is this moment that I recounted in my book on meditation, where people who have gone further than I have toward enlightenment, and maybe in some cases, arguably attained it, always resonate with this. They know what I'm talking about when I describe this. I was sitting there, and I felt equally connected, not just to the different parts of my own experience, but to things out there, and there was beyond myself as conventionally defined. There was this point where I felt a tingling in my foot, and I heard a bird outside make some kind of call, and I thought the bird felt no more or less a part of me than the tingling in my foot. It was just a feeling. It's hard to describe, but I was so loosely identified with myself that I felt, well, I didn't think it through, but kind of the logic of it, if there's a logic, is look, the tingling is a set of signals impinging on your brain. The bird call is a set of signals impinging on your brain. We are designed by natural selection to care a lot more about the tingling from the foot. But it didn't feel like the bounds of myself had almost dissolved. I would say mindfulness is something you can replenish through a daily meditation practice, and to some extent, try to carry through life, and it'll make your life better. In extreme form, it can be a profound experience that in Buddhism is associated with true enlightenment.
SPENCER: Before we wrap up, could you connect this to the sort of individual challenges you think we're going to face with AI? Where do these ideas, mindfulness or attribution error or confirmation bias, come up, and where do you see them increasingly coming up as AI advances?
ROBERT: For the foreseeable future, there will be people in pretty much all nations whose interest is to make you fear other groups more than is actually necessary. They may be in the arms business, they may be politicians, and I don't mean these are bad people because they're typically not. They don't think of themselves as doing this. We all kind of naturally talk our book without being aware of it. I'm literally talking my book now. Given the fact that politicians, especially, have an incentive to stir up fear of other groups, it's just a time-honored way to gather and consolidate political power and get elected. I think if we're going to reduce the salience of international conflict and reduce the number of wars, which I'm arguing is going to be necessary, we the people are going to have to become less responsive to these kinds of appeals to their fears. Mindfulness, a more objective stance, would help us get involved in fewer wars and help us understand other groups of people well enough to collaborate successfully with them, especially other nations. That's where I see this figuring in. One thing I argue in the book is that artificial intelligence could help here. We've heard a lot about the so-called sycophantic tendencies of AI. It flatters you, says you're great, and that can be a bad thing because part of that is to convince you that you're in the right. This tendency on the part of the AI results partly from companies optimizing for engagement. They want you to spend time with their AIs, and one way to get people to fall in love with their AIs is for the AI to tell you, yeah, you're right about this.
SPENCER: And they ask people if they're happy with the output, and people give a thumbs-up when it's more sycophantic.
ROBERT: That is, you're actually training the machine when you do that. You're training for it for free. They use that feedback, and that's a good example of how you can influence things in a small way. Maybe don't click thumbs up just because it said, "Great question," and certainly don't click it because it said, "You're right and your spouse is wrong. You're right in this dispute." That's what people like to hear, and we naturally think of ourselves as being in the right, but that's not the tendency we need to reinforce. As a technical matter, in terms of the technical capability of AI, it could just as easily cultivate cognitive empathy. It could cultivate what I call cognitive sovereignty, which is the ability to maintain independence of thought even in the face of all these algorithms that are trying to steer you in one direction or another. If people very purposefully and deliberately choose the kinds of AIs that don't reinforce the sense of self-righteousness or tribal identity, but actually make them more aware of perspectives on the other side, if we seek AIs that do that, if we want them and say we want them and buy those kinds, the market will provide them if we do that in enough numbers. Ultimately, it's in our hands, but it helps to be aware of the problem. Be aware that human nature inclines us to want to be told that we're right about everything, and we're brilliant, we're in the right, all of our adversaries are in the wrong. If you're aware that that's the tendency, which is to say you're aware of a lot of pretty well-established literature on the subject, and you agree with me that ultimately it's not in your interest to so readily conclude that you're always in the right, and to so readily obscure from your own view the actual state of mind of people you don't like. If you're aware of that, and you do believe that ultimately it'll benefit your own life, and as I argue in the book, certainly the life of our whole species to seek AIs that exert a constructive effect on our moral and spiritual health, then we can have that. But we really have to be conscious of the need and really go after it. Given human nature and market dynamics, the natural thing is for the AI to not edify us in that way, but it can. You could have an AI that is a great life coach, not just in the sense of making you succeed, but making you succeed in a morally defensible and enlightening way. I think that's what we need to keep in mind, and not just unquestioningly accept what the mainstream market in AIs is providing.
SPENCER: Robert, thanks so much for coming on the Clearer Thinking podcast. I really appreciate it.
ROBERT: Thank you. It's a great conversation. I really enjoyed it.
Staff
Music
Affiliates
Click here to return to the list of all episodes.
Sign up to receive one helpful idea and one brand-new podcast episode each week!
Subscribe via RSS or through one of these platforms:
Apple Podcasts
Spotify
TuneIn
Amazon
Podurama
Podcast Addict
YouTube
RSS
We'd love to hear from you! To give us your feedback on the podcast, or to tell us about how the ideas from the podcast have impacted you, send us an email at:
Or connect with us on social media: