JOSH: Hello, and welcome to Clearer Thinking with Spencer Greenberg, the podcast about ideas that matter. I'm Josh Castle, the producer of the podcast, and I'm so glad you've joined us today. In this episode, Spencer speaks with Tyler John about the pillars of utilitarianism, and comparing cognitive and rational endorsement to intrinsic values.

SPENCER: Tyler, welcome.

TYLER: Thank you. I'm really excited about this.

SPENCER: Are you going to convince me today that utilitarianism is true?

TYLER: I think last time we talked about this, I made very little headway, so I hope to make a little bit more this time, and maybe you'll be 1% more utilitarian by the end of this conversation.

SPENCER: Awesome. Looking forward to that. Just for the audience, can you define utilitarianism in a broad sense? Later, the conversation will go into more depth, and we'll talk about different versions. But what's sort of the broad headline of what utilitarianism is?

TYLER: Utilitarianism is a moral philosophy. It's the idea that every action we take should be devoted to maximizing the total amount of happiness in the world and minimizing the total amount of suffering.

SPENCER: Tell us a little bit about the history. How did this idea develop? Is this a new idea? Did it exist for a long time?

TYLER: Yeah, so actually, it's a fairly old idea. The first formulation of an idea that was similar to utilitarianism comes from the fifth century BC China, in what's known as the Warring States period. In the Warring States period, China was split up into a hundred different states, and there were hundred different philosophies. They were called the Hundred Schools of Thought. Among these philosophies were Confucianism and Mohism. Confucianism, which is well known for its presence in China today, and Mohism, which was an early precursor of utilitarianism. That's the earliest form of the idea. We see it pop up now and again throughout history. There were some early Indian philosophers in the seventh century AD called Cārvāka, and they came up with the idea that pleasure is intrinsically valuable, but utilitarianism is most known for its place in Great Britain in the 18th and 19th century, when it was put into a more rigorous formulation by John Stuart Mill and Jeremy Bentham.

SPENCER: So is the modern conception of utilitarianism quite different from the sort of more ancient ones that came out of China, for example? Or is it really kind of the same thing, just made a little bit more precise?

TYLER: It depends who you talk to. I had a heated argument with Brian Van Norden, one of the great scholars of Chinese philosophy, about this. I think this is hotly contested. What we know about the early formation of utilitarianism in what's called the Mozi, which is the book of the scholarly work of the philosopher Mozi. What we know about Mohist philosophy is that there was a group of Chinese philosophers who were going around trying to convince kings of their philosophy, and they thought that in order to have a rigorous set of policy proposals for these kings, you needed to have a rigorous set of standards that you could apply. The rigorous set of standards that they thought you should apply to test whether a policy is good is: does it benefit everyone? By benefit everyone, they meant: does it increase order, population, and wealth? On this basis, they argued for a bunch of things, like they argued against war. They were actually legendary military engineers and Mohists, and they would build up a powerful kind of contract military and then go defend countries that were about to be invaded. They were so powerful that the invading armies would simply flee and not attack. This is how they defended their pacifism. It was a philosophy that was very practical, but it didn't have a very rigorous, deep foundation. At least, I argue that. I think some others, like Chris Fraser and Brian Van Norden, think that actually it did have a rigorous foundation. The more recent formulations in Britain from Mill and Bentham have much more surviving text. We don't have all the texts from the Mohists; the British philosophers had a lot more leisure time to write because they weren't out building military siege engines and defending armies. They were involved in politics, so they had more time to write things down. The later formulations that you get from Mill and Bentham are called Classical Utilitarianism. This idea says that happiness is the only value and suffering the only disvalue, that you should maximize it, and that this should be for all people, indeed, all sentient beings everywhere in every time period. The Mohists, I think, didn't have something quite this well worked out at the time.

SPENCER: What were the disagreements at the time among utilitarians in the 18th century, like Mill and Bentham? Did they all come to the same conclusion, or were they actually differing on some important points about the theory?

TYLER: Mill and Bentham were part of a group of various other utilitarian thinkers at the time. Another famous one is Henry Sidgwick. There was a whole groundswell of utilitarianism in Britain at this time, and they agreed on a lot in politics. There was a school of utilitarian radicals where Mill, Bentham, and some others argued for a wide range of political views on the basis of utilitarianism. They agreed that we should abolish slavery, that we should be feminists, that we should give women the right to vote, that we should decriminalize homosexuality, reform prisons, and abolish the death penalty and capital punishment.

SPENCER: It seems so modern. So many of those things we now take for granted, but at that time, they were not at all taken for granted, and some of them, people are still pushing for today as, "Well, obviously we shouldn't do this, right?"

TYLER: Yeah, absolutely. I mean, I didn't mention animal rights, which is one that is certainly still being pushed for today. I mean, it's ahead of today much, much more ahead in the 18th century.

SPENCER: Were there substantive disagreements, either about these policies or just about the underlying theory of, for example, what do we count as happiness and suffering?

TYLER: So there were some political disagreements. I think Mill and Bentham famously disagreed on whether colonization was net positive for the world. I think the most significant disagreement that you'll hear about from the history of utilitarianism was between Mill and Bentham. Mill was a student of Bentham's, and Bentham had this view that all happiness is good happiness; it doesn't matter what you get your happiness from, he said, push pins or poetry, from games or from highfalutin philosophy, all pleasure is equally good. Famously, it's better to be Socrates dissatisfied than a pig satisfied. That's the quote that sticks to Mill. Mill thought that there are certain kinds of pleasures that are, depending on your interpretation, kind of infinitely more valuable than other pleasures. So if you are contemplating the life of the mind and studying utilitarian philosophy and engaged in politics, living this sexy murder poet kind of lifestyle, Mill thought this was the ultimate kind of pleasure that you could have. And if you had this kind of life, then no amount of playing games or eating food or having sex or whatever other pleasures Mill would have thought of as base pleasures, none of those could stack up to the profound pleasures of the life of the mind. So these were called higher and lower pleasures, and that was a big disagreement between Mill and Bentham.

SPENCER: So it sounds like, in essence, they agreed we should be maximizing happiness, but they disagreed about how to convert different things into comparable units. This type of happiness versus that one. We've got to get them down to some base unit so we can maximize the expected value. But how do we actually do that operation?

TYLER: Yeah, exactly. I mean, they wanted to maximize the good. They wanted to find a way to measure it. They measured it differently. Bentham thought pleasures are kind of equal; no thought is somewhat better than others.

SPENCER: So it seems a notch in utilitarianism's belt is that it pushed moral progress pretty fast on a bunch of topics where it took society a long time to come around to it. But the utilitarians actually kind of foresaw that, "Okay, these things were going to be considered immoral."

TYLER: Yeah, exactly. Some people take this further and think this is a really good argument for utilitarianism. When we judge scientific theories from history, we praise the theories that got really good results, that were far ahead of their time, made strong predictions, and ended up being right. Utilitarianism makes these incredibly strong predictions. It says, "The only thing we should care about is maximizing total welfare, and nothing else matters." On this basis, we're way ahead of our time, predicting all of these political currents. You could see this as one, and you could say the same thing about the Mohists. The Mohists are critical of war, mass conscription, stealing peasants' tax money, and turning it into lavish palaces for the elite. You take a maximizing, consequentialist, measurable view that just maximizes the welfare of all, and you get these really impressive predictions about where morality should be headed. Some see this as quite a powerful argument for utilitarianism.

SPENCER: Even in these early, or much earlier times, using this kind of thinking seemed quite helpful. Funny story about Bentham. One time I was walking around UCL, turned around the corner, and I ran into Bentham's preserved body. Have you encountered this?

TYLER: I was there two weeks ago actually paying homage to Bentham, yeah.

SPENCER: And I think there's some kind of note suggesting that people might gather there from time to time to think about his philosophy or something like that. It's an interesting choice.

TYLER: So he dedicated his body to the museum. So his head is actually preserved in a jar underground. It's actually locked away in a vault because some students stole it and used it for soccer or football here, but the purpose of his body wasn't for sports. He dedicated his body to the museum so that people could forever look at his preserved corpse and remember utilitarianism, which is kind of ironic because the plaque, I think the plaque actually just credits him as the father of liberalism. It doesn't actually mention his utilitarianism. So I think they screwed it up and should check.

SPENCER: Oh man, yeah. Okay, so this brings us to the modern day, and I would argue that while many effective altruists are not utilitarians, utilitarianism has heavily influenced the Effective Altruism movement. Would you agree with that?

TYLER: Yeah. So utilitarianism has had a big influence, both on the modern animal rights movement, inspired Ingrid Newkirk, Peter Singer, inspired Ingrid Newkirk to start PETA, is my understanding, and also on Effective Altruism, where I agree a lot of the early contributors to Effective Altruism, like Nick Bostrom and Toby Ord and Peter Singer, were utilitarians or close to it. The idea certainly seems to have utilitarian roots.

SPENCER: Yeah, it's funny, because I find if you press individual leaders in the Effective Altruism movement, they'll often say things like, "Yeah, well, I don't really fully believe in utilitarianism, or I have these objections to it," or whatever. But then if you step back and look at the movement as a whole, you do see this strong utilitarianism undercurrent. I think it reflects the movement more than it affects the individuals, if that makes sense?

TYLER: Yeah, I think that's fair. Utilitarianism is the idea that we should maximize total welfare for all individuals at all times. You can clearly see that echoing through Effective Altruism in its three priority areas: helping the global poor, because marginal dollars go further in developing countries than they do locally; helping non-human animals, because, as you know, Jeremy Bentham and Peter Singer argued they also suffer and should be included in the utilitarian calculus; and existential risks, because utilitarians, at least in the most paradigmatic form of utilitarianism, say we should care about all harms, no matter where they take place in time, including making sure future generations exist and flourish. You can see it as a kind of backbone in these priority areas, where maybe you would get different priority areas if the utilitarian undercurrent wasn't so strong. I do think various leaders, like Will McCaskill, don't fully embrace utilitarianism. They're attracted to it but also to other ideas and are more pluralistic. I've actually seen a shift in the community, which I've been a part of for the past decade, away from utilitarianism and towards more pluralism, which I see as kind of sad. But maybe this is what happens when you grow up and get older; you drop your idealistic ideas like utilitarianism and go toward something more pluralistic.

SPENCER: Yeah, and we'll definitely get into that. I think that's a really interesting question. For those that are really interested in all the different variants of utilitarianism, I recommend going back to Episode 42 with Nick Beckstead, where we really go into that in depth. But let's just do a very quick touch on the different kinds of utilitarianism. We've got classic utilitarianism, which is the one that you believe in. Is that correct?

TYLER: Yeah, absolutely. Classical Utilitarianism can be broken down into five pillars. The first pillar is totalism. The way you measure value is you add it up. You just add up all of the local places of value, and that's the total value of the world.

SPENCER: Rather than, say, take the average or something that might get you different conclusions, in some cases.

TYLER: Yeah, thank you. Average utilitarianism is at least a theoretical competitor, but I don't think anybody accepts it. So maybe you take the total, but you don't count future generations, or you discount them, or something. We can talk about that too potentially.

SPENCER: But for classic, you're just saying the sum of all beings that can experience suffering and happiness, right?

TYLER: Yep, you look at the space-time manifold, you find all of the locations of pleasure and suffering across all space and all time. You just put a plus sign between each one, and that's the value of the world. The second pillar is that it's maximizing. And I'll contrast this as you did, because that was helpful. Classically, totalitarianism says you should do the maximum amount of good. This can be contrasted with a version of satisficing utilitarianism, which says you just have to do enough good. One view is, for example, as long as your life is net positive, as long as you've done more good than bad, you're a morally okay person. But classically, utilitarianism is not gentle. It says, "Every breath you take, every action you choose, must create the maximum amount of value that it could possibly make relative to all other actions you could take." So it's very hardcore.

SPENCER: But you could interpret that in a little bit of a gentler way. You could say, "Well, there's always something better you could do. If you could get a higher expected value utility, there's always something better you could do." But who draws the line of when you know who's a good person or a bad person? That's sort of an arbitrary thing that we have after the fact, right?

TYLER: Yeah, yeah, no. I think your view is much better. So maybe you think that maximizing is the best thing you can do. It's almost by definition true. It's not. But maybe you think that maximizing is the best thing you can do, but you can still do better and worse. You can come closer to the maximum, you can be further away from the maximum, and we could evaluate your actions and say some of them are better than others, and it's not clear why we should draw a line and say this is where a good person is, and this is called scalar consequentialism. It's the idea that, instead of thinking there's one specific place on the value scale that measures a person, we should just compare all the actions and say which ones are better than others. And some of them are great.

SPENCER: This brings up an interesting point. Utilitarianism here, I think, is about evaluating actions, saying how good or bad each action is given the set of actions you could take. But that's different from evaluating a person, correct?

TYLER: Yeah, that's right.

SPENCER: So evaluating a person is sort of a whole different thing. That's a whole other conversation, potentially. You could have utilitarianism be true, and that's the objectively correct way to evaluate actions. But then we still have different theories about, "Okay, what makes a good or bad person based on that?"

TYLER: Yeah. So, while we're taxonomizing, there's a version of consequentialism called global consequentialism. So, let me zoom out for a second. There's a third pillar of Classical Utilitarianism I was going to mention, which is that you evaluate actions, and actions are the right thing to measure when you're thinking about what rightness consists in. Global consequentialism says, "Well, you know, we can evaluate all kinds of things. We can evaluate norms, rules, and we can evaluate people." And how do we evaluate these things? "Well, we look at whether that norm, whether that person, whether that virtue, that character trait, that rule, whether that feature of the world maximizes good consequences." So say, for a norm, we'd ask if you follow this norm all the time, or there are different ways to measure this, but if you were going to propose a norm, we could ask, "Okay, if that norm had uptake, would that be the best norm we could adopt? Or could there be other norms that are better from the perspective of maximizing good consequences?" And so, as you say, there are lots of things you can evaluate according to utilitarianism, and global consequentialism, or global utilitarianism, which I find attractive, says, "There's lots of stuff we can measure. If you want to know what a good action is, measure the value of the action. If you want to know what a good norm is, measure the value of the norm."

SPENCER: Got it. So they can be applied at all these different levels of abstraction, exactly. And then what's the final pillar? Pillar five.

TYLER: So I mentioned hedonism, that the thing we're trying to maximize is happiness, the balance of happiness over suffering. You can contrast this with other things that we could find valuable. Peter Singer, famously, was a preference utilitarian. He thought that you should try to satisfy as many preferences as possible and try to make sure that you have as few dissatisfied preferences as possible. And that's different from just pleasure and suffering.

SPENCER: Let's give an example of that. Suppose that someone discovers that they can secretly spy on someone in the shower. And they know that they're never going to get caught, let's assume for the thought experiment, so the person is never going to know about it. You could argue, "Okay, maybe that's not wrong from the point of view that there are going to be no negative consequences to that person being spied on,." But very likely that person would have a preference against being spied on, and so you might say, "From a preference utilitarian standpoint, that's wrong, even if it doesn't lead to negative consequences."

TYLER: Yeah, I think that's a good example for distinguishing them. Now I do think, even if you think happiness is the only intrinsic value, you probably still shouldn't go around engaging in voyeurism.

SPENCER: Because, as a rule, it might lead to, on average, bad consequences. So it's not a rule you would recommend to people.

TYLER: Yeah, it's a bad rule. It's a bad norm. But for the purposes of a thought experiment, for thinking about what is intrinsically valuable, this is a good example. There's another example that's similar that I find pretty compelling. The hedonists call this a coincidence problem. The idea is, most of the time when our preferences are satisfied, we feel happy about it, and when they're dissatisfied, we feel bad. Most of the time, happiness and preferences coincide, but every once in a while, they come apart. We can imagine a species like a Vulcan that is not conscious, doesn't feel anything, but does have preferences, wants some stuff, and is motivated by them. We can ask, "Do they matter?" Or similarly, we can imagine a very successful person who goes away on hospice to spend the last days of their life on a remote island by themselves, and while they're on this remote island, their whole life's work collapses, which they very strongly disprefer. But the news never reaches them, and they die shortly thereafter. We can ask, "Was it bad for them that their whole life collapsed during that time?"

SPENCER: Even though they didn't know about it. They didn't find out about it. So there's no consequence exactly, but they would definitely not prefer it to happen, yeah.

TYLER: They strongly disprefer they didn't find out about it. They're not unhappy. When I look at these examples, I'm like, "I don't know. Maybe we should care about the Vulcan; maybe we should care about the life work collapsing, but it's not super clear." One way of interpreting these thought experiments is, "Actually, pleasure was the only thing that mattered, all along. We thought preferences mattered because they coincided with pleasure." But we look at these cases, it turns out, actually, preferences aren't that big of a deal, after all.

SPENCER: You mentioned Peter Singer as a preference utilitarian. My understanding is he actually switched to being a hedonistic utilitarian. Is that correct?

TYLER: Yep, that's correct.

SPENCER: And so would he now basically agree with your perspective, he'd be a classical utilitarian like you?

TYLER: Yeah. So Peter Singer is now a classical utilitarian. He was convinced by meta-ethical arguments for moral realism and thought that evolution could explain very well why we seek to satisfy our preferences, but can't explain as well why we care so much about happiness. And so evolution suggests that we should care about happiness, or evolution — I might be butchering his argument, so I want to apologize to Peter Singer if you hear this. But so that people can understand, get a gloss on why he changed his mind, he thinks that evolution very easily explains our desire to satisfy preferences, but not as easily our valuing happiness. And so if morality is this real thing, and not just this thing that's based on our instincts and our base desires, then we can have a debunking argument. We can see that we only really cared about pleasure because of evolution, but that's not why we care about happiness. We care about it because it's morally right. I don't find this very convincing. We can talk about realism and anti-realism at length later, and I hope we will, but that's just a gloss on why he changed his mind.

[promo]

SPENCER: Now I would like you to give what you think are some of the strongest arguments in favor of utilitarianism, and see to what extent you can convince me of it. But I first want to say some nice things. I want to say some nice things about utilitarianism first myself, which is, first of all, I was utilitarian. So when I was in high school, I was reading Bentham, and I found it very, very persuasive. In fact, I would say the experience I had with reading Bentham was maybe not even that persuasive, but it was just like, "Oh yeah, that's what I already believe." It just resonated so much with me when I read it. It just felt totally natural, aligns with my thinking, and then in college, I would have said, "Yeah, I'm a utilitarian." I would have said, "I'm a classical utilitarian," like you are. At that point, I stopped believing. We'll get into later kinds of reasons for that, but I do think it has a lot of appealing aspects. One appealing aspect, it's a really concrete, specific theory, which is really nice. You can write it down. It's very simple. I mean, yes, there's a kind of weird edge case and things like that, but it's very simple to describe. You can say what you should do in some cases. You can literally calculate it when you have enough information. I think it's super appealing,

TYLER: Yeah. I think this is a part of the reason why you had all these nerdy scientists coming around utilitarianism. We had the mohist, these legendary military engineers who figured out that light travels in a straight line, like real mathy people like yourself, and I guess, like me, and like Indian hedonists, who were empiricists and thought, "There's no spirituality in science. You can only learn things from experience and testing them." And Mill and Bentham, who are also these empiricists, and we're like, "There's no God, there's no spirituality." I think these really mathy, hard-nosed, scientisty kind of people are attracted to it for similar reasons. You can just measure stuff and say very clearly what you can do and calculate it. And in addition to that being attractive for people like this, it's just handy; it's nice to be able to figure out what you should do.

SPENCER: Yeah, absolutely. I think that's a big benefit. I would also say that it captures something that's very important to the vast majority of people, which is to say, "Sure, there's the occasional person that doesn't care about happiness and suffering, or they only care about it for their friends and family, and they don't care about it at all for anyone else in the world. But the vast majority of people will agree that, in general, it's bad if there's suffering in the world, even if it happens to people they don't know. In general, it's good if there's happiness in the world, even if it happens to people they don't know." This is an important aspect of what people care about. Now, that's different than saying it's the only thing people care about, or that it's objectively the right thing to care about, but I do think it captures something important, and in particular when used in a government context, I think it can be a very important tool. Saying, how should we make decisions on behalf of the government? I don't think it should be the only tool. I think it should work alongside other tools, but I think it's a very useful tool. Considering, "Okay, we're thinking of implementing this policy. What is the utility impact of this policy?" I think it's very powerful as a guiding principle to say, "We should try to only implement policies where the expected value of them is positive from a utilitarian point of view," and if we have two policies and one has a much better utilitarian impact for the same cost, I think that is an argument in favor. It's not necessarily a dominant argument where it trumps everything else, but I think it is an argument in favor that should be taken seriously at a policy level.

TYLER: Yeah, I strongly agree. I think those are good arguments.

SPENCER: Okay, so let's get into your favorite arguments. Why should we use utilitarianism? Give us your first one.

TYLER: Yeah, so, I gestured at a couple arguments, which I'll just quickly rehash, and then I'll introduce a new one. So the couple arguments I mentioned were the track record of utilitarianism. It just really predicted some big moral changes in an impressive way. If we judge moral theories like we judge scientific theories based on their track record and strength of their predictions, the second one to go back over was this coincidence problem. If a lot of the stuff that we thought mattered, we think about it long and hard and we're like, "Oh, actually, we only like this thing because it tends to make us happy," then that's a good argument for hedonism, for pleasure being the only intrinsic value. So we talked about that with preferences. But you could also imagine this for various other things, like rights. "Oh, it's good to preserve rights because then people don't kill each other, and that makes people happy." A lot of moral theories tend toward utilitarian outcomes. This is the coincidence argument that, "Maybe the reason you liked all this stuff was because you're utilitarian."

SPENCER: And on that point, I think it's a very interesting point. It's a point I disagree with, and it'll be interesting to hash that out. But I do think there's something there which says, "Look, a lot of the things that you think you care about, if you really ask, 'Well, why do you care about that?' and then whatever you write down, you say, 'Why do you care about that?' You end up with something around happiness and suffering." I do think that's true, but I don't think it's true for everything we care about. So, there's a difference between it's true for many things we care about versus everything we care about.

TYLER: Yeah, very nicely said. So let me introduce another argument that I think is pretty powerful. You might have heard of the philosopher John Rawls. He's a very famous liberal philosopher, and one of his most famous arguments was called the veil of ignorance.

SPENCER: Yeah, everybody talks about Rawls' veil of ignorance, right? I think it's a little bit unwarranted.

TYLER: Yeah. Well, it's particularly unwarranted because he stole it. He stole it from John Harsanyi. So John Rawls' veil of ignorance says, "You know, if you want to figure out what a good society is, imagine you're behind the veil of ignorance, and you could be anyone in society, but you don't know who you'll be." Which society should you pick? Rawls is like, "Well, you know, the one that makes the worst people the best off." He stole it from John Harsanyi, who wrote it up like four years before. I sure hope he cited him.

SPENCER: We don't know if Rawls is to blame or not for that. For everyone associated with him, it might be that he tried to give credit. We're not sure.

TYLER: I mean, yes. So Harsanyi's argument was this original veil of ignorance. And he says, "Imagine yourself. You're behind the veil of ignorance. What principle would you choose for society?" And Harsanyi says, "If you imagine that you have an equal probability of being anyone in society, you could end up being really rich. You could end up being really poor. You could end up on various sides of social oppressions. The principle that you would accept is utilitarianism." Why? Because utilitarianism, by definition, maximizes expected value, maximizes value for society. And so if you're behind the veil of ignorance and you're trying to maximize your own expected welfare, you're trying to pick the policy that, when implemented, will make you the best off on average in expectation. Then you'll pick utilitarianism, because any deviation from utilitarianism will make the world at least a little bit worse. And so your prospects, your expected well-being is lower on any other policy.

SPENCER: It's so funny, because the Rawls's version of that always seemed incoherent to me, because Rawls, as I understand it, argued that if you were such a person behind the veil of ignorance, not knowing who you would be in society, you would choose to maximize the welfare of the worst-off person. But that's totally not what someone would do, actually, because that's only what the most risk-averse possible human would do. If you were maximally risk-averse. But I don't understand why he thought that they would do that. This seems like just a nonsense argument, yet it's so famous and popular.

TYLER: Yeah. So the reason Rawls thought you would do this is he formulated the thought experiment a bit differently. He said, unlike Harsanyi, who said, in the thought experiment, you are equally likely to be anyone, he was like, "Oh, the veil is so opaque. You have no way of assessing the probability at all of what person you'll be in society." There's just no way to even put a number on the likelihood of you being various people in society. And so he explicitly rules out, by the design of his thought experiment, expected value reasoning. You can't do expected value reasoning if you don't have any probabilities to work with. And so he's like, "Now that we've ruled out expected value reasoning, I guess we'll apply some other principle. So let's, you know, be really risk-averse." I think it's basically what gets them there.

SPENCER: Yeah, I still don't find that convincing for two reasons. One, because even if you have true ignorance, it doesn't mean you can't use expected value reasoning. You can be Bayesian about it and say, "Well, I've got some priors," or I can say, "I have an uninformative prior; I assume equal likelihood." There's a pretty strong argument behind that. If I say, "I've got an urn, it has some colored balls, I'm not going to tell you how many balls. I'm not going to tell you what colors they could be." It's not that you can't do any kind of reasoning. And then second, I still don't see how that gets you to the maximize the worst-off principle. It still seems like it doesn't follow. Obviously, I'm not criticizing you. I just don't understand Rawls's point.

TYLER: You're a mathematician and a smart one, Spencer, so you probably know of these paradoxes of setting priors, like the Bertrand paradox.

SPENCER: Tell us what that is.

TYLER: So you're a mathematician, Spencer, and a smart one. So you probably know about various paradoxes for setting priors. There are various paradoxes. You just don't have a natural way of setting a prior over a situation.

SPENCER: Yeah, one I like that's really simple to explain is, okay, first, the simple case: imagine that you're at a game show. There are two doors; one has a goat, one has a treasure chest. What's the probability that it's the left door that has a treasure chest? It's very natural to say 50% without any other information because there's a symmetry argument. Anything in favor of the left door is also in favor of the right door. So you choose that. But now let's say you're like, "Okay, some year in the future, some event is going to happen, like, AI is going to kill everyone or something." What's your prior over what year it's going to happen? You know nothing about AI. You know nothing about this threat, right? And there actually is no way to have an informative prior over an infinite future. It's just mathematically not well defined.

TYLER: That's a good example, yeah. Or, throw a dart at an infinite dartboard; which spots are you going to hit? You don't have a prior over this. It's not mathematically well defined. So I think, Rawls at least wants it to be one of these situations where you can have an informative prior. But I don't know why he's entitled to this. I don't know why you'd go for maximizing this position to the worst-off person, even if you didn't have a prior. And what I like about Rawls's argument is, I think, there's no proof of utilitarianism. It's an ethical postulate. It's not something you could prove or disprove, but what it suggests to you is a very natural way of spelling out impartiality, a natural way of spelling out treating everyone equally, which is this veil of ignorance, and you're having equal probability of being anyone. It's just a very natural way of spelling it out. Utilitarianism looks like the best principle for achieving equality.

SPENCER: And I think it is a nice argument. However, I do think it makes assumptions, some of which I disagree with. One assumption is that we're trying to maximize the expected value or the average. While that seems very natural, it is not automatically true. It actually requires a different argument to say, why the average? Why not maximize the median or the minimum? We're all starting to do that. Not saying Rawls is right, but why is he wrong? Another assumption baked in there is that we only care about utility. Because what if, in the veil of ignorance, you actually cared about something else? Maybe I also care about this other thing, truth or something. And then sort of assuming that there's one thing we care about off the bat.

TYLER: Yeah, I think those are great arguments. If you care about other things, maybe you get an impartial consequentialism about those things, but you have to care about them as consequences, as states of the world, not like they're virtues or something.

SPENCER: You could even imagine a version of his argument that says, "Well, we have a veil of ignorance. We also don't know what we're going to value when we're in the world." Now we are ignorant about who we're going to be, ignorant about what we're going to value. Maybe you're like, "Okay, I want something over the distribution of different values, right? Some kind of other form of it."

TYLER: I actually really like that. It ties up to some things I've been thinking about in another domain. If I didn't know what I was going to value in society, I'd probably want a policy that empowered everyone equally, because there's diminishing returns on power. What I'd want to do is basically ensure that people are empowered to understand and pursue their own values in society, figure out what they are, and then act on them. I could value anything. I could value screaming at purple or value my own demise. If you don't have a good way of having an informative prior on what all the different values are and how likely I am to value all these things, probably you just want people to be able to pursue their values. I don't know if it's connected to something else I'm thinking about around AI. If you're trying to figure out what to align superintelligence to and all these people have different values that are incremental and can't be combined, and they can't agree on them, then probably you'd want to do something like this. You kind of want to carve up the universe so that everybody gets an equal portion. They all get equal power, so they can just pursue whatever it is that they want. I like this idea. With expected value, I think this is a good point. The arguments for maximizing expected value are pretty good. You want to evaluate whether you should buy a lottery ticket. You calculate the probability of winning times the amount you win if you win, and then you subtract how much it costs. That's how valuable a lottery ticket is. For any gamble we take in life, we can think about all the different outcomes, the likelihood of each of these outcomes, and how valuable they are. This is our ordinary way of making decisions. "Well, I could do this career or this career; what's going to happen and how good are they?" I think this is a really good starting point, but there are some pretty nasty objections to expected value theory. One of them being that if you take it literally, if you're offered a tiny chance of getting an enormous amount of value, you should be willing to pay a huge amount for this. This is called fanaticism. You should be willing to give up a million dollars for a one in a million chance of a quintillion dollars. You should strongly prefer the one in a million chance of a quintillion dollars. A lot of people think, "No, that's not right."

SPENCER: My favorite version of this, which maybe should now be named after Sam Bankman-Fried, is, would you be willing to have a high probability of destroying the world. Let's say a 49% chance of destroying the world. If you had a 51% chance of doubling all the utility or something, the vast majority of people are like, "Hell no." But on strict utilitarian grounds, it says, "Well, maximize expected value." And then it gets even crazier if you get to play that game multiple times. Because, "Okay, well, each time you have a 49% chance of destroying all value in the world," and you, "Oh, I get to play again. Great." Every time you play, you increase the expected value, so you just play it until the world is destroyed with probability one. So in fact, that strategy is a 100% chance of destroying the world.

TYLER: I'm pretty hardcore. I'm pretty fanatical in my expected value theory. But that just sounds stressful. That doesn't sound like a great opportunity to me.

SPENCER: I think one of my more controversial philosophical mathematical takes is that we don't actually have great reasons to maximize expected value, except in specific domains. So if you're making small bets, let's say we're talking about money here, for example, and the amount you're betting is a small amount of your total bank account, there's a really, really good argument in favor of maximizing expected value, which is essentially that it will be the best strategy in the long term. You can kind of prove this mathematically, that other strategies will all be dominated by just always maximizing expected value. So that's really cool, and that's a really strong argument in favor of doing it for small bets. Now, if you're making large bets, let's say you're making bets that are a proportion of all your savings, like you're betting X percent of all your savings every time, you can actually show that if you use the standard expected value-based theory, there's a very high likelihood in some of these situations that you'll end up bankrupt. In fact, in some of them, you'll even end up guaranteed to be bankrupt with 100% probability in the long run as time goes to infinity. Even real gamblers don't use maximizing expected value. In those cases, they switch to something like the Kelly criterion, which is actually not maximizing expected value. It is an expected value of sorts, but it's a different quantity.

TYLER: Yeah, I find these arguments pretty compelling. But maybe you discuss this with Nick Beckstead, who has the best arguments on this topic, but the alternatives also look pretty grim. There's not a generalizable decision-theoretic approach that you can apply that just gets all the cases the way that you want them to; they all end up with something pretty awful. For example, I think this is from the Beckstead and Teruji Thomas paper, having an arbitrarily sharp cutoff, where if you're 1% likely of getting something, that's awesome, but if you're 0.99% likely to get that thing, you don't care about it at all. A lot of different ways of avoiding this kind of fanatical problem end up biting something really nasty bullet like that.

SPENCER: And I think what you're describing is trying to patch utilitarianism to deal with this kind of objection ends up making sort of an uglier theory that's maybe not even better. My view is we're on the wrong track. We shouldn't be trying to patch the theory. So, you were too far away from what I think is the right viewpoint. I also want to make another argument for utilitarianism.

TYLER: Hold on. Hold on.

SPENCER: Yeah, yeah.

TYLER: But it's not just utilitarianism. This is your personal finances. This is if you're going gambling. This is if you're trying to be prudent and make good choices for yourself. You're always thinking about consequences and applying expected value theory. If you've got an alternative expected value theory that looks good, that you endorse all the way down, that you're like, "Oh yeah, this Jagged Edge actually fits my moral compass exactly, I want to flip the table and interview you for the next hour about this, because that sounds amazing."

SPENCER: Yeah, it is hard to come up with an alternative. What I would say is that in real empirical matters, humans care about risk. They don't just care about the average value. When we start talking about what people care about and how to build a theory based on that, I think you've got to take that into account. If you do pure expected value maximization, even with just dollars, where the only good is money, you find out very quickly that you've got to take risk into account as a separate quantity.

TYLER: One great argument for that with money is money has diminishing marginal value, which isn't the case with utility, but yeah, you're right. Most people are risk averse.

SPENCER: But that is a great point. With real money, every extra dollar you get is worth slightly less to you than the dollar before, because the first dollars you're going to spend on food and shelter, etc., and eventually you're spending it on some random thing on Amazon you don't really need. True, but even if money didn't have diminishing marginal returns, I think you would find that people care about risk anyway, and that kind of raises, and I don't know how to get around that.

TYLER: Yeah, most economists and philosophers just adjust expected value theory for risk. I'm not going to be able to articulate it rigorously. Lara Buchak has done some of the best work on this for people who are interested in it, but you kind of take the bad outcomes and weigh them more, and this gets to expected utility theory that's weighted by risk, but that still runs into fanaticism and all these other problems that you're talking about.

SPENCER: Yeah. So I think one way that can be done is you say, "Okay, well, let's suppose that people care about risk. Let's say people have diminishing marginal returns, etc. You just rewrite the utility function so that their utility function now reflects that," and now you can still say, "Maximize the expected value of utility." But the weird thing about that mental move is it sounds like a fine thing to do, but now actually the utility function is no longer necessarily utility in the utilitarian sense. I think this is a really common source of confusion: there are two different meanings of utility that we mean, and they are not the same thing. Utilitarian utility in the utilitarian sense means something very specific. It's like the difference between pleasure and suffering. Utility in the economic sense refers to some arbitrary function describing an agent's value or what it's seeking to optimize, which could be anything.

TYLER: That's right. Utilitarians want to define utility in a particularly rich and loaded sense where it's like one utilon, and it's like an atom of pleasure or something, or it's a privileged scale that you define beforehand. Economists, for example, are just mapping functions to people's existing values. So economists are just mapping functions to people's existing values, and so not using it in the same way.

SPENCER: Exactly. Because if you're an economist, the utility function could be anything. Your utility function could be that you just care about accumulating popcorn. That's the only thing in the world you care about. You don't care about happiness; you just care about popcorn. And that's fine from an economist's point of view. I will mention that sometimes people try to justify utilitarianism using economic arguments and theorems from economics, and I think they kind of fail for this exact reason: those theorems are about utility functions broadly, and so I don't think they work that well for supporting utilitarianism as a theory. I do want to give one more argument in favor of utilitarianism, which I believe Peter Singer makes, but certainly other people make it as well. When you reflect on your life, one thing that almost everyone knows for sure is it's bad when they themselves suffer and it's good when they themselves feel happy. That kind of gets us off the ground; we all know our own suffering is bad and our own happiness is good. To the vast majority of people, that can extend to the people around them, at least their loved ones. But we could say that, "Most people will agree that to make something a moral theory, rather than just a theory about self-interest, it needs impartiality." You can't be like, "Well, it's immoral if Bob does this, but it's fine if Sally does this," if every single thing about the situation is identical. It can't depend on who you are. If you take this idea that we all know our own suffering is bad and our own happiness is good, and then you add this idea of impartiality, it gets you pretty quickly to something like utilitarianism, which says all suffering has to be bad, and all happiness has to be good. Therefore, what we should be trying to do morally is maximize it for everyone. It's a very short step from there to say, "Well, what does that mean?" Let's take the sum; that's a very natural way to combine it, sort of the most obvious way to combine it. Now you're pretty close to Classical Utilitarianism.

TYLER: Yeah, thanks for raising that one. It's similar to Harsanyi's argument. Utilitarianism is generalized prudence. "What do you want? You want welfare. Now, be impartial about it." It's not as simple as that, but it at least gestures in that direction. I want to say that the real argument for utilitarianism is that you come to my graduate seminar at Rutgers, and I sit you down for 14 weeks a semester, and we walk through every single argument in ethics and every thought experiment. We look at empirical psychology, and we march through the entire history of ideas, testing every single moral hypothesis. We look at stuff like the ones we've talked about: the veil of ignorance, the coincidence problem. We look at thought experiments like the trolley problem: should you turn the track on one person to save the five? We march through all these kinds of thought experiments and ideas and try to get rigorous about ethics. That's what I did for 10 years, and I was slowly dragged, kicking and screaming, to utilitarianism from a very conservative evangelical Christian morality.

SPENCER: Can you elaborate on that? Why does it get you there? Why do those kinds of thought experiments get you to utilitarianism? There are also classic thought experiments that seem to violate people's intuitions.

TYLER: Yeah, there are certainly arguments that violate people's intuitions. People don't want to turn the trolley onto one person to save the five, and utilitarianism recommends that you should. They don't want a doctor to cut somebody up and redistribute their organs to the five people in a hospital who are waiting for an organ and would die without it.

SPENCER: Just to clarify those two cases, I think people will allow those things in some cases, but not others. For example, people will say, "If you just have to flip a switch, it's better to save the five, even if it means one dies. But if you have to throw someone off a bridge in order to save the five, then you can't do that." And for the doctor case, it's fine for the doctor, with the patient's permission, to use their organs to save other people's lives. It's not fine for the doctor to prematurely kill the person so they can use their organs against their will to save the five. So it's really conditional in the context.

TYLER: Yeah, thanks for clarifying. I'm so utilitarian. I just forgot about these things when you write up the thought experiments in certain ways. So you're pushing somebody off a bridge to stop a train, you're carving somebody up without their consent when they thought they were just coming in for a routine procedure to redistribute their organs to save people's lives. People really reject the utilitarian hypothesis when you raise examples like utilitarianism says to value arguably, says to value insects more than humans, because there are so many of them that could be suffering. People really want to reject that. And so I think arguments point in different ways. You get arguments for utilitarianism. You get arguments against utilitarianism. And, as I'm saying, it would take a whole graduate seminar or longer to march through all of them. But what I'm gesturing at is, if you're a person who's been kind of thrown into the world, and you've got to figure out how to act, you've got all these different motivations pulling in different directions. You're convinced by these different kinds of arguments. You find different ideas compelling and different ones not compelling. If you want to have a value system that you understand and can apply and that's consistent, you want to start reasoning through all these cases and try to figure out what your true values are. And I don't think everyone — I'm an anti-realist like you, Spencer — I don't think everyone who reasons through these cases is going to become a utilitarian. I think maybe there's a surprising amount of people who are like me and Mill and Bentham who, when you march through all of the thought experiments, are like, "Oh, wow. The thing that most fundamentally explains all of my intuitions in the neatest and most compelling way is Classical Utilitarianism," and all these other theories look like they have really had to bite really bad bullets, like arbitrariness, that just really sucks, and I don't think I endorse. So that's why I'm saying the real argument for utilitarianism is a philosophy PhD.

SPENCER: Or maybe the real argument is that it's less sexy than the other guys.

TYLER: That is, yeah, utilitarianism is the best. It's the worst moral theory, except for all the others.

SPENCER: Now, when you say utilitarian, what do you really mean? Because people might assume you mean that you think there's an objective answer to what's moral, and it would be just as true to some alien civilization that we've never encountered as it is to us. Maybe you might think all rational beings, if they think about it hard enough, will all agree on the same morality, or something like that, or it's an objective fact about the universe that's moral. But that's not what you mean, right? So what do you mean?

TYLER: Yeah, so a lot of utilitarians are moral realists. A lot of utilitarians think that utilitarianism is objectively true. It's a fact about the world that we can discover. And they argue for this with arguments like they'll say, for example, "That if you just reflect, if you just introspect on the intrinsic nature of pleasure and pain, you'll figure out that these are the only actually valuable things." And I don't find these arguments compelling at all. So why don't I break down the different ways you could approach this? In general, you've got moral realists, some of whom think morality is kind of this supernatural thing, the non-naturalists, or to be more charitable, maybe they think about it like logic. It's not something we can find in the world, but it's something we kind of all agree and think is objective. You've got people who are naturalists who think it's like a truth, like in science, and we figure out what's morally true with our senses, like we figure out physics with our senses and instruments. And maybe our senses here are like emotions. Then you've got the Kantians who say, "There's just attractor states in morality, where, if you think long enough, you're just all going to be attracted to the same place, and we're all going to agree," and these are the realist constructivists. So I reject all of those arguments. I think you could give a full reductionist account of our moral psychology that doesn't refer to moral facts anywhere. It's just psychology. And if you can do that, then I don't know why you should think you should add morality to the picture. We can already explain all of your judgments with psychology, or we will when we get a better psychology. So why are you adding these moral facts? And I also think if there were these moral forms in Plato's realm, this golden moral function in the sky, I have no idea how we would figure this out. And no one's given a convincing account of how we figure out what was morally true. So I am an anti-realist. I think morality follows from your values. So when I say, "Utilitarianism is true," for example, I don't actually mean it's a fact that you can discover. What I mean is utilitarianism is the best representation and expression of my reflective, thought-out values, and I would recommend it to you. I'd recommend that you follow utilitarianism, and I like it a lot. These are the kinds of things that I'm saying here.

SPENCER: Do you think it reflects my values? Obviously you and I don't know each other well, but when you're saying I should be utilitarian, it seems like that's kind of an implicit assumption.

TYLER: Oh, I want you to be utilitarian because I like it, and so I want you to do good stuff.

SPENCER: Okay, got it.

TYLER: This is Harris's view. It's called Universal Prescriptivism. It says moral statements are prescriptions. They're saying, "Hey, do this. Hey, be a utilitarian," and I can do that without thinking it's actually what you value. Other than me wanting to coerce you into doing utilitarian things because I'm utilitarian, set that aside. Do I think that you would be a utilitarian? Do I think that if you did a PhD in Philosophy like me and were as smart as me, and marched through all of the arguments, you would come out a utilitarian and that it reflects your values? Probably not. I don't know. Maybe I'll give it one in 20. It seems like you've thought about this a lot. Maybe you've not thought about all the things I have, and maybe you change your mind. So one in 20 is kind of high, a 5% chance that actually you're just totally wrong about what you care about. I do think the arguments are pretty compelling, but that's 19 out of 20 times that you're not utilitarian.

SPENCER: And I'm sure you've thought about many cases I've never even heard of. You certainly have been on that point.

[promo]

SPENCER: Yeah. And so, I don't believe that there's objective moral truth, that there's some kind of universal answer to what morality is. My philosophy, which my audience has heard before, I'll just briefly recap. I call it valueism. It's basically the view that intrinsic values are a reasonable model of human psychology where we have these things that we fundamentally value for their own sake, not as a means to an end. Most of our values are not like that. Most of our values we care about because of what they get us. But if you keep pushing on, "Well, why do you care about that?" you end up with intrinsic values. There aren't that many of them, but there's way more than one, and they differ for different people. We were able to find in some research we conducted, we ran studies, we tried to get people's intrinsic values. We were able to find 22 categories of them, and each category has multiple within it, so things like nature, longevity, happiness, learning, and truth. These are different categories of intrinsic values. When I reflect on my own intrinsic values, I think that I really, truly do have more than one. While happiness is one that I definitely care about, and suffering is one that I definitely care about, I think I care about other things too. My life philosophy says, "Figure out what you intrinsically value and then try to take effective actions to increase it."

TYLER: That's pretty nice and neat. Something I'd be interested in better understanding is whether you and I disagree on or agree about because we agree on meta-ethics. We agree that these values are subjective. They ultimately come down to what you care about. They're not superficial; you've got to kind of investigate them and figure it out. Most people haven't done that. You probably agree with that.

SPENCER: Absolutely. I totally agree.

TYLER: You might be surprised by what you find when you dig down. One thing I'm curious about is, I take the view that if you do a bunch of moral reasoning, you might come to this really revisionist conclusion. When I was an undergraduate student, I went to a Christian college. I was an evangelical Christian. I thought that gay marriage was the great moral crisis of America. I just ate tons of meat, 4,000 calories worth a day. By the time I finished my undergrad, I was a utilitarian, a vegan, and an animal rights advocate activist.

SPENCER: I grew up in transition, yeah.

TYLER: I went back and forth a couple of times. Some of this is empirical, but I went from a Republican to a Marxist, and then became a more centrist liberal. Some of these are instrumental reasons I read Hayek and was like, "Oh yeah, Hayek's knowledge argument for capitalism is pretty good," but a lot of it is intrinsic, me changing, coming to realize that actually the only thing I care about is pleasure, for example. Based on our conversation from the past, I wonder if you and I agree about how radical moral reflection can be for your values, how radically it can make sense to depart from what you naively think you care about, and whether we agree on the methodology to figure out what you care about.

SPENCER: Yeah, I think that it can be a radical change. First of all, some intrinsic values depend on facts about the world. I think many people have spiritual intrinsic values, but I think that some of those, if they found completely convincing evidence that they were wrong about the nature of spirituality, would disappear. Take someone who has a certain conception of God, and they think it's the best thing in the world to worship God, and then they become convinced that God doesn't exist. That value is probably going to disappear. I also think we can very easily think our values are one thing because we haven't reflected on it and just be wrong about ourselves, making an empirical mistake. A good example of that is people who have lived much of their life doing what other people wanted from them. They can start to confuse other people's values with their own.

TYLER: Yeah, and people, I guess, defer a lot to authorities, initially to their parents, maybe to their religious communities. My slightly cynical view, but I think a fairly well-validated view on human nature is that the best way to predict people's actions is to look at the local norm; you just look at what's normal, and people are doing that.

SPENCER: Very good predictor of people's behavior, what does the local norm say?

TYLER: If that's true, then maybe a lot of what people think of as their values are actually what other people's values are that they're kind of modeling or mimicking, because it's socially successful to do so, or because they don't have the self-confidence to dig inside themselves and stand up for their own values, and instead want to defer to authorities' values. If that's true, then people could just be massively wrong about what they care about.

SPENCER: Yeah, and I think many people are mistaken. I don't think it's quite as bad as it sounded in the statement you made, because I do think people have some self-insight, but I agree. I think a lot of times people are confused, and it's part of why we made our intrinsic values test, to try to make it easier for people to reflect on their values, to give them a standardized process. If anyone's interested, you can go on our website, clearerthinking.org, search for the intrinsic values test. Even that is still somewhat challenging. It doesn't make it easy, but it at least gives you a process to go through.

TYLER: Yeah, it's been a while since I've done your intrinsic values test. I wonder, just trying to seek a point of disagreement, something we could find between the two of us. I think if I looked at your intrinsic values test, I might think, this is kind of a shallow way to get at your intrinsic values. That's not knocking your test. You're the first person to try to do this, and it's an important project, and as you said, it's hard to do. But I'm here thinking, "If you want to know what you care about, you have to get a PhD," which is, on the one hand, a very elitist and esoteric perspective, but then it shows how much digging and how many arguments you might have to engage with if you really want to deeply get in touch with what you care about. Maybe this is just a typical mind fallacy. Maybe I'm just thinking, "I had to do that, and so I think you do," but that's kind of where I end up, thinking that if you just took a...again, Spencer, I haven't looked at your test in a long time.

SPENCER: Let me summarize how it works. Essentially, we first teach the distinction between intrinsic value and instrumental value. We quiz you on it to help ensure you understand it, and we correct any misunderstandings you have about it. Then we ask you about a wide range of potential intrinsic values, and for each, you reflect on, "Is it something I don't value at all? Is it something I value but not intrinsically, I only value it because of what it gets me, or is it an intrinsic value and if it's an intrinsic value, how strong is it? Is it a major, important one to you, or a more minor one?" Now that process has errors that you could make in it. You could be wrong during that reflection. But I think what it has going for it is that, first of all, we try to ensure you understand the distinction. Second, for each particular value we have, you reflect on them separately and really consider whether it's intrinsic, instrumental, or neither. I think it's better than the process that the vast majority of people go through, although they could still be making errors.

TYLER: Yeah, that makes sense. I want to applaud you genuinely for making a big improvement on the way people might already be thinking about their values. The concern I have about this is, if I took this as a college freshman, I would have been like, "Oh, yeah. Intrinsically, I hate homosexuality; it's intrinsically disgusting. I intrinsically value God and nothing else." Some of the things you're mentioning: if you realize that you are spiritual, if you come to change your view on a spiritual matter, your intrinsic values might change. I also would have said things about intrinsically valuing liberty in a particular libertarian, Republican sense. But once I marched through the arguments in my ethics class, it turned out I either didn't value those things, or I really changed what I valued, which, if true, is quite a striking thing because from the perspective of my former self, I never should have taken those classes, because I really drifted off course.

SPENCER: Maybe this is a good opportunity for me to challenge your view a little bit. When people tell me they're utilitarian, especially when they tell me they're an anti-realist, that they don't believe there's an objective answer, they just think they're describing their own values. My first question is, "Okay, but are you really describing your own values?" Because if you're truly utilitarian, it means, for example, that you don't actually assign higher value to your loved one's happiness than you do to the happiness of an insect, as long as the amount of happiness is constant. I'm not sure I believe you. What do you think about that?

TYLER: Oh, man, if you had me on here three years ago, I would have given you a much spicier answer, which is, "Yes, you shouldn't donate your organ to people; you should donate it to maggots. They'd get pleasure from eating the organ." I would have said something really spicy. But I think my views on this have changed a lot over the years. I now take the view. Let me be clear, I think probably I was in bad faith when I took that spicy view. That's grist for your mill; that's kind of what you probably would have told me. R.M. Hare, this utilitarian philosopher, published notes posthumously. In his posthumous autobiography, he says, "The dream opened up. I was surrounded on a mountain by the graves of other philosophers, great and small, who had the same ambition and thought they achieved it. I've come to see, reflecting in my dream, that ever since, the hardworking philosophical worms have been nibbling away at the system and showing the achievement was an illusion." He realized, right on his deathbed, that he thought he had proved utilitarianism to be objectively correct, but there were just too many problems nibbling away at it. If I may grandly borrow this, I've come to feel the same way in recent years. I built my argument for utilitarianism on a kind of tower of assumptions, and at the bottom of this tower of assumptions, my bedrock argument for utilitarianism was that if you march through all of these arguments for utilitarianism, you'll find that, if you're like me, you'll find that at the end of this process, the moral theory that you endorse, the theory that you cognitively endorse, that you say, "Yes, this is right," is utilitarianism. If you look at all of these different thought experiments, and you look at all these arguments, and you reason through them in a cognitive, rational, reflective sort of way, you'll arrive at utilitarianism if your brain is kind of like mine. From there, I thought, "I should do whatever I endorse doing." But over the years, I started to question this kind of bedrock assumption: why should I do what I cognitively, reflectively endorse doing? I have all these other appetites and motivations. I care about my loved ones more. One thing that really sped up this process was the death of my partner in 2022. This stirred up so many things that were deeply meaningful in a way I hadn't expected them to be. I started caring a lot about posthumous harms, like preserving her agency and pursuing the things she wanted to pursue — things that didn't really make sense on utilitarianism but seemed really important. From a certain standpoint, I can walk through a philosophical argument and be like, "Oh, yeah, the right thing to do would have just been insects or something." But that's not fully reflective of what I care about. What I care about is a broader set of values than what I rationally endorse. Nowadays, I think I have these two parts to my morality. It's kind of like an internal family system. There's this part of me that is rational and reflective, a hardcore utilitarian, and this part of me that is motivated by all these other things and cares about all these other things. They're just kind of roommates in the same brain, and I've got to figure out how to make them get along.

SPENCER: I think that's really reasonable. I think this tension between what you intrinsically value and what you rationally endorse might lead to disagreement. I favor intrinsic values over what people rationally endorse, but the way I propose valueism is not as a theory about what's true in the world, but as a life philosophy. I'm just saying, "Hey, I'm putting this out for your life."

TYLER: That's great.

SPENCER: Which kind of lets me off the hook because I don't have to prove it's true. I'm saying, "Look, this is a way to live. Figure out what your intrinsic values are and try to take actions to effectively increase them." You might say, "Well, I also want to do this other thing. It's not about my intrinsic values." I'm like, "Okay, cool." I don't know how to put that in my framework. Maybe you just have an intrinsic value around rational endorsement, like acting according to your rational endorsement. Maybe that's a way to synthesize it, and then it kind of comes together somehow. My main problem with the rational endorsement view is that I don't actually think you can rationally prove utilitarianism. If I did think it could be rationally proven, then maybe I'd be like, "Okay, I see it," but because I don't, I'm like, "Well, okay, I know you're convinced rationally, but I'm not convinced rationally."

TYLER: Let me say a couple of things about this. I definitely don't want to say you can rationally prove utilitarianism, like it's a mathematical theorem or a scientific finding that's bedrock. Proofs in mathematics depend on your assumptions. Proofs go through if you assume different axioms. I think people have different axioms in their moral bedrock. You can prove utilitarianism from a certain set of moral axioms, but not everybody's going to go there. When I say rational endorsements, I'm saying something really loaded. Oftentimes people say rational endorse to mean being rational in this very narrow, general sense that everyone must obey this. I don't really believe in that kind of rationality. To the extent that there is that kind of rationality, it just means-end rationality. You value something, do it. There's that kind of rationality, but I don't think there's this universal rationality. When I talk about rational endorsement, I'm not saying everyone has to accept this. I'm saying there's a certain style of thinking that philosophers do. When philosophers do this style of thinking, they end up at places like utilitarianism, virtue ethics, or contractualism, these systematic philosophies. I think for a bunch of people, maybe not everyone, one of your epistemic or aesthetic values is to go through this process of cognitive reflection. This gets to a challenge I wanted to raise to valueism, but I think you're too modest for it to bite. I worry valueism is too meta-ethically prescriptivist; you're prescribing, "Hey, look for your intrinsic values and follow them." But maybe when somebody takes a step back and thinks about as a creature thrown into the world, how they want to live and tries to reconcile all the different motivations inside them, that is actually where they end up. Maybe they end up like me, somewhere more like cognitive endorsement. There's almost a meta valueism; there's a higher level of values that we have that's not just our intrinsic values, but our values about what kind of people we want to be, how we want to navigate the world, and what kind of way we want to engage in moral reflection. You can't just say, "No, you're not allowed to reason in that way."

SPENCER: I think I escape this charge by saying I'm just throwing out valuation as a life philosophy. You take it or leave it. If you want a framework for living your life, consider this one. That's all I'm saying. But it also feels like it lets me off the hook a little too easily. My main pushback on the view of what you rationally endorse is that I think I just fundamentally don't get what the endorsement is based on. If you were to claim, "Oh, I think I can prove mathematically or empirically that utilitarianism is the right answer," then I would know what you were talking about. But if you're not claiming those things, what is the endorsement really fundamentally based on? What is underpinning that endorsement? Is it not just intrinsic values? If it isn't, which maybe it's not, then what is it? That confuses me.

TYLER: I think it's a good question. It's a good challenge for my view because as I started to spell out what cognitive endorsement is, I said it's what you look at and say, "Yes, that's to be done." I'm saying yes, that's to be done about both cognitive endorsement and these other kinds of motivations that I have. It can't be that. It would be interesting to have moral psychologists drill into this, and maybe they have. What I think is there's a paradigmatic type of psychological process that is considering moral arguments and having a warm glow, a positive yes. I don't know what that psychological process is. I tend to be pretty skeptical about philosophy of mind. Maybe I'll find out it's not real. It feels really real, but it's no more real than consciousness or beliefs, which are also fake.

SPENCER: I think there's a version of that that happens with intrinsic values where, let's say you're evaluating some thought experiment with someone acting in a certain way. At first, you're like, "Oh yeah, I think they acted okay." Then someone points out something about that case, and you might say, "Oh no, wait. Now I realize it's not okay." What have they really done there? They've shown you that not endorsing that case is more aligned with your intrinsic values than endorsing it, which you didn't realize before. Your intrinsic values are there, and then they've pointed out implications of what this case says about your intrinsic values, and you're like, "Aha, now I know better that I don't endorse it, or I thought I did." There may be other versions of that. When I talk to people like yourself, who say, "Well, I endorse utilitarianism," or "I'm convinced of it," I get a little bit stuck because I'm like, "Either tell me your logical proof, or tell me where that endorsement comes from, if not from that." I got a little stuck in that conversation. I think it's interesting to explore.

TYLER: There's a type of thinking that we do when we feel the call of duty to be an impartial, selfless person and do something, and feel tension in the other part of our life. "Oh, but I actually care about my family. I don't want to make a sacrifice," or "I care for myself. I don't want to take the sacrifice on." My hypothesis is these are two distinctive kinds of mental processes. Maybe they're shallow, maybe they're deep, but we're doing two different kinds of things here when we're feeling the pull of our selfishness or our family or our aesthetic values, and when we're feeling the pull of arguments that we see reason in and want to follow. Maybe I should say that kind of mental process is another intrinsic value in your value scheme. But maybe I should just say that you cognitively endorse valueism, and actually, you were subservient to me.

SPENCER: Isn't cognitive endorsement fundamentally dependent on the strength of the argument? Wouldn't you say that if you cognitively endorse something, but it turns out that the logic behind it is flawed and the thing's not really true, then you're making a mistake with your cognitive endorsement?

TYLER: I don't think that moral arguments have inherent directions. You have premises that lead to a conclusion, but if you reject that conclusion, you can just go back and deny the premises. I don't think these arguments have inherent directionality. I think that the directionality depends on what you cognitively endorse, or maybe to say something less loaded, it depends on your motivational pull of various kinds. You can be wrong, you can be in bad faith, or you can just have not considered an argument or hypothesis. You may not notice a tension between two different kinds of arguments that you need to reconcile inside yourself. When I say you rationally endorse something, I don't mean you've gotten all the arguments right or that you got an A on the test. I mean you followed the pull of your heart on all these arguments.

SPENCER: Yeah. Then I'm just not sure it isn't just another form of intrinsic value. Because if it was based on truth, if it was a cognitive endorsement, it's basically like making a truth claim and saying, "I believe this truth claim, therefore I cognitively endorse it." Then it would sort of hinge on the strength of the arguments. You would say, "Well, if I made an error in my argument, that could completely undercut it," but it sounds like that's not what you're saying. There's something more, what I would describe as value, going on with the cognitive endorsement. Anyway, I don't think we're resolved today, but it's interesting food for thought.

TYLER: Yeah, I'll subordinate you to my principal. You subordinate me to yours, and we all get along that way.

SPENCER: Sounds good. Agree to subordinate each other or something. Before we wrap up, the final thing I want to ask you about: How do you think about utilitarianism with regard to work in AI? Because you mentioned that you think that has implications there.

TYLER: Yeah. So, you know, Spencer, that my career is in AI safety and security. What I do when I'm not talking about metaethics on your podcast is I help philanthropists understand AI safety and security so that they can contribute and make AI go better. This is connected to my utilitarianism in a pretty significant way. I got into this because I'm utilitarian, because I wanted to systematically figure out what I could do to try to make the world as good as possible. I consider lots of things. I was an animal rights activist for a while and was persuaded by the argument that there are just so many animals and you can help them so easily that you should clearly just spend all of your time helping animals. There are 100 billion chickens, I'm going to get the numbers wrong now, and trillions of fish. Five cents can save a life. The utilitarian arguments for animal rights were really strong. Now I work on AI because I've increasingly, frankly, since the release of ChatGPT and the things starting to heat up in AI and progress starting to get really fast, thought that if you want the future to go as well as possible, then being on the cutting edge of the most potentially important technology, which could shape the entire future, is just an enormously important thing for you to do, whatever you care about. In particular, I focus on AI safety and security. I mentioned I'm an anti-realist about morality. I don't think that we're going to build smart AI systems, and because they're smart, they're just going to figure out what's morally right. I don't think there's an answer to that question, let alone one you could figure out. I think morality is this incredibly complex thing that we disagree about and that we evolved over millions of years. We can't even write it down. If you want to try to get an AI to understand this by prodding it in various directions, reinforcement learning and saying, "No, do this. Don't do this." It's really hard to do that and get it to really internalize human values. I'm concerned that as artificial intelligence gets very, very powerful and starts making most decisions instead of humans, because we trust it, because it's very economically useful, because it's very powerful, and takes power from us, whatever reasons, AI eventually just starts making most decisions in society, which I think is coming. I'm very concerned that we won't have high assurances of alignment to human values because so far, we're failing at this. Even if we do have high assurances of human values, I think it's just a really hard question: what value should we align AI to? Spencer's valueism? Tyler's utilitarianism? Some democratic regime that takes everyone's values into account? If so, which democratic regime? There are about 300 of them, and voting theorists have come up with a thousand more, and very few people are thinking about this question of what we should align AI models to, if we even succeed in making AI aligned with human values. I think this is just really important. Please go ahead.

SPENCER: Just on that point, this might sound super hypothetical, but actually, as soon as you have AI agents where you give it a goal, and it goes and does things in the world, like send emails for you, log into accounts, etc., effectively, it will be doing things that are making judgments about values, and it's either actually doing it, or it's doing it implicitly, without anyone being aware of it. Because if you say, "Hey, here's my trading account, go make money," it has to make judgments about, "Okay, can it also go on Twitter and spread lies about the company so it can short it and make money?" Suddenly, as soon as there's access to your Twitter and your trading account, that's a choice it could make, and whether it makes that choice or not is going to depend on its implicit values. Additionally, it's pretty likely that you didn't tell it, "Hey, by the way, when I said, go make money for me in my trading account, I don't want you to specifically do this thing by going on Twitter and spreading false rumors."

TYLER: So that's exactly right. Whenever we say anything or take any action, we're kind of implicitly embedding just hundreds of different values about humility and deference, about epistemics, about niceness and meanness, about our political values, about who we trust and who authorities are. We're embedding all these different values in every sentence we say about efficiency and brevity and in every action that we do. So they're already embedding lots of values. Anthropic and the Center for AI Safety have started the study of what the values of these AI systems are. But as you say, as companies are trying to build agents that can do work, because it's incredibly economically valuable to replace human workers with AI systems that can do this work, who can trade, who can code, especially, if you are economically incentivized to create these agents who do human work, then you increasingly replace humans with AI in the marketplace, and you increasingly substitute human judgment for AI judgment. The more we do this, the more we put AI in the workforce, the more we have politicians asking ChatGPT for policy advice, which I'm sure they're already doing. The more these values are going to matter and shape the entire trajectory of our future.

SPENCER: Yeah. These seem like very important questions to me. Maybe you could just tie this back into your utilitarianism. How does that help you make decisions around the AI topic?

TYLER: As a utilitarian, I am particularly focused on risks that seem like they will have a very, very large effect on a lot of individuals. Some of the things that I'm concerned about are AI destroying the entire world either by taking over and being misaligned with our values, or by speeding up Cold War era dynamics like great power war, development of doomsday weapons, and so on, because the destruction of the world would, from many perspectives, simply be the most valuable and important and most terrible thing that could happen. I also think about things like, what's the long-term trajectory of these values? How can we ensure that these values stay in line with what we want for a long time, and maybe even improve on our values? Even if there's no objective morality we agree on, maybe they can do a better job finding out what we care about than we can and make the future a really good one from our perspective. A last kind of set of questions that I'm really interested in as a result of this is AI itself as a moral patient. We have no idea what consciousness is, even in humans. Dan Dennett has some great examples of cases where, "I have no idea if I'm conscious. I have no idea if I'm conscious of which way is up. I have no idea if I'm conscious of left and right. I don't know if I'm conscious when I'm driving and zoning out. I couldn't tell you at all clearly what consciousness is, even from a first-person perspective. Now you're trying to not only do neurology and figure out what consciousness is in the brain, but you're looking at computer lines of code of an AI system that's acting just like humans are, and you're trying to figure out which lines of code are conscious." That just seems like a crazy problem. There's infinite variation in computational space. You can write all kinds of different programs, and how are you going to decide which ones of those are conscious? I think making sure that we don't hand over the universe to unconscious AIs or suffering AIs, or upload ourselves and we're not conscious, are really hard and critical problems that we have to get right, even if they sound insane.

SPENCER: We talked a lot about utilitarianism today from different perspectives. What would you want to leave the listener with as kind of a final note?

TYLER: Utilitarianism is probably more plausible than you think it is. Same as Spencer, I bet there's a one in 20 chance that if you reason through the arguments, you would end up finding it to be a very big part of your value system. Actually, I'll have that one half that you would come to think that this is a really important part of your morality, even if not.

SPENCER: And I would say that too about myself. I think I'm more utilitarian than the vast majority of people, and it is a useful carve-out. It's not my whole value system, by any means, but it's a useful carve-out to think about and a powerful piece of my value system.

TYLER: Yes, and if that's true, then part of your life should be dedicated to actually trying to figure out how to make the world as good as possible. And doing that, you can read some more arguments on utilitarianism.net. A great textbook that I contributed to, but just generally, I really endorse. You can follow me on Twitter, @tyler_m_john. If you're interested in AI safety, I would check out BlueDot Impact to learn more and 80,000 Hours. And if you happen to have $8 billion and want philanthropic advice, my Twitter is always open to chat about effective ways to spend that.

SPENCER: Tyler, thanks so much for coming on.

TYLER: Absolutely. Thanks. This was really fun.

[outro]

JOSH: A listener asks: "What's the professional accomplishment that you're most proud of? And what is your greatest professional regret or shame?"

SPENCER: So I tend to not be somebody who really feels pride significantly or very often; but if there's one thing I'm most proud of, it's the work at Clearer Thinking where I'm happy with what we've created. And I do feel like we've done work that a lot of people appreciate and a lot of people say benefits them. And so that's that's a really nice feeling. In terms of shame, I also don't really feel shame very much. I'm just not that sort of person. But I've certainly failed a lot of things. I've created projects that failed, and I expect to keep creating projects that fail. I don't know if I'm ashamed of that. Obviously, you know, I wish I had not failed those things; but I don't know if something to feel ashamed about.