JOSH: Hello, and welcome to Clearer Thinking with Spencer Greenberg, the podcast about ideas that matter. I'm Josh Castle, the producer of the podcast, and I'm so glad you've joined us today! In this episode, Spencer speaks with Adam Mastroianni about cautionary tales and bright futures of psychology research, misunderstandings and takeaways from the replication crisis, and the pains of peer review.

SPENCER: Adam, welcome.

ADAM: Thanks for having me.

SPENCER: Is the field of psychology totally messed up?

ADAM: [laughs] No, no, no, it's not. If it keeps doing what it's doing, yes, but if it changes, then no.

SPENCER: So there's still hope?

ADAM: Yeah. I think in the same way that we're in our prehistory, and just like you could have been looking at the field of alchemy in the middle of the 1500s and thought, "I don't know, man, it doesn't look like it's really going anywhere," but eventually it did. I think we're in that same period.

SPENCER: So does that mean we're in a period where we're learning stuff, but we just have so much left to learn? Or does it mean we're in a pre-science period where most of what we think we know needs to get discarded?

ADAM: I think some of both. So we are discovering some empirical truths that might end up being useful later. For instance, a pretty well-established fact is that people seem to not be very good at predicting their emotional reactions to things. This is the lab that I came out of. For about 10 years, my PhD advisor was running these studies where people would predict how they would feel after certain things, like how will you feel two months after the presidential election when your favorite candidate loses? People would say, "Oh, I feel terrible." And then they feel fine. They feel normal, the way they always feel.

SPENCER: And do they feel normal immediately after? Because you might think that they might be very upset for a few minutes or a few hours.

ADAM: No, exactly. That's why it matters that we're talking about the event that just happened, and now two months have passed. Some amount of time has passed, and this is the mistake that people make — that they think that the way they feel right away is the way they will feel for a long time — when, in fact, you got to go back to doing what you do, and doing what you do normally makes you feel normal. That empirical regularity — that there is some mismatch between the predictions we make about our emotional states — I think that will both remain true and will ultimately help us figure out what psychology is. So it's not impossible to make progress or do useful things at this stage. It's just really hard, because we don't yet have a way of figuring out what are the things that matter and what are the things that don't. That's where I think we're at right now.

SPENCER: It seems to me that we know a ton about human psychology, but part of the challenge is that naturally, people know a ton about human psychology. We're all natural psychologists, and folk psychology that we kind of apply every day is actually pretty accurate. Yes, there are lots of interesting ways it's wrong, but it's pretty darn good. The bar is pretty high. We have to actually beat our own intuitions.

ADAM: I think that's part of it. This is true for any science. For humans to survive, they need pretty good lay intuitions about how lots of things in the world work. Although it is possible to show all the ways that our folk intuitions about physics go wrong. If you show people a diagram of a spiral, like a coiled tube, and you're like, "If I put a ball in this tube and it ran through the tube and came out at the end of the spiral, what would happen next?" About half of people say it would continue to spiral. It doesn't do that. It goes in a straight line. It doesn't get endowed with spiraling motion. We can identify the ways that our folk intuitions about physics go wrong. However, everybody knows that if you knock a glass off a table, it's gonna fall, and they'll be really surprised if it doesn't. People know enough intuitive physics to pilot their SUV at 85 miles an hour down a highway. I think that bar exists in a lot of fields. I agree with you that it is higher in psychology because we need to know so much more about the intricacies of humans than we actually need to know about any other part of the natural world, because humans structure so much of the world that we're trying to live in. I agree that we have a lot to overcome, but that is not unique to psychology.

SPENCER: It seems that subjects like physics also discover lots of areas where our intuition just doesn't work. We can't see things that are microscopic. We can't imagine situations that involve things in space. We've just never had an experience with that, whereas psychology is typically dealing with realms that we've all had experiences with.

ADAM: Totally. I think this is exactly why I think psychology has taken so long. Here's a mystery that I've been grappling with recently. You can kind of trace the beginning of modern physics to Galileo. You could round it off and say that in the late 1500s to early 1600s is when physics really starts. When does psychology really start? Kind of the 1860s. So why is it that there's 250 years in between the start of the modern empirical idea of physics and the modern empirical idea of psychology? I think it is exactly the reason that you just highlighted: it is a lot easier to get yourself into situations where your intuitions break down in physics than it is in psychology. In psychology, our intuitions are so strong, so convoluted, and so satisfying that it's hard to realize that we don't know what's going on. Whereas when you look at a piece of math describing some physics, you immediately know that you need to think in order to understand this. Whereas when someone is like, "Why do people differ from one another?" And someone says, "Well, some people are just raised differently." You go, "Oh yeah, that's it. Anyway, moving on." That's what I think is going on anyway.

SPENCER: Do you think staring at these places where our natural intuitions don't actually get the right predictions about human psychology has been a big source of growth for psychology and a lot of the progress that has been made?

ADAM: I think so, yeah. I think this is part of what got the field started in the first place. One strand in the history of science that leads to the beginning of psychology is actually astronomers trying to take measurements of movements of things in the sky, and they had to record things very quickly. They eventually realized that there's a reliable difference between how quickly someone can react when, I don't know, when you're looking at the sky, you see a thing pop up, you have to press a button. This is sort of the schematic way of thinking about this. Some people reliably take a tenth of a second longer to do that. This created something called the "tenth of a second crisis": "Oh no, we're not actually all measuring the same thing because some people are faster and some people are slower." That's actually some of the first empirical measurements for psychology. That's reaction time, which is now within the remit of what we think of as psychology. But we got it because we were doing something totally different. We could have started thinking, "Wow, I wonder if people can react faster or slower to things." That's not the way that we did it. We discovered this by accident, doing something else, and only then were we like, "Wow. Why would it be the case that some people have faster reaction times than others? Maybe there's something different going on in their minds." So yeah, I think it is when we stumble into these situations where we don't know what's going on that we can actually discover something.

SPENCER: It seems to me, a lot of the cognitive bias paradigm has true findings, and I don't think they all hold up. I think a lot of them do. Much of it seems to be violations of our intuition about the way humans work.

ADAM: Yeah, and that is also how we know that there's something there. How do you know which facts matter and which facts don't? When you go to run a study, you could study anything. You could be like, "Well, I wonder if people would make different judgments if we spun them around 10 times first and then made them take our questionnaire. If they do, what does that mean?" I have no idea. I have no way of assimilating that knowledge into a schema about the human mind. However, if you show me how humans should act by the raw laws of rational decision making, or how I expect them to act because of my intuitions, and they act in some different way, that's at least something that I know, like my priors are being violated here, and I need to update them. I think that's why that tradition has been so successful. I think that's why it's basically won two Nobel Prizes for the idea that humans have biases and deviate from the laws of rational and optimal decision making, because we can see what those findings meant. So many of the other findings that we're dealing with, we don't know what they mean. There's this paper that I think about sometimes about the implicit association between women and birds, and apparently, if you give people an IAT, they will be faster to associate women with birds than men with birds. What does that mean? Does that mean anything? I don't know. If you know, please tell me.

SPENCER: Isn't there an old-timey saying of "bird" for a woman? Am I making that up?

ADAM: Is there? I would believe you if there was.

SPENCER: Could it just be that?

ADAM: But the fact that we're both sort of uncertain about it, and I don't know if I've ever heard it before, makes me wonder, could that possibly be it?

SPENCER: Reddit tells me I'm right.

ADAM: I'm glad.

SPENCER: I don't know. Maybe it has nothing to do with that. Could it be a human similarity function of what is similar to what is very mysterious, right?

ADAM: Yes, and it's not clear to me that even if that's the reason, do we learn anything by running the women-bird IAT? There is an infinite number of things in the universe that we could run IATs on. What if it turns out that there's a closer association between men and rhombuses and women and pentagons? That could be — and we won't know until we test it — but what would it mean if we found it, and why should we run that study versus any of the other billions that we could run? This is why I think it's really hard to make progress in psychology.

SPENCER: If you think about there being sort of n different things you could study, and then you could say the relationship between any of the other things, that's already N squared. It very quickly becomes completely ridiculous. There's just way too many hypotheses out there. You need some kind of structure, and that's where theory comes in. You have to have some theory that's kind of pointing you to study certain things and not other things.

ADAM: Yes, yes. And this is why I'm not a fan and not an enemy of this new trend of doing mega studies, because I think it falls into this trap. A mega study is like, "We don't know which interventions are going to work, so let's run all of the ones that we think have any chance of working."

SPENCER: Yeah, see, I love mega studies, so I'd love to disagree with you on this. Let's get into it.

ADAM: So I'm interested in what you like about them. I'll describe them so everybody knows what we're talking about. Then I want to hear your pro-case before I give the con-case.

SPENCER: Perfect.

ADAM: So, an example of a mega study is we have people who go to gyms. A lot of people sign up for a gym and want to go more than they do. Can we help them go more often? Here's what we'll do. We will take all the possible interventions that we think will increase gym attendance and test them against each other. We'll run a giant RCT where some people will get no treatment, some people will get a baseline treatment where they get a few nudges, and then we'll run a bunch of things on top of that. We will send you a reminder when you lapse in attending, or we'll tell you the base rate of other people going — a bunch of things that I would say are lightly informed by theories that have come out of behavioral science. So that's an example of a mega study. I'd love to hear what excites you about them.

SPENCER: There are a few things that excite me about them. First of all, if we can dream and say maybe there are interventions that are cross-domain, not to say that they would work in all situations no matter what for all people, but let's say they are quite robust across many domains, then mega studies should be able to ferret them out. Because even though you're just studying gym attendance, if you've got a really robust intervention that's going to help improve habits broadly, then you should be able to find it in your mega study on gym attendance. So that's my first point, if you want to react to that.

ADAM: Yeah, I agree with you that they're a good way to find those effects. I'm not optimistic that those things exist, and it doesn't seem borne out that those things exist.

SPENCER: That's where you get a little of the dreaming that is required but that'd be awesome if we found such things.

ADAM: No, it'd be cool. To that point, it seems worthwhile to run a few of these, because it might be that there was some gigantic effect staring us in the face the whole time. Although, if there was, would we not have already run into it? If it was just, "Oh, all you need to do is give people a gold star every time they go to the gym, and they'll go to the gym all the time." And guess what? That also works for returning library books, and that also works for getting your tax return in on time. You just need the "Gold Star Effect." If we hadn't discovered that yet and a mega study could show that to us, that would be great.

SPENCER: But let's differentiate between large effects and robust effects that work across domains. I agree with you, if there was some super large effect that just sort of always worked, then, yeah, we would have probably just noticed it in daily lives. But let's say there's an effect that just makes you 8% more likely to succeed in habits. But it's cross-domain. It works across almost anything that's really useful, that could be used in all kinds of ways, but it might not be that easy to notice, because we're talking about an 8% increased effect.

ADAM: Although, in this case, do we actually get that out of a mega study? Because the thing that we're varying in the study is the intervention, not the domain. So rather than run the same intervention at the gym, the library, the yoga studio, or with tax returns, we run a bunch of different interventions in the same situation. So we're not going to learn how robust they are. We're going to learn how they compare to other interventions run in the same situation.

SPENCER: But the point is that if the robust ones do exist, we should be able to find them in pretty much every mega study that includes them. So the fact that it's in the gym doesn't matter, because if it's really robust, you're going to find it at the gym, you're going to find it at the library, etc. So one mega study points us, "Hey, look. Here are three things that worked. Maybe this will work across the board," right?

ADAM: I see. So the biggest study doesn't give you that second part. It can identify out of these 53, here are some candidates to run in a different situation. And what makes me despair is the number of different situations. If all we know is that this worked at the gym, why should we expect it to work at the library, and why should we expect it to work with a tax return? How do we know? If we don't know why the effect works and the parameters of the situation in which it works, we have no way of knowing that it's going to work over here and not over there.

SPENCER: I think it depends on what we find. So let's talk about this specific mega study before I go into my second reason why I like mega studies. This mega study that you brought up makes me laugh, to be honest, for reasons that will become clear in a moment, which is that they tested this incredible range of things to see if they increased gym attendance. And you wanted to say how you interpreted the paper when you read it.

ADAM: So I'm already reading this paper skeptically, and I think there's a trap you fall into where you find the first hidden thing and think that you have found all the hidden things. This paper, I think, got a lot of attention and praise for many good reasons. My take on it was, this is a really nihilistic way of doing science; there's no theory here. We have no hope of knowing what's going to work. In fact, the study confirms that we do not know what's going to work because they ask people to predict it, and they can't. So it means that for all we've been doing for the past decades, we cannot predict beforehand which interventions are going to make people go to the gym, and that seems like such a low bar, and we're not clearing it. That's why I see this study as a cautionary tale rather than as a triumph. By the way, half of the interventions don't work, but we don't know which half that's going to be beforehand. My takeaway from it was, "Oh, half of these interventions didn't work, and we didn't know which ones. That's pretty disappointing, and we should all think really hard about what that means for progress in science." But I missed something. You want to tell me what I missed?

SPENCER: Well, so I saw you talk about this in a blog post. The thing is, if you look really carefully at the study, they tested tons of things. They tested 50 things, and you come away thinking, "Okay, maybe half of them worked." It seems pretty depressing because they didn't have any ability to predict it. But it's actually way more depressing than that, because I argue that they misanalyze their data. In fact, if you analyze it properly, you find that only four things worked, and probably at least one of those is a false positive.

ADAM: Could you go into the analysis that they present as the headline and the one that you think actually reveals what they did because I think this is an informative distinction?

SPENCER: If you read the paper, they compare their interventions against two different things. There's a control that involves nothing, where they're really not trying to improve your behavior at all, and there's a second control that involves a standard package of things, like reminders and incentives. They have these two different controls. The thing is that their interventions also included the standard package, so it involved incentives and reminders, etc. This raises the question, if you're trying to decide if one of their interventions worked, what do you actually compare it with? Do you compare it with the control where you provide nothing, or do you compare it with the control that had the standard package of incentives and reminders? Their headline result actually compares it with the control when you do nothing. The problem with this is that their interventions all included the standard package of goods, and they additionally prove that the standard package actually works. Basically, they're doing interventions that have a specific intervention plus a standard package, and they're saying, well, that beats nothing. Of course, it beats nothing, because we know the standard package beats nothing.

ADAM: So it's kind of like running a study where you give everybody the vaccine, and then some people you give a vitamin, and some people you give a Pedialyte or some drink or something. You're like, "Look at how many of our interventions made people better off than the control condition where you get nothing," except everybody got a thing that we know works.

SPENCER: Exactly, very well said.

ADAM: This makes me even more despondent that, in fact, it's not that half of things worked; it's that there was a baseline package of things that worked and then very few things that seemed to work on top of that. This is, I think, the top of the line in terms of psychological research. This is as good as it gets. These are serious people with a lot of resources doing this work really carefully. This is our best, and I think it should really make us rethink what our best is and how to do better.

SPENCER: See, it excites me, though, because they only found a few things that worked. If there are these universal interventions out there, we only have a short list here. I think it's pretty exciting, even though I found the study depressing overall. But let's just talk about what they found worked, because I think people might find that fascinating.

ADAM: Yeah, go for it. You know better than me.

SPENCER: One intervention that seemed to work is giving people bonuses for turning in after missed workouts. So basically, imagine you go workout, you miss a workout, the next day, they give you a special bonus because they're giving you points, which you could redeem for rewards or money, something like that. They say if you give extra bonus points for recovering from a mistake, that actually helps people stick to their habit. I thought that was pretty cool. It's not so mysterious. You can totally see how this could help people. The causal story is clear; whether it actually will work in the future on other tests, we don't know, but it's an interesting hypothesis.

ADAM: That's great. I'm happy we learned that. I think the value of learning is going to be really limited, right? Now, if I run the New York City Library System and I want to know how to get people to return their books on time, it's not like I can pick this up and implement it. I'm not like, "Okay, now what I need to do is remind people to turn in their books when they don't," because I don't actually understand the underlying system. What was it that happened that made that work? What are the mechanisms of mind that this is working on? How can I hit those same mechanisms in a different situation? It seems like all I've learned how to do is make people come back to this gym. It's not even clear that it would work for a yoga studio, like missing a yoga class instead, or if we go to Slovenia, would it still work? Fifty years from now, would it still work? These are the things I think we really want to know, even though, obviously, in the meantime, it's great that more people go to this gym. Will we be more effective at making people go to other extremely similar gyms in the future? Maybe, and that's better than nothing, but it still doesn't seem like very much to me.

SPENCER: Exactly what I got to test. We went through the river, we found a few nuggets of possible gold, and let's go test them and see if they're actual gold or if they're just trash. If it only works for this one set of gyms, obviously, it's useless. But if it turns out, "Oh, wait, this could be applied to all kinds of things." I don't think it's that mysterious how to try it. The library could, when someone misses their deadline for turning in a book, say, "Hey, we've waived your fee. You've got two more days to turn it in. Now we're going to give you this special thing where you're not going to have to pay anything." Maybe that won't work, but that's a pretty clear application of the concept. They could try it out.

ADAM: Yeah, they could. But now we have to run this study every time we want to know whether it works or not. The real valuable knowledge is the knowledge that we understand so well that we know whether it's going to work or not because we have the model down. Every time you want to run a chemical reaction, you don't have to test it to make sure that the reaction happens the same way that we know it is supposed to happen. It could be that you got the purities wrong or something like that, but the underlying model has to be the same. This seems to me like we are trying to discover the Periodic Table of Elements by testing each one against each other, and now we're going to spend the rest of time doing that because we have no way of knowing beforehand which elements are going to react with which other elements, or if you change the temperature and pressure, will it change? The only way to get that kind of knowledge is to understand the underlying system rather than just go with this pure, brute force empiricism approach.

SPENCER: Yeah, I guess where I get more optimistic is I think the brute force empiricism, while inefficient, could still find interesting nuggets, and then you're like, "Aha, that's something I can kind of dig into further and see if it's actually a true regularity or if it just disappears again." Maybe it's just a thing about this one moment in time and the specific case, but maybe it's a regularity, and that's what we want to find. There are regularities that we can find again and again.

ADAM: I don't think we're going to succeed very much in finding those without some kind of theory of how the system works. For instance, if instead we were trying to fix cars, when a car breaks down and comes in, we don't know anything about the components of the cars, but we open it up and take this thing out, put another thing in, and now the car works. Every time another car comes in, we do the same thing. Sometimes that works and sometimes it doesn't, but we have no way of explaining why replacing the spark plugs on this car worked, but on the next car it doesn't. We want to be able to understand when a car comes in what you need to do in order to test: is it spark plugs? Is it the carburetor? Is it a chip? So that we know which of these interventions to do before we just start taking the spark plug out, replacing the carburetor, or giving it a new chip. That is the measure of understanding. In the meantime, we can make extremely modest progress by doing it the other way. But look at what it took just to get this far. This study, I don't know exactly how much it costs, but this is gigantic from the point of view of running a psychology study. We can't run that many of these, and so it's going to be really hard to then try to test it in another situation because now we need to do 53 more of these in another context to make sure that what we found holds. That's why my take is pessimistic rather than optimistic.

SPENCER: Yeah. Whereas I feel you need a set of facts from which to start developing theories, so we got to find a set of facts. These might not be real facts, but possibly, that's what excites me. I don't want to leave the audience on tenterhooks about what are the other things they found that worked, so I'll just go through the rest of them. They found that higher incentives work. Paying people more to do things works relative to paying people less. Shocker, I would say that is probably a genuine fact that tends to work. You give people more incentive to do a thing, and they do it more often. Another one they found is that this is a really weird one: giving people the choice of framing it as a gain or a loss seemed to improve people's retention at the gym. What this means is, if you give people rewards, one way to think about it is that every time you go to the gym, you get some points that you could then redeem. So that's a kind of gain frame, but an alternative way to think about it is that, at the beginning of the study, you give people all the points, and then you say, every time you don't go to the gym, we're going to take them away from you. What they found is that letting people choose between the two frames seemed to help people stick to the gym more, so they got to decide whether they're thinking of gaining points or losing points.

ADAM: How should we think about the size of these effects? Is this the difference between, "I go to the gym all the time," versus, "I go to the gym once a month?" My impression is that even the things that work are very small effects.

SPENCER: I think they were moderate effects, but I'd have to check more carefully. The other thing that makes it complicated to think about is when you're testing 50 plus things, and then you're picking the ones that work the best, you should expect some reversion to the mean because you're selecting on the things that seem to work the best. Those that were lucky and had some positive noise are more likely to show up. So yeah, even if it's moderate effects, you kind of have to downgrade it somewhat. You have to think about that.

ADAM: My intuition is that if we did this for another 100 years, we might discover some things that, "Yeah, maybe now the standard thing to do when you're trying to make yourself go to the gym or stick to some habit is that there's an app you open up, and it helps you frame things in gains or losses, or it gives you points." But these effects actually tend to show up in some places and not others. We don't know why, and sometimes they're larger and sometimes they're smaller, versus what it has taken to cure disease, which is an understanding of the underlying physical structure, like "What is going on at the molecular level?" This thing binds to this thing. The way we stop that is by putting something in there that prevents the protein from binding to the thing that makes you sick. I think if you're serious about understanding the mind and trying to make people's lives better, that is the approach you ultimately have to take. While we're doing that, we're gaining facts about the world, but I think there are so many facts out in the world that are just sitting there waiting for us to explain that we don't actually need to run giant studies to see them. People are behaving all day, every day, and you can get behavior for free. You don't have to assign people to different artificial conditions to see the fact that they do different things. We can't even explain those things, but the things that people do over the course of a normal day, I think there's a lot of raw material to start there.

SPENCER: That's a great point. I think studies like this can let us peer into behaviors that we don't normally have access to. What is the difference between sticking to going to the gym and not going to the gym? That's a question of particular interest. But I agree with you. There's tons of raw material all over, but maybe not about some of these phenomena, but you have to actually do a study. This actually brings me to the second reason why I like mega studies. Let me describe to you a study we ran, why we ran it, and why I like it. We implemented about 20 different habit formation techniques. We created this intervention where it randomized people. Each person got five out of the 20 picked completely at random. We then tracked how well they stuck to their habit for quite a number of weeks, and then we ran a regression to see which of the techniques, when assigned to them, caused them to stick to their habit more frequently. This led to identifying a small handful of techniques that were associated with sticking to their habit more. We then packaged these together into a free tool on our website called Daily Ritual. We took the ones that seemed to work the best from that, packaged them together into a single intervention, and all the ones that seemed to work, we then ran a randomized control trial on that tool. We brought people in and said, "Pick a new habit you want to form, go through this exact tool." We then showed in that randomized control trial it actually causes people to stick to their habit more often, and we released it to the public for free, so anyone in the world can now do it and use it in exactly the format that we studied it.

ADAM: Cool. That's great. Here's a question for you, though. Can you tell beforehand which thing is going to work for which person?

SPENCER: Hell no.

ADAM: Yeah. That is the thing that we're missing, right? Right now we think of that variance as noise, but that variance is information. There is a reason why one thing will work for one person doing one kind of habit, and it won't work for another. In the meantime, we can scrounge together a few extra points of variance explained, but we're just not going to get that far until we understand the structure of the mind and how this intervention interacts with it. These are all good things to do in the meantime; I think they help people live better lives. But there is another approach, one I think that almost no one is taking right now, but that has led to basically all success in the history of science. It goes from the other direction, trying to build a model that explains the facts we perceive, rather than testing each thing in turn to see which one works.

SPENCER: So describe that process a little more. How does science actually operate and make progress in your view?

ADAM: I'm borrowing this idea from my friends, Slime Mold Time Mold, who are working on a series about this that's going to come out soon. I've seen the draft; it's very exciting. What they propose is that a lot of people think science is, "Oh you're testing things or Oh you're making theories." They argue that science is coming up with proposals for the entities that make up the world and the rules that govern those entities. You could think of this as trying to figure out how a board game works when no one is there telling you how it works. All that happens is if you break the rules, the game stops. You can imagine playing chess on the computer, where it won't let you make an illegal move, but it won't tell you what moves are legal. This is really hard if you have no idea what the entities or the rules are to begin with; you have to come up with some guesses of what the pieces are, how they can move, and how they can interact with one another. If you look back at the history of science, this is what people were doing. If you look at Lavoisier during the chemical revolution, he was observing things getting bigger and smaller when they heat up and saying, "Well, maybe it's because each thing is made of a bunch of small things, and heat is a fluid — like a bunch of little particles that push the particles apart — and there's some other force that pushes the particles together." That turned out to be wrong, but it was wrong in a useful way. If you look at the development of physics, it's about asking, "Well, what would the world be made of that produces the effects we see? Well, what if there are tiny little things that we call atoms? Oh, actually, you would need these extra things called electrons that would have to be charged in this way." This is so different from what we do in psychology right now where we go, "Oh what made this person do better in this math class? They had more self-efficacy." What is self-efficacy? It refers to nothing; it's like a tautology. Self-efficacy is being more efficacious. It does not correspond to some part of the diagram. "What is a belief? Where in the model does belief go?" Belief could be a thing, but what does belief refer to? I know it's a weird and annoying question, but it's the kind of question you have to ask. If you're trying to fix the car and you take your car to a mechanic, and you ask, "What's wrong with my car?" If he says, "Your car has broken downness," you'd be like, "What? But which part of the car is broken?" If he says, "It has the quality of being broken down," you would be confused. This is basically what we do now with mental illnesses. "What is depression?" We don't think of this in terms of what pieces of the system have broken; we think of it in terms of the quality of being depressed. We categorize it in terms of symptoms. "Oh you're depressed because you are sad and sleep all the time, or because you don't sleep at all." Bringing your car to the mechanic is like saying, "Your car is broken down because it doesn't go." These things give the illusion of knowledge without actual knowledge.

[promo]

SPENCER: I have a proposed theory of depression, and I'm wondering if you would say it's unscientific or not. So my theory is that depression is fundamentally about your brain predicting that you can't produce anything that you value. We have these things that we value, and we can talk about what it means to value something, but basically, if you start predicting that nothing you can do can produce what you value, then your brain goes into a state that's essentially like, "Oh, there's no point in doing anything. So I'll just sleep a lot, feel tired all the time, and feel helpless, etc."

ADAM: I think this is a step in the right direction. Can your proposal explain why sometimes a symptom of depression is sleeping all the time, and sometimes a symptom of depression is not sleeping at all?

SPENCER: Yes, but it's not quite as clean an explanation. Okay, so depression and anxiety are incredibly linked. If you check the correlation between standard depression scales and anxiety scales, it's ridiculous. It's like a 0.7 correlation. This has led some people to think that they're literally the same thing. They're like, "Why are we even differentiating these, right?" But I would argue they're actually quite different things. The reason they're so correlated is partly because they cause each other very often. There's a causal loop between them. Anxiety can cause depression, which can cause anxiety back and forth. Also, there are certain variables that can cause both of them, like poverty, that could cause both depression and anxiety. My argument is that they're different things, but they cause each other and end up highly correlated. Anxiety is linked to lack of sleep, and depression is linked to excessive sleep.

ADAM: What does it mean for anxiety to cause depression?

SPENCER: In my model, whereas depression is a prediction that you can't produce anything of value, anxiety is a prediction that value is going to be destroyed. Probabilistically, if you knew value was going to be destroyed, that would be different. That wouldn't be anxiety, exactly. But if you predict that value might be destroyed, things that you value might be destroyed, then you experience anxiety. So what does it mean that they cause each other? It means that when you're depressed, you tend to behave in ways that lead to states of the world where you start to predict that value is going to get destroyed. Similarly, if you're very anxious, then you tend to behave in ways that lead to feeling like there is nothing of value that you can create. You can alternate between these states.

ADAM: In this setup, what is value?

SPENCER: Oh, man, I've written a lot about value. Basically, I think that the best way I can describe it is that if you think about something in the world, like, you imagine something, your brain seems to be able to say whether that thing is valuable or good or not. You could be like, "Okay, I can imagine being tortured, like, okay, that's bad. I can imagine falling in love, okay, that's good." This seems to be a fundamental operation of our brain. I think of it as seeing some things as valuable and some things as not valuable, and this is part of what drives our behavior, this evaluation of what's valuable and what's not valuable. You can do it both for things in front of you in the world, and you can do it hypothetically. You can imagine things. You can do it for things in your past. You can remember them and assign value. So that's sort of the best I can do at describing what it is.

ADAM: Yeah, to continue to be extremely annoying, it sort of sounds like value is what the brain thinks is valuable.

SPENCER: Well, this particular mental operation. It's like, when something's in front of you, and you look at the thing, you feel an experience that's valuable. It's not valuable. When you imagine something, you feel an experience that's valuable. It's not valuable. So I would say it's not exactly an emotion, but it operates a lot like, if you look at someone you love, you might experience a feeling of love; your brain is doing a thing. But, yeah, I don't think I can get beneath that level, at least with my current knowledge. I don't think I can go lower than that.

ADAM: I think this will be super interesting to compare this series that my friends are about to put out, where they propose a theory of what value actually is that you can point to in the system, like you can point to this part of the circuit diagram and say, "This is what value actually is." But I'm getting ahead of myself because they haven't released it yet. So I have to stop there. But by the time this comes out, it might be out.

SPENCER: Great. Well, I look forward to seeing that. Taking a step back here, there's something that you said that I thought was really interesting, which is that people sort of misunderstood the replication crisis, or they learned the wrong lessons from it. The replication crisis being the idea that lots of papers in psychology that were 10 or 15 years old or whatever don't seem to hold up. If you redo the study from scratch, try to rerun it in a faithful way, you don't get the same result. So what do you think we should learn from the replication crisis?

ADAM: There's the part of this that I think everybody, we all agreed that we should learn, and we did, which is, right now we're using methods that can very easily mislead us. That's an important lesson we all realized, "Oh you can't just put 17 people in an experimental cell — not a literal cell — to do that, you have to go back to the 70s, and you need sufficient power. You can't just be running t-tests all over the place. You'll get a bunch of false positives." Stuff like that. All that's great. But then there was this secondary lesson, I think, that everyone felt that, "Oh the thing we need to do with every new piece of research that comes out is first ask, is it true? When, in fact, I think the first thing we need to ask is, would it matter if it were true? And if the answer to that question is no, then you don't have to care whether it's true or not." I actually think that most of the research that comes out of psychology fails the first question: there are no stakes for it being true or not. We create artificial situations and then see how people act in them, and mostly these situations correspond to nothing. There's nothing that can be learned. For instance, a lot of people were very interested for a long time, and still are, in whether ego depletion exists. This is the idea that using self-control depletes your ability to use more self-control in the future. Again, these are working in abstractions. The canonical test of this was if you have to cross out all the e's on a piece of paper, do you then choose to eat a cookie instead of a radish? That's not exactly what it was, but schematic. So we're testing this all over the world; labs got involved, everybody made people cross out e's and then see if they eat the cookie. Ultimately, nothing depends on whether crossing out e's makes you eat a cookie or not. This was always supposed to stand in for another idea. I think we've come to think that it all hinges on whether this empirical fact turns out to be true or not. That is what makes ego depletion a thing. But I think from the beginning, ego depletion was never a thing. It is an abstraction, a generalization that's maybe sometimes true and maybe not. We're never going to learn whether it is true or not from everybody crossing out e's and eating cookies.

SPENCER: Is that because it's not defined precisely enough? Is it a matter of, we've got to really define ego depletion, and then we can answer questions about it, or is it something else going on?

ADAM: You could really specifically define it and not get very far. You could say ego depletion is only your tendency to eat cookies after crossing out e's, and then be like, "I don't care. In normal life, people don't cross out e's." I should explain when I say that: you get a page of text, and you're supposed to draw a line through all the e's. So it's not just a matter of the definition not being tight enough. Once again, this is not a proposal for the entities that make up the world or the rules that govern them. Imagine we did find, imagine we were like, God himself came down and was like, "Guys, you don't have to run the studies anymore. I'm here to tell you, when you make people cross out e's, they eat more cookies." I'd be like, "Okay. I never thought that was impossible. I always thought that you could construct a situation where you could do something that looks like making people use more self-control, and then they do something that seems like a lapse in self-control." Obviously, you could create a situation, given that we all have unlimited researcher degrees of freedom in the situations we can construct. Of course you could. This would be different if it was like, "We can make people levitate in our lab rooms." I'd be like, "I don't think you can do that." So if you do that once, and I'm really sure that you did it, now I've learned something.

SPENCER: You make me yearn for the old days of psychology studies where if they could get something to happen three times, you were really impressed because you thought that would never happen in a million years. The Asch conformity experiment, yes, getting someone to misreport the length of a line that's clearly bigger than the other line. That's pretty impressive, right?

ADAM: Yeah, that's why that stuff rules. Same goes with Milgram. A lot of people have tried to debunk Milgram. I think Milgram remains bunked. The whole thing with the shock experiments, the reason why that was crazy is because people literally went on record saying, "I don't think anybody other than a tiny proportion of psychopaths would ever conform in that situation." And then it's pretty easy to construct a situation in which they do. That's why that study mattered. But so many studies, when people lay them out, "I'm like, yeah, it could work. I don't have a strong theory that it could not work. Showing that it can work doesn't tell me anything." I think people mix up the idea that showing something can work does not mean that you show that it happens all the time or that it could happen in any other situation. All that you can prove with a study is that something happened once, and so in order for that study to be informative, you have to have a strong theory that that thing should not happen. This, again, is why all the cognitive bias stuff really kicked off, because here's the rule — oh, people break the rule — you only need to show that once to show that something interesting has happened.

SPENCER: I really share your view that most psychology studies, you're like, "Yeah, so what?" So let's suppose it did turn out the way that you're saying, "So what?" To help check my intuition on this, we've been keeping this big spreadsheet of replications, and I just pulled it up and picked some random ones. Here are some things that fail to replicate, and I'm going to read them. These are completely random failures to replicate, okay? You can tell me whether these matter. Does it make any difference whether they replicate or not? Here's the first one: By subtly priming empathetic feelings, simply asking participants questions about their empathetic understanding and personal capability of similar transgressions, men's feelings of vengefulness towards transgressors are reduced, but not women's feelings of vengefulness. Does this matter?

ADAM: I love that you've described it at that level of specificity, that you've actually described what it is that people were doing, because that's probably in the method section of the paper, I'm sure that's what they said. But the way that paper is written is something like, "You know, empathy makes men nice, or something like that." It is described at a level of abstraction that makes it sound important, when, in fact, the literal thing that people did was answer some questions about some time they felt apathetic or something like that. Then they probably read a vignette where it was like, "Oh, if someone were mean to you, if someone broke a rule, would you be mean to them?" I'm guessing that's the kind of thing that they did. And I was like, "Why did you do that? Why did you waste your time doing that?" There's so much good stuff to watch on Netflix. You should have done that instead.

SPENCER: I really wish that every psychology paper just had a table at the top that was like, "Here's exactly what we actually did, here's literally what we had participants do. It just had to spell it out." Honestly, it's often buried deep in the paper. Sometimes you honestly have to look in the appendix. There are even situations where you actually can't figure out exactly what they did because they describe it in sort of only semi-precise terms, and they don't provide the material so you can actually look at it.

ADAM: Yeah, it's crazy. The whole paper will be about empathy and altruism. It actually turns out what they did was decide whether to give a penny to a stranger on the internet. If you'd written the whole paper about whether people would give a penny to a stranger on the internet, you'd be like, "Okay, why?" Maybe you can convince me that this actually turned out to be really interesting, but you've tricked me into thinking it's interesting because you talked about it in terms that sound important, like empathy and altruism, and not in a way that doesn't sound important, like pennies and people on the internet. I think of a list of these, and I've waylaid us on the first one.

SPENCER: This really reminds me of this concept that we came up with called importance hacking, which is about the ways that authors essentially trick reviewers into thinking that what they did is valuable or worthy of publication. The reviewers end up approving it for publication, but then you don't really learn anything important from the study. A lot of this, not all, but a lot of this important hacking has to do with the way they talk about their findings in terms that make it seem like it really mattered. What they are actually doing is just getting people to give pennies to people on the internet.

ADAM: Yeah, it makes it really hard to learn anything from papers because you have to dig in so much to realize what they did. It also covers up a lot of things that are interesting, whether or not they're important, because we have to talk about them at this level of abstraction. Here's a study I ran recently. I haven't published this yet. This is the stupidest study of all time. I wanted to know what is the dumbest, lowest effort study I could do that would teach me something. At a party, I got two buckets. I filled one bucket with ice water, and I filled the other bucket with nothing. I took people upstairs, and each bucket was in a different room, and I randomized the rooms. I took them in and said, "Please put your hand in the bucket for as long as you want, and just timed how long they put their hand in the bucket." I'm not saying this is going to uncover the mysteries of the universe, but it did teach me something. There was so much variance in the amount of time that people put their hands in these buckets. The cold bucket is literally a task that psychologists have developed to induce pain; this is the cold pressor task. This is supposed to be when you want people to suffer in your study, you make them put their hand in a bucket of cold water. A lot of people didn't like it. One guy put his hand in the cold water for seven minutes and 36 seconds. Why did he do that? He stood there and looked at me and said, "Oh, yes, I understand that putting your hand in cold water causes pain." I was trying not to say anything because I didn't want him to stay there while we were talking. The fact that sometimes you will encounter a person who will just put their hand in a bucket of cold water and cause themselves pain — I didn't think the variance would be that big. Meanwhile, one person put their hand in the empty bucket for 4 minutes and 46 seconds or something like that. On average, people did not put their hand in one bucket or the other; there was no difference in the amount of time that people put their hands in each of these buckets.

SPENCER: Well, that's interesting. You think you predict that the painful bucket would get less time.

ADAM: Yes, that, "Okay, people don't like pain, so obviously they should not spend any time in the cold bucket and some small amount of time in the empty bucket." But no, there's no difference. I don't have an explanation exactly for why this happened, but it was the stupidest study of all time, and it still taught me something. It taught me something because I talk about it in the terms that it actually happened. I could have talked about this as pain aversion versus emptiness, but no, this is people putting their hands in buckets.

SPENCER: Yes. I'm sure you could drum it up to make it sound really impressive. I was trying to predict what I would do in that. In my prediction, I don't know if I would actually do this, but my prediction is I would keep it longer in the cold bucket because the cold bucket, I'd be like, "Oh, this is interesting. I wonder how long it takes me to feel significant pain and where's the empty bucket?" It's just boring.

ADAM: Yeah, exactly. I do think part of what's going on is that naively, we think of pain as pure aversiveness, like it is just the feeling of wanting this to be over, when in fact, pain is also stimulation and it's a thing that changes over time. People felt their hands go numb, and all the stuff that, I think if you describe this to most people beforehand, they'd go, all that's bad. I'd want that to be done as soon as possible. But why is it that plenty of people, when they were doing it, found it aversive on some level but interesting on another level? Something weird is going on there.

SPENCER: It reminds me of the study where they left this electrical shock device in the waiting room or something, and people would shock themselves, yeah. But I could totally see myself doing that. It's just like, "What does that feel like? I wonder."

ADAM: Yeah, where's the feeling? "Oh, it hurts. Will it hurt next time? Oh, it does, yeah. I'm still here. Let's see if it hurts a third time." I think in that study, there was someone who shocked themselves 197 times or something like that. An advantage of running a study at a party like this is that you give people the capability of responding in weird ways. When I ran studies in labs, I ran a study on conversations where I'd have people come in and choose how long they want to talk to one another, but they've signed up for an hour slot, so maximum they can only go 45 minutes, or I have to stop them because they have to answer questions afterward. If someone had wanted to speak for six hours, they could not have shown me that behavior, whereas when you're out in the world, someone could have kept their hand in the bucket for six hours. I would have had to go and get some more ice. There wasn't a ceiling effect on what I was able to observe. That was another thing I ran from running the stupidest study of all time.

SPENCER: Love it. It reminds me. I have this idea of micro experiments, where they're not well put together, they're problematic in lots of ways, but the way I learn from them is I make predictions in advance of what I think I'm going to find. I do this on Twitter all the time. I wonder about human nature. I will run a Twitter poll that tries to tap that thing I wonder about, and I'll register a prediction at the bottom of the poll of what I think is going to happen. The way that I learn from it is when it deviates drastically from my prediction, I start to wonder, "Okay, what did I get wrong? Did people interpret it differently than I thought? Or did I think something was true about human nature and it's not true? Or is it something about the demographics?" I find it's a great way to do these rapid micro-learnings.

ADAM: Yeah. And again, just like all the cognitive bias literature, you've discovered something when people's responses deviate from the thing you expected or the thing they should do. This is a hack for discovering informative facts. It's also really historically weird in a good way. It is wild that humans did not do this very much at all for most of our existence. I was just reading the other day. Cicero, the famous orator, wrote this whole book against the idea that divination works. At no point in the book does he propose or conduct a test of divination. He never suggests that you could tell whether it works if you cut the goat open and it says it's going to rain, and then see whether it rains or not. It does not seem to have occurred to him. He's got all these logical arguments, all these historical arguments, but he has no empirical arguments. It's not part of his schema. In general, it's not part of anyone's schema. You have to learn that this is a thing you can do. You can make a prediction and then test it to see if it's true or not. It just doesn't seem to be part of the starter kit for our cognitive systems, so I think it's good practice to do it.

SPENCER: I heard about these long debates on anatomy that could have easily been resolved by someone checking a cadaver, but people just argue about it forever instead.

ADAM: Yeah, yeah. And there's this account might be apocryphal, but I was reading about Aristotle said that there's this certain massive vascular tissue at the base of the brain, and people dissecting human brains claimed that they saw it, when, in fact, humans don't have this; this is only on whatever it was that Aristotle saw. And in part, because, I don't know, when you're cutting stuff up, it's probably gross. There's a bunch of stuff in there. And so maybe it's kind of ambiguous. And so even though it should have been easily tested, the thing is, literally, the structure is not there. People can think that it's there when they go looking for it, which is why doing empirical stuff, you really have to hammer it all the time to get something out of it.

SPENCER: All right, let's go back to our randomly sampled things. We'll do a couple more. And this is totally random. So the second one, not believing in free will, but instead having been exposed to a deterministic message, increases the frequency of cheating on an arithmetic task.

ADAM: The idea here is, I understand why this idea is seductive, that it's like, "Oh, if people don't believe in free will." Then they'll be bad, because they'll be like, "Well, I have no control over my actions." If we knew the answer to that question, that would be interesting. Instead, what we have is the question: does reading this story make people cheat more on this test? And could you create a situation where you make people feel like they have no control over themselves, and then they do a bad thing? Yeah, sure, you could. I'm sure if given infinite time and resources you could. You could also construct a situation in which people do the opposite. So, the fact that we've witnessed one thing or the opposite of that thing does not inform my view of human nature or how the mind works.

SPENCER: So, but if I want to steal man this for a moment. So they have a theory that says that, "Well, look, if you don't believe in free will, maybe that lets you off the hook for doing bad things, and so maybe people will do more bad things." "Okay, let's think about if that were true, what would that predict? That would be surprising, right?" "Well, it predicts that if we can prime people with the idea of things being deterministic, then they will immediately cheat more, right?" "Okay, cool. I have a prediction about the world. It's like an unusual prediction. Isn't that exactly what we should be doing? Now let's go check if that unusual prediction holds. If it does hold, then that's evidence for our theory, because we didn't have any reason to predict that otherwise." Now, of course, this one didn't replicate, so all this is off the table, but let's say it did replicate, what's wrong with that?

ADAM: I would have believed you beforehand that if you were to ask, is it possible that sometimes people who don't believe in free will might do more bad things, I would say, "Yeah, it really could be." If you show me that it's true, I would agree. I also think I could construct a situation for you where the opposite happens. So, if people don't think they have free will, why would they default to doing bad things? If I put you in a situation where I convince you that you don't have free will, and then I give you a default option that involves giving away the money I would pay you to be in the study, would I also not see you do the altruistic thing? Now it's like, do they just do bad things, or do they just do easy things? It doesn't convince me one way or the other that not believing in free will makes you act poorly.

SPENCER: But let's say they're really truth-seeking researchers. I'm not saying they're not, but let's suppose that they're super truth-seeking. They really just want to know the answer to this. It's an interesting idea that maybe not believing in free will does cause you to behave differently. Maybe it does make you behave more unethically. Let's say they could design an experiment or series of experiments that would be pretty convincing. They could come up with a bunch of different scenarios where they vary how much people experience free will, and they could show that uniformly across them, lack of belief in free will is always linked to these bad behaviors. It doesn't really matter what the stimulus is. It doesn't really matter what the bad behavior is.

ADAM: Yeah. If it were the case that this effect were that strong and that robust. First of all, it would be crazy that we hadn't seen it already, and then it would be interesting. If you could show me that it really doesn't matter how you do it, where you do it, or who you do it with, and you get this strong effect 100% of the time, now that's interesting.

SPENCER: But what if it's just boosted?

ADAM: The thing is, they didn't find that effect, and they never will.

SPENCER: Well, it's hypothetical, but let's say it's not that it happens 100% of the time, but every single time they do it, when they prime people with a lack of belief in free will, they're 20% more likely to do the bad thing, right? Isn't that still interesting? Even though it's not 100% of people.

ADAM: It is less interesting to me. Now, if you could show me, "Okay, what bad thing?" If you could randomly sample from the universe of bad things that people could do, if you show me over and over again that you change the parameters and it doesn't change the outcome, now it makes it more interesting to me. This is why I'm not that interested in a one-off demonstration that this thing was possible. If you showed me 100 of these, where the effect does not go away no matter what you do, now I'm more interested, and I'm more confused as to why we didn't already know.

SPENCER: I think for me, I'm still very interested, even if they only show it in, let's say, four scenarios, and it's only a 20% increase in bad behavior. I find that very interesting if they're truly being truth-seeking and trying to disprove themselves. My problem is just that I don't necessarily believe that with any given study, and this one failed to replicate. Maybe it was their fault. Maybe it wasn't. Maybe they were trying to truth-seek, and they got kind of lucky or whatever. But my problem is, "Why did they cook up this exact scenario?" Maybe it didn't work with other scenarios, and they just didn't mention that. But if I really believed it, then I'd be like, "Oh, that's interesting. Maybe Sam Harris's podcasts are convincing everyone that they have no free will and actually causing more unethical behavior." That's interesting.

ADAM: Yeah, I guess even if I watch this study happen, and I'm 100% sure that they didn't do something like test a bunch of different vignettes and see which one works, or do all the things that would have made this work, even if I saw it happen once, I'd go, "Okay, I would never have bet against it happening once." But now, if you can show me that it happens across all sorts of situations, now my intuition is being overcome, because I would think that it'd be very easy to encounter a situation where this doesn't happen. I can actually give you an example of a study I ran where this opposite happened, and that's why I stayed interested in it. I ran this study where I asked people how X could be different, and X was 52 different nouns. So just like, how could cars be different? How could your computer be different? How could your life be different? How could your personality be different? People typed it out, and then afterward, I had them rate how much better or worse it would be if it were different in that way. It turned out that overwhelmingly, people told me how things could be better, and they did this for every single item. 90% of participants did it. That was counter to my intuitions because I thought that for some items, it would work and some not, and for some people, it would work and some not, but it seemed like for every single item and almost every single person, it worked. That made me want to investigate this more because this seems like a powerful effect, and it didn't. So I ran a bunch of other studies on it, and it did, in fact, turn out to be, we never were able to find a situation where this didn't happen. That's why I was interested in it, because I didn't have the intuition that you would find this everywhere all the time for almost everybody. I did have the intuition that you could find it somewhere for some people. I was interested in figuring out which thing for whom, but that's why for this free will study, I would want to see, "What were the 52 items for them?" That's what I'm looking for.

SPENCER: You actually failed to mention the most interesting thing about that paper, in my view, which is, I think it's an incredible paper that you wrote where you basically said, "F you to all the academic norms about how to write," and you just wrote an extremely interesting paper in whatever way you wanted to write it, and it's really fantastic. I recommend people read it; I'll put it in the show notes. But maybe you want to just comment on how what you wrote there differed from what you would have written if you were submitted to an academic journal.

ADAM: I could show you the point-by-point comparison because we originally wrote it for an academic journal, and the reason we stopped was because it was terrible. We had run the set of studies that I found really exciting and confusing. I described to you the first one, and then we were like, "Do Polish people, people living in Poland but answering in English, show it too? What about people living in China answering in Mandarin? Do they do it too?" Anyway, and on and on for eight studies, and they were so fun to run. Then when it came to write about them, it just felt dead, it felt stupid, and it felt like we had to lie. For instance, we forgot why we ran study eight; we forgot the justification for it. It was pre-registered. We had written the code; all that was there, but I forgot the rationale we had been thinking about when we ran it. That is a really embarrassing thing to put in a paper, and so we tried to write it such that we weren't outright lying, but we were sort of glossing over it, and after study seven, of course, study eight, and it felt bad. It felt like we were dissembling, and so we just kind of broke down, and we were like, "Why don't we just write exactly what we did using the words that we think are most informative?" And then we put it on the internet. Before we did, I thought this is a stupid and embarrassing thing to do, not stupid. I think this is the right thing to do. I feel good doing it. I don't think it's going to work or do me any favors. At the time, I was trying to get an academic job, and I thought this is basically like setting a $100 bill on fire. Why would I? This could have gone into a journal and gotten me some credit. Instead, it's going to go on the internet and get me none.

SPENCER: You're going to write a mirror blog post instead. That's lower status.

ADAM: Yeah, it's cringe to blog, but in fact, it's the most beautiful thing in the world because that paper was viewed more than my previous two papers combined, and those papers were published in prestigious journals or whatever. Overnight, it got so much more attention, so much more engagement, and so much better engagement than any other paper I had written up to that point. I thought this thing that was right but stupid turned out to also be a good way of getting this information to the world.

SPENCER: It's really just shocking how different it is than an academic paper. If you read it, it's so much more engaging to read and super honest about its own process. You're not sugarcoating things. You talk about how you forgot why you did the study. You just put it all out there.

ADAM: And why wouldn't you? One of the responses I got on Twitter was someone saying, "I read this out loud to my eight-year-old daughter this morning, and she understood it." I was like, "Yeah, why wouldn't I? Why would I write my papers any other way? I would like that eight-year-old to know about this, because maybe she'll be a psychologist one day. Regardless, she's going to live in a world where science matters. I'd like her to have a sense of what people do when they do that thing." She could be like, "Maybe I'll do it too, or here's how I'll assess it." I also know that people read this paper from beginning to end, because so many people commented on study eight as the last study. I think it's close to the last study. You have to read a lot of the paper to get to that point. Yeah, why wouldn't I want people to read my papers from beginning to end? They never did that with my old papers. That's not, actually, that's not why we did it, but that is our retrospective justification for it.

SPENCER: Yeah, what is the real fundamental reason that you did it?

ADAM: It was the right thing to do. I was sick of writing the bad version of the paper. I was sick of lying and falling over myself to explain how this clearly fit in with previous research, when it was like, "No, this just totally came out of us pulling something out of our ass and then thinking that we knew what was going on, and then not knowing what was going on." By the way, the reason we ran the first study is not confirmed by the first set; our hypothesis is immediately disconfirmed, and then we end up doing something else for the rest of the paper. That's an awkward thing to try to explain. I was just like, "I don't know, the reason I'm doing this is because I want to understand the world better. Why do anything that would conflict with that? Why pick the wrong words? Why write the worst paper, even if it means I have to set a $100 bill on fire? Why not just do it? I've given up everything else to do that. I can't give that up too." That's why we did it.

[promo]

SPENCER: I often find in my psychology research that the goal of figuring out the truth is at odds with the goal of being explicable to the people reading the write-up later. What will happen is, for example, let's say we're going through a bunch of cases, and I have various reasons why I think certain cases should be excluded based on kind of a priori arguments. But actually, if I was trying to show that I'm doing good science, what I would do is say, "Oh, we had this simple criteria to decide which cases to include, and we applied it uniformly across everything." That's an example. Of course, someone could critique me and say, "Well, but you're just doing it after the fact, and you're just making justifications, and humans are riddled with biases, and on some conscious level, you actually want it to come out a certain way, and you're just going to bias it in that way." I could say, "Yeah, fair point. These are all fair points. Yet, I actually think the right thing to do is screw these cases if I'm being totally truth-seeking." I just often feel that tension, and I'm wondering if that's a tension that you felt in your academic work.

ADAM: No, no, I always felt tension, I guess not for explicability, but for prestige. I felt tension to tell people what I thought they wanted to hear, or to make it look good, but it actually felt like the easiest thing in the world once I decided I was just going to be honest about everything. When I hit the point where, "I made a decision, you could have made a different decision. Here are my reasons for the decision. I would make it again. But I want you to understand that this was a decision that I made. It was subjective. If you want to make a different one and see if you find a different thing, you can do that." Honestly, that felt so easy.

SPENCER: I think that's exactly the right way to handle it. You just say, "We made this decision for this, this was our reasoning. Others could disagree with that, that's why we did it." But I think if you're trying to look scientifically, you want to kind of be like, "Well, we apply this uniform criteria, and blah, blah, blah, but that's a big objective, etc."

ADAM: Yeah, when you let go of that, everything's so much easier. I'm not a nudist. I've never been to a nudist thing, but I imagine this is what it feels like to go to a retreat where everyone's naked, that you're just like, "Now, I am what I am, and I could look good, I could look bad, but this is how I look, and I don't have to worry anymore about how I look. It's so freeing." That's why I don't want to go back, because I don't know, I'm sure you feel that tension when you want to look a certain way, and it just all goes away if you don't have to look anyway at all. If you embrace the ethos of writing on a blog that is inherently cringe, you are already dead. Now you can do all that you want.

SPENCER: I love that. Let's talk about peer review, because my work sometimes gets critiqued saying, "You didn't get this through peer review, so why should I trust anything you have to say?" Actually, we do submit articles to academic journals, sometimes. We have two articles that just got accepted today, coincidentally, very oddly on the same day. Yet, when I go through peer review, I always regret it. "I'm like, why did we do that? It was so painful."

ADAM: Why do you regret it?

SPENCER: Okay. Let me give you contrasting examples. We just got these two papers through peer review. It was months of getting the paper just right for the journal, dealing with all the minor objections from reviewers. It was so much work, back and forth. They weren't happy with this change and that change. It was just so painful. Coming out the other side, "Is my paper better? No, my paper isn't better. It just felt like a massive waste of time. Yay, in a year, people will get to read it. But we wrote fantastic work." Compare that to, we write this up for our newsletter, we finish, we try to do a good job, we try to make clear the limitations of our work, we're done in a few weeks, and then we put it out to our newsletter of over 250,000 people. "Okay, now we're on to the next thing." The comparison is ridiculous to me. Why am I torturing myself? And yet, it's a real critique. I want to say there's validity in saying, "Well, putting it through three experts who are going to critique it, isn't that a really good thing? Isn't that a really good way to maintain quality and prevent problems?"

ADAM: I don't know. It seems, in your case, it wasn't.

SPENCER: I don't know. In my experience, when I push things through peer review, usually reviewers will have a few good suggestions that modestly improve the paper. Every once in a while, they have a really good suggestion, but I'd say that's not the norm. I don't think I've ever had an experience where they found something that I thought was a major flaw that I needed to rethink, or something like that. Generally, it's really minor, "Oh yeah, we could clean up a little blah, blah." So the level of pain versus how much better I made my paper just has almost never seemed worth it.

ADAM: Yeah. I just agree. I feel like I've gotten much better feedback from people who volunteered to comment on something on the paper because their motivation is to make the paper better, whereas the relationship I had with any reviewer was adversarial. They were always like, "Should I open the gate and let you through?" Which always seemed weird to me because it's not like they're running the journal. The editor is a different story, I guess, but it's not like they get paid if this is good or bad.

SPENCER: Nobody even knows it's them. It's very interesting.

ADAM: Yeah. So, why are they cops all of a sudden? They're mad at me for bothering them, but they signed up to do this. Whereas the people who email me when I put papers on the internet, sometimes they're mean just because there's a lot of people on the internet. By chance, some of them are going to be mean, but on average, the quality of the feedback I get is way better because people are doing this out of the kindness of their heart and because they care about this idea and they want it to be good. Of course, they're going to spend more time making their feedback more useful to me. I don't want to talk to the person who's guarding the gate. I want to talk to the person who also wants to figure out whether this is true or not. So it's just a no-brainer to put my paper on a blog instead.

SPENCER: That's my experience, too. Basically, we put things out in the world. We try to think of flaws in our own work and include them. We sometimes miss things and make mistakes. We get yelled at a lot by people who are like, "You made this mistake." Then we're like, "Oh, did we or not?" We think about whether we agree, and if we agree, we're like, "Go fix it." We fixed it. Literally, this happened a few months ago. This person found a real mistake in our work, and we're like, "Oh my God, you're right." We redid the analysis, and we fixed it. Then they're like, "Cool, thanks." They were shocked. They're like, "Wait, you just fixed it?" We're like, "Yeah, we fixed it. It's good now. It didn't change the results, but you were totally right. That was a mistake." Their minds were blown. I think they were just not used to something actually changing because if you publish a paper, it's not going to change. What are you going to do? Is it just for retraction?

ADAM: Most of the time, they would never get a response, or it would drag out for years. They're investigating, and "Can we get the data?" "No." I feel similar. I feel so free when I publish a paper as a blog post, and someone's like, "Hey, this thing was wrong." I'm like, "Oh, thank you. Thank you for spending your time on my work. I'm happy to fix it when it's clearly wrong." Whereas if I've published a paper in a journal, I've got some status for it. Now if someone comes and tries to point out an error in my paper, my status is in jeopardy, and I'm really motivated to make sure that my status doesn't go down. It's really anxiety-provoking, and it's a terrible way to live, and I don't do it. So then my question to you is, why are you publishing these papers in journals then?

SPENCER: Well, we have a new policy we've been doing, which is that we collaborate with academics. We'll do the research that we wanted to do anyway, and then the academic we collaborate with will write the paper. Of course, they often help a lot with the research. They can be really valuable in that way, sure, but basically, we don't write the paper. It's our policy. I think it's a nice balance. Most of the stuff we do, we still don't publish, but I think it's a nice balance because then we don't have to deal with the paper part. But in order to get rewarded, they have to do the paper anyway.

ADAM: Yeah, I guess if you're still in the dungeon, you've gotta eat your gruel.

SPENCER: That sounds so horrible.

ADAM: It is really bad. You said it takes a year, it takes all this time, and it's so inefficient, and you get these walls of text, and people are just misunderstanding each other. I had a terrible experience with the last paper I published, where the editor intended to send us a line-edited version of our paper, but instead they sent us, by accident, some document that contained all of the things that they had said about our paper to the editor above them. It was like someone accidentally sending us their diary where they were writing about us. It was so clear to me that this paper had not been thoroughly vetted at all. The key part of it was one of the reviewers was complaining that they couldn't access our data. What the editor wrote was, "I don't know, one person says the data isn't there. The authors claim it is." Who can say whether the data is there or not? You can literally click the OSF link and see if the data is there or not. Because if it isn't, that's a big deal. That was the last paper I ever published in a peer-reviewed journal. When this comes out, people are going to think that this was vetted, that people did something to this paper to make it more trustworthy, and they did not. There's a label on this that says approved. That doesn't mean anything, and I don't want the credit that comes with getting that label if it doesn't also come with what that label is supposed to mean. That's why I don't do it anymore.

SPENCER: Let's discuss peer review for a second, because I actually do think peer review has some good properties, three in particular. The first is, I do think it rejects a lot of nonsense work. If a person just literally made things up that made no sense, I think a lot of peer reviewers would just be like, "Yeah, that makes no sense." I think really abysmally bad work would get caught. That's the first thing, although I also think the flip side of that is it eliminates a lot of good work. That's kind of like a double-edged sword. What do you think of that?

ADAM: I would say empirically, it doesn't do that much. There's plenty of terrible work that still gets through. Would there be more?

SPENCER: I think there are certain kinds of terrible work that could get through, but I think there are certain kinds of terrible work that do actually get caught.

ADAM: Sure, so I would say two things. One is, I actually don't care about the bad work. I want a good work. This whole post has a longer story about science being a strong link problem, where we proceed at the rate that we do good work, and the mediocre stuff, in the long run, just doesn't matter. I am not willing to give up any good work if it means that I get to eliminate some bad work. I will never make that trade. I don't care that the paper that's bad that no one would read now gets rejected if it means that sometimes we reject the papers that might be really good.

SPENCER: It does raise transaction costs a lot. It just makes it so much more effortful to put your work out in the world that now, even if it eventually gets published, you've wasted a ton of time instead of doing science, just going back and forth with the reviewers, right?

ADAM: Yeah, and most of that information is also destroyed. A few places now will publish reviews, most don't. Not only do I just see the label, but I have no idea what anybody said about this paper. Did anybody open the data and check it? All that stuff I'd be interested to see with a paper I really care about. If I saw that the reviewers were like, "Yeah, I don't know. I ran the data and it didn't come out the same way, and I don't know why," I'd be like, "Oh, maybe I should run the data too and see." That's useful information, but you don't get provided that. That sounded like that was element one on a list with multiple elements with what's on element two.

SPENCER: So the second is, I do think there are some good practices that peer reviewing forces, and I think this usually happens when there's a best practice that's well known and is actually a good practice. Then reviewers will be like, "Hey, why didn't you do the good practice thing?" and that seems useful.

ADAM: I think the people who are truth-seeking are trying to do the best practices, and again, the people who aren't truth-seeking, I'm not interested in their work, so I don't really care about sending a police force to try to make them do more good practices. I don't think it's ultimately going to make a difference to the quality of their work or whether it's going to matter to me. And again, sometimes there are going to be false positives here. What people think is the best practice might not actually be. There's a lot of people forcing others to do things that aren't ultimately good. Stuart Buck at the Good Science Project just put a post out the other day with screenshots from someone's peer review process where they were forced to do a post hoc power analysis, which isn't a valid way of doing a power analysis, but one of the reviewers made them do it. Then later on, another reviewer was like, "Wait, you did this thing that is bad, now we reject your paper." You can see the receipts. I don't trust the process for doing that. But I really think that truth-seeking people are going to try to seek the truth. If a truth-seeking person doesn't do the right thing, the best practice, then someone will inform them, and then they can do it. I don't need to pay a policeman to tell them that. So that doesn't hold a lot of water for me. What's number three?

SPENCER: Three. I do think that reviewers sometimes provide really good suggestions. In my experience, usually they're minor good suggestions. Every once in a while, they may have a really good suggestion, but that seems like it's worth something.

ADAM: Sure. Is it worth $1 billion a year, 15,000 person hours of effort? Yeah, that I agree. I have gotten helpful comments from reviewers. It has not been uniformly bad. I think it's been net bad in my experience. But was it worth the year that I spent? Was it worth all that time spent writing responses to reviewers, and was it worth all the ways that I made my paper worse to please them, not even just at the review stage, but at the idea generation stage, knowing that I had to ultimately produce a piece of research that would please people who have certain interests and predilections and has to go in this kind of journal or looking for this kind of thing? Is it worth all of those costs? No way. Not even close. So if sometimes people catch a typo, or they're like, "Oh, what if you switch the axes." My hope is that if I put this out there, people are going to tell me that. I don't need people to do that because they signed up to be a gatekeeper for a while out of some sense of professional obligation. I hope that if my work matters, it will attract the attention of other people who are trying to seek the truth, who will also help me seek that truth better.

SPENCER: You and I might be a little bit spoiled, because I think when you and I put out things on the internet, a lot of people will be like, "That's wrong, or that could be better in this way." I find that incredibly valuable – all the commentary. I often disagree with the critiques, but I'm happy people give them, and sometimes they're right, and it helps my work improve. A lot of times people have good suggestions that also inspire me and give me ideas for things that could be done better. But it is interesting, you have to have a big enough audience to get that. You can't, not everyone can just get that feedback.

ADAM: I don't think so, actually. People tell me this all the time. When I tell them a story about how things could be better on paper, they're like, "Oh, you had a successful blog." What they don't know is that's not how people found the paper. In fact, people found the paper by accident. They found it. I didn't intend for them to find it. The day before I was going to post this on my Substack, I put it on SciArchive, where you can put PDFs of papers, because I wanted to be able to link to it the next day. What I didn't realize was that at the time, I think this has been broken and maybe replaced. I don't know what the current status is, but there was a Twitter bot that tweets out whenever someone uploads a new paper to SciArchive, and that's what happened. It tweeted just the name of this paper, and that bot had like 6,000 followers or something. Psychologists were interested in papers, and a few of them saw this weird paper and they read it. When I woke up the next morning, like 10,000 people had downloaded the paper or something like that. That's a freak accident. But it wasn't that I was a successful internet personality, and that's why people looked at this and took it seriously. It was a weird paper that came through a weird channel, and it still caught people's attention. I talked to a lot of grad students who are like, "Oh, I could never do this." "Yeah, I don't know, a lot of stuff on the internet doesn't go anywhere," but most papers also don't go anywhere. If your work is good and you keep doing it, it just seems extremely likely that it's going to find its way to the people who are interested in it. I'm glad that many people saw this paper, but I really only need a few people to see it, like the people who are interested in working on this problem. Everybody else is great, I'm happy to tell them about it. But I didn't need it to go viral. I needed to define the six people who it's relevant to.

SPENCER: Yeah, and honestly, you could have probably looked up different people you wanted the opinion of, and just emailed it to them and be like, "Hey, I see you do work in X, I'd love your thoughts on this." Maybe a third of them will get back to you and give you feedback. It's not that hard to get feedback. A lot of times, even top experts, nobody knows who they are. They're not celebrities, and a lot of them will write you back.

ADAM: Yeah, if you're working in a really niche area, there might only be a handful of other people who are also really excited that someone else is trying to work in that niche area. It is also, I think, a benefit to be doing this in a low-status way. Because people would rather help you. They're not trying to tear you down. This sounds like a thing that happens to you because you have success on the internet, and so people can see that a bunch of other people are seeing your work, and so that makes them want to criticize it. And so you get them mixed in with the people who care about making the work good. But if you're a nobody, but you're doing work earnestly, and anyone who spends any time looking at your work can see that you're trying to figure out what's true, I think it's a huge benefit, and it's almost a liability to be famous after that point.

SPENCER: I will also just add, I think you're a shockingly good writer. So as I was reviewing some of your blog posts before this episode, I was like, "Damn, this is so interestingly written." And so I think you might be underestimating how easy it is for you to get people to pay attention to what you're doing.

ADAM: Thank you for saying that. But I think the bar is so low. You read a normal paper. It feels like a tortured device for your mind. It's structured to make you not want to read it, and so literally, if you just write it in a normal voice, even if writing isn't mainly your thing, you're already going to be better than 95% of papers. Ten years from now, if everybody's doing this, it might be harder. But for now, it feels like podcasting in 2014; it's like, now's the time to get in. Sorry, were you going to say?

SPENCER: I have this thing I do when I read academic papers most of the time, where I read the abstract just to see if I'm interested in it. Then I immediately jump to the method section, and I try to figure out what exactly they did. What was the stimulus? What was the response? Then I immediately jump to the results section. And so I'm basically just trying to say, literally, what did you do? And then what are the statistics? And so I avoid all the writing where they actually try to describe anything, and then maybe I'll go back to that if I'm interested. But basically I'm just like, "Tell me what you did and tell me what happened."

ADAM: And when you're writing the methods and results, you don't have to be that good at writing, just be straightforward. You don't have to be funny, you don't have to be fancy. In fact, you shouldn't. Just be honest, and the funniness will come naturally. The most important part of the information will come naturally. The fact that it is so hard to extract information from the standard scientific paper is a flaw, and you can immediately overcome that just by not lying, not dissembling, and not using abstractions to stand in for what you actually did. I think that if you do that, your paper is already so much more interesting. I'd rather read the paper that's about the stupid thing you tried to do, and how, "Oh yeah, the first time we tried to run this, someone showed up and sneezed on the apparatus. That's why we had to use a different thing." I'd rather understand what you actually did than your abstract version of it that you would write for a journal.

SPENCER: There's this funny thing that happens in mathematics, which was my field when I did my PhD, where people will spend a year or more proving some theorem, going through all these crazy twists and turns. They have an intuition, and it doesn't pan out, and they have another intuition, and then they go write the paper, and the paper is like two pages. It's just this pure, perfect proof, right? It's as if it was plucked from the mind of God, but all of the things about how you actually figured it out, why you even thought it was true in the first place, are just gone.

ADAM: Yeah [laughs]. Have you ever read a paper like that, or have you ever discovered that additional information that made you feel differently about what's in the two-page version?

SPENCER: It's hard to know, because usually you don't get that information. You don't know. I love it when they give you an intuition; sometimes papers do, like, "Here's why you might think that this is true." Kind of setting the stage in your mind, because some proofs are very informative. You read the proof and you really feel like you get why it's true. But there are a lot of proofs where you read the proof and you check every step, you understand every step. You're like, "Okay, A goes to B, I get that. B goes to C, you get that." But then you get to the end and you're like, "I really don't get why A goes to Z." Every step in the chain makes sense, but the whole thing makes no sense to me still, even though I've checked it all.

ADAM: So if you could see someone explaining what they were thinking at each stage, it would help you understand how they got there.

SPENCER: Yeah, exactly. Here's why I pursued this. Here's why I thought this might be true. Here's my sort of intuitive argument. That's not a formal proof.

ADAM: Yeah. And this is why I think the standard paper makes no sense, because it destroys so much information. It's fine to have the two-page version for someone who, "I don't want to hear about how you were tying your shoes and thought about whatever. I just want to see your numbers." That's fine. But why not also write down the things you were thinking at the time? In my case, I have published papers where there were five more studies that we didn't include, not because we were filing them, but because they didn't end up being relevant to the narrative that we told about this. But why did we just destroy that data, basically? Why didn't we go, "Hey, we went down a rabbit hole. We ran the study that we don't think is ultimately relevant. But here's all the raw stuff if you want to see it." Because someone might want to see it, and it costs money to get it. It cost the time participants spent. It cost our time, too. Why memory-hole that?

SPENCER: It seems like a lot of it is predicated in a world where you literally have to go to a printer, print the things, and mail them around.

ADAM: Yes, exactly. Now that space is basically free, almost no one will read the additional information. However, the people who do are the ones that it will really matter to, because they might try to build on it. So why not give them your raw materials. What are the exact questions you asked in the order that you asked them? What is the actual data, in case someone wants to build on your study? This is also something I feel great about, rather than upset about. In fact, you're involved with the transparent replications, right?

SPENCER: Yeah, we found that, yes.

ADAM: So when you guys emailed me about that moral decline...

SPENCER: Oh yeah, we replicated one of your papers, right?

ADAM: Yes. When I got that email, every sphincter in my body shut.

SPENCER: I'm sorry [laughs].

ADAM: No, no, no, it's a good thing that you're doing. But I was anxious for all the wrong reasons, which I understood at the time. But it's hard to turn it off. "I believe in this paper, I know I didn't do anything untoward. However, what if I got a fluke or something went wrong, and they don't replicate it? Now this thing that brought me a lot of status is gone." What a terrible way to feel about that. That is the opposite of the way that I want to feel about the work that I do, and that's why I don't do it that way anymore. Now, when I just put all my stuff out there, and I find out that people have replicated it, I feel great. Even if it turns out they didn't find the same thing, because now there's something that I'm like, "Okay, why did they get a different result?" I found out a French YouTuber picked up the "things could be better" concept. I don't fully understand what he did because it's in French, but it seems like he found an item that didn't work, which is now really interesting. Why did that item not work? We were never able to find one.

SPENCER: First of all, I'm sorry I stressed you out. Second, hopefully it was for a greater good. Third, I want to say your paper did amazing. We gave it a five out of five stars on clarity, a five out of five stars on replicability, and four and a half out of five on transparency. That's incredible. Your paper crushed it.

ADAM: Yeah, I'm glad. But you never know, right? When someone's trying to replicate your stuff, maybe I got lucky on that study and it wouldn't happen again, or you run it with a slightly different population, and that turns out to matter a lot. I'm glad that you did. I'm flattered that you did, and I want to live in the world where, when I get that email, I'm like, "Oh, thank you. Thank you for paying attention. Thank you for submitting your limited time and resources on work that I think is interesting."

SPENCER: Well, not to remove the flattery, but we pick it randomly. So, yeah, I think it randomly from a top journal, but yours was in a top journal.

ADAM: But no, it's a good thing you're doing.

SPENCER: But, yeah, it is a hard element of the work because I know it stresses people out, and I really am a very agreeable person. I don't like stressing people. I don't want anyone to ever feel bad. So, yeah, that is a challenge.

ADAM: Yeah. And do people respond in the first place? How many people just ..?

SPENCER: Almost always, almost always. I think, maybe one time we couldn't, literally, the authors wouldn't talk to us. They almost always respond.

ADAM: Are they suspicious?

SPENCER: I would say there's a range of responses from friendly to quite salty right out of the gate, which is understandable.

ADAM: Like,"How dare you try to replicate my study?

SPENCER: Not that extreme. More like they're answering our questions, but with a thinly veiled anger behind the responses. Generally, what we do is we tell them that we picked their paper to replicate using this process, and then we say, "We've rebuilt your study from scratch. Could you please look at it, make sure it's a faithful representation of your study? We want to make sure it actually reflects what you did." Generally speaking, they'll look at it and give us some thoughts. The only real comments they ever have are really minor stuff, typically like, "Oh, could you make this font bigger? I don't want people to miss this instruction," or things like that. Then we're like, "Okay, cool. So it sounds like you approved. This is a genuine replication of your experiment." And then once we do the replication, we show them the report before we publish it, because if we made a mistake, like our report says something that's not true, we want to make sure that they can correct the mistake before we publish it.

ADAM: Yeah, it makes total sense. I also understand why people would be freaking out and just how that shows exactly how things are wrong in mainstream scientific institutions. You should feel the opposite about this. If your study doesn't replicate, you should want to know that. If it does, that's also cool. If people want to spend their time on your study, that's great, and I understand why people can't feel that way because their life is on the line, at least career-wise. But it now feels so freeing to not have to feel that way anymore, to be like, for me, it's gravy. Like, "You're going to do work for me for free? If you want to study, please, go ahead."

SPENCER: Yeah. We've also been doing this thing where, for a lot of the ones that were really good, where we got a really good result, we broadcast them in our newsletter and get them out to the public and say, "Hey, look at this really cool thing that people showed," which I love. I love that we can give more attention to the stuff that is really solid.

ADAM: Yeah, and that probably puts more eyeballs on that research than it had the first time around. I know even selecting for the fact that these come out in high-profile journals. But if you're sending an email to 250,000 people, most papers are not viewed by 250,000 people. And I think this is something that people also don't understand about publishing your work directly to consumers, that if your paper is in a paywall journal, most people can't access it anyway. And even if they can, they're not going to read it top to bottom, and they're not going to understand it; they're not going to invest a lot of time in it. And so if someone does a replication, writes it up in an accessible way, and sends it out to that many people, that's way more attention and way more access to the general public than you would ever get just doing something for a journal. And why wouldn't you want that?

SPENCER: Yeah, that's exactly how I think about it. And hopefully people feel good about that. We did that with your moral decline paper because it was a great paper. So we sent it out to our audience. We're like, "This is really cool."

ADAM: That must be why I'm still getting emails about that. People love to send me emails about that paper. Oh, nice. That's just like, "Well, my neighbor put up a fence." Literally, someone was like, "I know moralities decline because there are so many more fences in my neighborhood now. Why would they put up fences if they're nice people?" And I'm like, "You know what? I'm retracting the paper."

SPENCER: Issuing your retraction. Adam, before we wrap up, we talked about a lot of things that are wrong with psychology. But what would you like to see in the future? What do you think is going to push things forward, make this into a better project that can actually help answer important questions about critical topics?

ADAM: A few things, one is serious reflection on the history and philosophy of science as to, "What it is we're doing here? When you sit down to do psychology, what is it you're doing?" It sounds like the dumbest question, but it wasn't one that anyone talked about in graduate school. Everything was learned by pattern matching. I was lucky that my advisor was really good at being the mentor to me, being the apprentice. But most people don't get that. Most of what we get is just the Mad Libs version of what we've already gotten. I think what we've already gotten is not that good, and so more of it is also not going to be that good. So thinking about what it is that we're doing, what I think we're not doing is thinking about the world in terms of rules and entities and trying to do what has been successful in other disciplines. But beyond that, that approach might not work. What I want to see is more diversity of approach, more people doing more weird stuff. It is striking to me how mainstream scientific institutions are. I never see something that scandalizes me for being weird, and I would really like to, because it just seems impossible that all the good ideas left from here on out are going to look very normal and legible. The fact that we are not producing any weird ideas is really concerning. Whether that means we need to get more weird people into graduate school, or more likely, we need more strange institutions supporting more kinds of people doing more kinds of weird research and exploring the space that other people can't, those are things I think would help us out. We need more people putting their hands in more kinds of buckets. We need to fill the buckets with many different substances and see what happens when people put their hands in them. Those are the three things that we have to do.

SPENCER: Adam, thanks so much for coming on. It was great to chat with you.

ADAM: Thanks for having me. It's been fun.

[outro]

JOSH: A listener asks: "Has anyone reported life-changing impacts due to your work?"

SPENCER: Yeah, it does happen from time to time. One example that's very salient to me is a number of years ago, I got an email from a woman who told me that it was something like 2 or 3 a.m. and her husband had been beating her for a long time and she'd escaped into her car and was reading online about how to make difficult decisions because she was thinking of leaving her husband; and she read one of my essays and she felt that that was what she needed to make the decision to finally leave him. And I thought that was very meaningful. Yeah, that's a cool story.