SPENCER: Nick, welcome to the Clearer Thinking Podcast.

NICK: Thanks, Spencer. It's good to be here.

SPENCER: Many people who listen are probably familiar with the idea that we often can do more good with our money if we think carefully about how to optimize or be effective when we're doing charity, but optimizing can also be a problem. Do you think that people often are optimizing for the wrong thing when they do charity?

NICK: I think there's a set of pretty intrinsic incentives or structural flaws in optimization modeling that you've got to be really careful of. They don't inevitably happen, but it's just really easy for them to happen. They're things like biasing towards what's measurable. Because you put numbers on things, you might assume that you can be more confident than you actually are. The order of magnitude that you are able to understand or put a probability on something can give you higher confidence than you probably should have. I'm someone who's a big fan of optimizing for effectiveness; it's what I've spent my whole life and career doing. I just think there's a set of these structural incentives that it's really easy to underestimate, and that often cause people who optimize to bias towards the incremental, to bias towards the easily measurable in a way that has really unintended consequences for how to actually have the most impact that you possibly can.

SPENCER: A critique that is sometimes leveled at the effective altruism movement is that if there are interventions or ways of helping the world that are really hard to measure, but are actually really effective, for instance, trying to create regime change in a really authoritarian regime, that's really hard to measure, and it's a bit binary, but it could be really effective if it worked. Effective altruism and other movements that focus on measurability may struggle to find those interventions, and they may simply avoid them.

NICK: Yeah, I think that's exactly right. I've certainly been part of plenty of efforts to try and assign values to how probable a certain political change is if we were to work on it. I've worked on dozens of campaigns in my life that aim to influence political policy, and you can often do a napkin math estimate of how often this has happened in the past, what levers we have, how much probability we can assign to it, and give yourself a sense that maybe there's a 5% chance that this shot on goal will result in a particular policy change that could affect a hundred million lives, or whatever it is. But the number of assumptions that have to go into that and the order of magnitude that you can be wrong with those assumptions is really wild. I'll give you an example. When I was 19, I worked on a campaign in Australia that aimed to influence Australia's aid policy. We were trying to increase both the quantity and quality of the Australian aid program by campaigning as part of a federal election to get the major parties to make commitments to increase Australia's aid. We had no right to be doing that campaign. We were a bunch of bold kids taking this shot, and it was a great campaign. We engaged with both parties, we created platforms for them to make announcements from, and the leader of the opposition at our event used that event as an opportunity to pledge to increase Australia's aid by about $3 billion a year. I can't say with certainty that we were the only factor or the driving factor in that decision, but I know we were a factor, and I feel really good about that. What probability would I have put on that campaign working when we started? I don't know. This was before I really started to get into cost-effectiveness models and those kinds of things, but that policy ended up lasting for three years before it was wound back by the following government. Over the course of those three years, there was an additional $3 to $5 billion going into Australia's aid program, which then cascades through all of the normal political decision-making dynamics. It goes to some programs that are effective and some that are ineffective. I was super bummed when it got rolled back because we'd fought so hard for it, and all of that was just so hard to forecast. If we had tried to put a really high-confidence model behind that in our decision-making, I don't think we would have tried at all, and I still think it's plausible that it's one of the most impactful things I'll ever do in my life. So, what do I do with that? I think we still need models for decision-making, but it's really easy to end up biasing towards these known incremental, easy-to-measure interventions that trap us in incrementalism.

SPENCER: It's also crazy to think even in retrospect, based on the fact that you don't know what the probability was that it would have worked, right?

NICK: Totally, maybe it was a one in a thousand chance.

SPENCER: It's so hard to say. But I am surprised to hear you say that you think it's one of the most impactful things that you've ever done, given that you're the CEO of GiveDirectly and you were the CEO of Change.org. Are those not more impactful?

NICK: I'm not sure if we'll ever know, but certainly, is there any other moment in my life where I can, with some sort of confidence, say that nine months' worth of work led to $3 to $5 billion dollars of net new money over the following few years? It's really hard to replicate that. So maybe I've been chasing this high ever since.

SPENCER: That's, yeah, it's pretty staggering. On a per day basis, that's a lot of money.

NICK: Exactly.

SPENCER: So, do you think that it's a mistake to try to put numbers, or do you think that we need to try to put numbers, but we have to do it carefully?

NICK: I generally think that it is useful to try to quantify things, but where I have landed is I have a mental model or a way of quantifying numbers that allows me to understand at a high level what's the order of magnitude that could be possible through this intervention. Are we talking extremely high, very high, high, medium, low, and then are we talking from a probability perspective, low, medium, high? I'm not sure once you get more precise than that how valuable that is. It seems to me that it starts to get to false precision. I still think it's useful to enumerate all of the assumptions that you need to make for something to be true, and that is often the purpose of quantifying things. It turns out this funnel is actually longer and narrower than you expected. More things need to go right, and the probability of those things going right is actually lower than you would expect. That can be a really useful forcing function in and of itself, but for me, it's about using a set of principles to correct for the biases that exist in over-quantifying. Those principles end up being around weighting interventions that have the potential for significant scale and saying, "I'm just going to make a structural choice in the way that I spend my time to prioritize interventions that I think have uncapped upside." If they do well, the upside is very high. I could go through an exercise to quantify what that upside could be or what the range could be, but I actually think that's a little bit of false precision. I don't think it's going to take us very far, but I can say with some degree of confidence that if it goes well, the impact is extremely large. For me, at least, it's about having a set of principles that correct for some of those systemic biases in the optimization mindset that are about correcting for scale, correcting for risk and unknown, and making sure that those are forefront in my mind for the things that I prioritize my time on.

SPENCER: The idea of trying to work on something that, if it goes really well, has this huge upside potential. I think it's a more general, useful life principle. Obviously, it doesn't apply to everything; it probably doesn't apply to your personal hobby, but if you're doing a startup, I think it's a good principle. You're going to devote many years of your life and work hard for this, so why not work on a problem where, if it goes super well, it has a huge scale? Or even on something like a dissertation, if you're doing your PhD, work on something where if it goes super well, you feel like it could have a really big impact, a really big upside in whatever field it is you're working on. Of course, it has to be balanced against other factors; you don't want to work on some possible problem just because it has a big upside, but it is at least one consideration. I think that's useful.

NICK: Totally. A lot of my thinking on this has been informed by my time working in the technology industry. The 11 years I was at Change.org were steeped in startup culture. We raised venture capital money and were in and around that operating scene, and that really seeped into my brain in many ways of seeing the world. One of the things that I think the venture capital ecosystem gets right is the first question they often ask is, what's the market size for this problem? Before they even ask, is this a good founder, is this a good idea, is this a good operating model? It's just, is this problem big enough? The vast majority of startups that get denied VC funding don't get denied based on the quality of the idea or the quality of the founder; they get denied based on the market size. They just say this problem isn't big enough to support the venture capital model, which requires billion-dollar-plus returns and exits. Now, there's certainly a role for businesses that don't rely on the kind of venture capital economics, and I think there is a role for nonprofits and organizations that do work on the margins and try to get the small things right and incrementally move the needle forward on the little things. But I wish that more of the doing-good infrastructure in the world had this orientation of organizing ourselves and orienting around really big problems that potentially have really big solutions, because the outsized upside of getting that right is just so big.

SPENCER: It seems venture capitalists almost have no choice, at least at the somewhat later stages, to focus on things that have at least some chance of being worth over a billion dollars, because if you think about the power law of returns in venture capital, which is a well-established phenomenon, a small percentage of startups will return most of the return. The key thing is you've got to have that set up in your portfolio. So you're really betting on some relatively small chance that it has an absolutely massive return, rather than trying to say, "Have 90% of your portfolio give you a decent return." That's just not the way a power law works.

NICK: Exactly right. That is the economics of the venture capital model. You get all of the results from this kind of power law, and I would hypothesize, honestly, that from an impact perspective, the world is somewhat similar. From a ways to improve people's lives perspective, there is an extraordinary power law, and the interesting thing is I think that effective altruism thinking is focused on the power law of cost-effectiveness, which is this thesis that you have some charities that are significantly more effective than others. But that's very focused on optimizing for the $1. There is a power law return on the use of this $1; one charity can be a hundred times or a thousand times more effective on the use of this $1. I think it's far more interesting to think about it from the perspective of the intervention that has the potential to be at that billion-dollar, $2 billion, $10 billion scale. It's a lot like hits-based giving. There are plenty of people who do think this way, and I think this has become normalized in the culture a little bit more over recent years. But I think the downside of optimizing for the marginal dollar means that you often lose sight of that bigger picture potential return of the giant problem at scale.

SPENCER: I want to come back to this idea of focusing on the marginal dollar and whether that's a trap to some extent or can be a trap. Talking about power laws is interesting to think about why you might find a power law in impact the way you do in venture capital. As a mathematician, I think about when do we get power laws, what generates power laws, and there's a really interesting mathematical finding, which is that when you sum a bunch of things together that are independent, you get a normal distribution, a bell curve, which doesn't have the same dynamics. In a bell curve, the tails kind of go away rapidly; they disappear rapidly. However, when you have a product of factors instead of a sum of factors, that's when you tend to get things that are like a power law or a log-normal distribution, which have these fat tails where you get a really wide distribution. Why would we have a product of things rather than a sum of things? You tend to get products of things when you have lots of "ands," when you have to have A be true and B and C and D. In other words, there are many ways something could kind of die off, and you need to multiply all of them together to get a result. I think that is true in venture capital; a startup has a kind of multiplicative effect of the size of the market and the talent and the luck and all these different factors. I think in charity it's probably true as well, this multiplicative effect of many factors rather than more additive effects, and it can create this fatter tail distribution.

NICK: Yeah, that's really interesting. Intuitively, when I think about, "Why I'm attracted to the idea that there might be a kind of power law distribution in doing good?" I immediately think of both economics and technology. When I think about the things that have most raised quality of life for the most people over the course of the 20th century, those have often been technological breakthroughs, whether it's in wheat production or economic growth, which has been the single largest driver of reduction in poverty over the last 30 to 40 years, with India and China driving economic growth. Those are very hard problems to solve, but absent the economic growth of China and India in the last 40 years, I think the world looks a lot different from a suffering and overall well-being perspective. Obviously, many factors go into that, everything from political leadership to governance decisions and industrial policy, but that's where I see some of those power laws coming in as well.

SPENCER: Do you think that we can put regular charity, as done by nonprofits, in that same bucket as economic growth? Because I agree with you, economic growth has caused tremendous benefits, but I wonder if it's just too different to even compare.

NICK: If I thought that I had the silver bullet for catalyzing economic growth at a national level, I think this is, we're talking about decades of development economics that has been circling around this question of what are the factors that go into enabling that in a sustainable way. What are the factors that enabled the Asian tigers to grow at the level they did post World War II? How do you replicate that in other markets? I think anyone who says that they have the exact formula hasn't spent enough time looking at all of the models and all of the different things that have been tried. I certainly think that there are some interventions, some charitable interventions, some nonprofit interventions that have the potential for the types of outsized power law returns. Is that going to be competitive with three decades of economic growth in China? Probably not. That's a level of impact that I think is going to be hard for any one nonprofit or organization to achieve, but certainly there are technological breakthroughs on the R&D side that I can imagine being true. I obviously am a believer in cash transfers, and I think that cash transfers have the potential to deliver at really extraordinary scale in a way that very few interventions do. That's one of the reasons that I'm at GiveDirectly, because I see the potential for this outsized upside. While I think there aren't that many, I do think that there are some charitable interventions that have the potential to compete at that level, maybe not quite at China economic growth levels, but certainly very high ones.

SPENCER: Let's talk about thinking on the margin, because this way of thinking, where you say, "What is the impact of the next dollar?" It's a very powerful way of thinking and often leads to good decision making, because very often, it's not like the second dollar and the third dollar and the fourth dollar are really going to be that much different than the first dollar. If you're thinking about impact per dollar, it's a reasonably stable quantity, and it can help guide you to things that are really effective. This is true in charity, but also could be true even in other things, like investing. You're thinking on the margin, the marginal impact of how you're spending $1, for example, but it's not a perfect way of thinking. It can be a trap. So, where's the trap here?

NICK: I think the trap is if we had perfect information, optimizing for the marginal dollar makes perfect sense, because we would factor in for this dollar that I spend not only how much direct impact it is having, but how might that dollar unlock further dollars or enable scale or have downstream catalytic effects that contribute to a long tail of impact from the way that this dollar is spent.

SPENCER: So either a multiplier, so it counts as more dollars, or maybe not just multiply, but might change the future marginal per dollar impact.

NICK: Exactly right, and so I think if we had perfect information, optimizing for the marginal dollar makes perfect sense, because we would have that multiplier built in. We would say, "For this dollar that I spend on this advocacy campaign to affect government policy, it has an immediate impact now, but it also has this potential downstream impact of changing the probabilities of this policy being changed," and so on and so forth. I just think it's practically impossible to factor in that type of downstream forecasting at the marginal dollar, and so what ends up happening when you optimize for the marginal dollar is that you exclude all of that downstream stuff. You basically say, "We don't have enough information, we have no way of actually factoring that in. Is it a 0.1% probability? Is it a 1% probability? Is it a 10% probability? Who knows? Therefore, let's just leave it out, or if we do include something, let's discount it for uncertainty by so much that it becomes negligible." In practice, the decision to optimize for the marginal dollar becomes a decision to optimize for a local maximum, not a global maximum of what you could actually be achieving. For me, that's the thing that gets lost: the factoring in of that downstream consequence. You could say, let's take GiveDirectly as an example, "Depending on what cost-effectiveness model you use, you can make the argument that cash transfers are reasonably effective. Maybe there are some things that are more effective out there, maybe there aren't, but a cash transfer to someone is worth x amount of human value. Compare that to vaccines, compare that to malaria bed nets. You can come out with your numbers, compare them, and decide where your dollar goes." One of the reasons that I am excited about cash transfers is that it's one of the few interventions that has a credible pathway to effectively absorbing and deploying tens of billions of dollars quickly, and the world needs interventions that are capable of doing that. There is no world in which that happens without being committed to growing that over time in a way that builds out the infrastructure, builds out the evidence base, and increases the probability over time that someone wakes up in the morning and says, "I would like to sign a $20 billion check, and I would like it to be deployed in the next 24 months. How can I do that effectively?" When you ask that question, there's really only one answer, and that answer, I think, is cash transfers. Now, if you had to tell me to put probabilities on how likely is that to happen, how confident am I that that's going to happen, and also what are the things that I'm doing to increase that probability over time? It would be so speculative as to almost be meaningless, but I think that every minute I spend working on cash transfers has a flaw of impact, the marginal dollar, but it has an upside that is almost unbounded, because I think it has a huge market size as a potential intervention. I think there are many interventions that have a similar profile, where you have some local maximum of confident impact, but then some unbounded upside of potential impact on top of it.

SPENCER: There's a lot that's really appealing about cash transfers, especially when the cash transfer is giving money directly to incredibly impoverished people, because then you can assume that there are a lot of needs that they can fulfill with money, and even more so if they're living in an area where things don't cost very much. They can actually get a lot of their needs met very cheaply. One thing is it's just relatively easy to administer. I know that GiveDirectly does a lot of work on figuring out how to administer it, but compared to giving something like a vaccine that you might need to keep cold and you have to ship, money is relatively portable. It's also something that's needed almost anywhere you have poverty. You don't have to understand the local needs. It also has this nice libertarian argument to it, that people are pretty good at knowing what they need. That could break down; in theory, you could have a place where people just spend it on drugs, but that doesn't seem to be so much the case. There are a lot of studies, and it seems people spend on things that they care about.

NICK: You just said it better than I probably can. I think the scalability profile of cash transfers is one of the most interesting things about them. There's no cold storage, there's no supply chains. It is more complicated than you would think to deliver cash to people in extreme poverty. There are a bunch of negative externalities to manage, and there are a bunch of positive externalities to maximize. It's operationally far more complicated than many people realize, and yet it is still so much simpler. I can click a button and watch money go from the bank account of GiveDirectly in New York and trace it all the way to the individual in Rwanda or Malawi. Those people, with the most information about what they need and the most incentive to stretch that dollar as far as they possibly can, will go out and solve their problem in a way that is totally fungible. I was reading about someone who got cash from GiveDirectly last year. She's a 75-year-old woman in rural Kenya who historically has had to walk five kilometers every day to get clean water. With her cash transfer, she used almost 90% to 100% of her cash transfer to buy a 10,000-gallon water tank. She pays to have that tank filled with clean water once every two weeks, and then she resells it to her local community for a small profit. There are no nonprofits in the world that are going around saying, "We need to find 70-year-old women in rural areas and help them buy 10,000-gallon water tanks to resell to their local community." No one else would have come up with that except for her, but she has generated a sustainable income for herself, provided clean water for her local community, and done it in a way that stretches the dollar far further than if some army of consultants came in and said, "How might we solve clean water in this village?" I think that's the kind of ingenuity that you get when you really follow this instinct to ask, "Who has the most incentive to use this money well, who has the most information on how to use this money well? And let's get it to them as efficiently as we possibly can." Yet when GiveDirectly started, there was a lot of skepticism about whether or not that would work, but I think now it's pretty widely understood that it is an incredibly effective thing to do.

SPENCER: How would you compare giving people cash to the graduation model, which there's a lot of interest in, which involves, as I understand it, some cash, but also bundles it with other things? Proponents of it claim that it is actually more effective because the other things bundled with it allow for longer-term impacts.

NICK: Yeah, so for your listeners who don't know, graduation is pioneered by an organization called BRAC, an amazing organization that came out of Bangladesh. Graduation is a package of interventions. It involves an asset transfer. The asset transfer sometimes is cash, sometimes it's a cow, sometimes it's livestock or something like that, but there's an asset transfer of some kind. Then there's often coaching, like rigorous coaching over a long period of time, that's about how to use this asset best, how to maximize return on it, and so on and so forth. Then there is a range of other interventions. The original graduation bundle was five different things all packaged together over the course of a couple of years that aim to take someone who is ultra-poor and sustainably graduate them out of poverty. I'm a big fan of graduation. I think it has really good evidence behind it, depending on the context. I think it has performed better or worse depending on the implementation, but I think there are really promising examples of graduation. If the question is, "How does cash compare to graduation?" I think there are some contexts where the evidence for graduation is actually more compelling than cash at a cost-effectiveness level. I think there are other contexts where the evidence is more compelling for cash on a cost-effectiveness level. The core reason I get more excited, or at least why I personally work on cash rather than graduation, is that scalability quality profile. Graduation is a much more complex program to administer. It involves many more moving parts with many more staff over a much longer period of time, and so the quality variation is quite high. You get some programs that are extremely cost-effective, and you get some that are actually fairly marginal in their impact. The costs to deliver are very wide as well, and it takes a lot of time to scale up because it's very operationally intensive. Do I think the world is better for having good graduation options? Yes. Do I want graduation to scale? Yes. Is it a good thing to donate to? Yes. Why do I work on cash? I think it scales faster. I think it is more consistently high quality, and it will ultimately reach more people. It's about that upside for me, which is why I work on cash. One really interesting thing about cash and graduation is that over time they're getting closer together. When graduation started, it was this very complex program of five different things administered over two or three years. It was quite expensive to implement, and it was hard to maintain quality control. Over time, people have been trying to strip back, "How do we do graduation light? How do we make it lighter touch?" It's starting to look more and more like a cash transfer with a little bit of coaching and a little bit of something else. We're also kind of going in the direction of asking, is there coaching that we can layer on top of a cash transfer to allow people to have longer-term impacts?

SPENCER: Just convergence?

NICK: There's a little bit of a convergence that's happening over time, which I think is really interesting.

SPENCER: Yeah, that's fascinating. It's great to work in a field where you can talk nicely about your competitors, where you're like, "Yeah, we're all just trying to help people."

NICK: Totally. It's natural, I think, in many organizations to sometimes feel competitive with other interventions. Sometimes you're competing for the donor dollar or anything like that. Whenever I hear that coming up, I'm like, "Guys, there are more than enough problems in the world to go around. We can take a total abundance mindset here, which is that the goal is to end extreme poverty and to increase human welfare. If people can find ways to do that in a more cost-effective or scalable way, or in a more compelling way that mobilizes more money, hell yeah."

SPENCER: One thing I wonder about with cash transfers is, my understanding is that GiveDirectly started giving them to individuals or households, I guess I should say, but moved to a village-based model. How much of that is logistical versus it helps avoid envy or other social issues versus it's actually more effective?

NICK: It's interesting. Yeah, so when we started, we would go into a community and we would try to identify the poorest households in that community.

SPENCER: Using roofs and things.

NICK: It would use a proxy, and the proxy was roofs. Basically, if you had iron sheets on your roof, then it's actually a pretty good proxy that you're more well off, and if you're using a thatch roof, then it's a pretty good proxy that you're on the poorer side. So that's how we started. Over time, we have now shifted, where our base model, there is some variation, but our base model is to identify a community in which you have an extremely high overall extreme poverty rate, say 80 or 85% of the population is below the extreme poverty line, and then we saturate every single person in the community to get a cash transfer. There are a range of drivers for that. The first is operational ease and simplicity. It's definitely more complicated to go through and try to identify which houses are poor and which ones aren't. It's much faster to saturate, and because we are interested in optimizing cash transfers, not on the margins, but for the billions of dollars range, ease, simplicity, and scalability are core to all of our program design questions. We're always asking how we can make this so that we could absorb hundreds of millions of dollars tomorrow without having to bat an eyelid. From an ease and operational simplicity perspective, that's the first driver. The second driver is definitely negative externalities when you pick and choose who gets transfers and who doesn't. Firstly, you will get that wrong sometimes. Roofs are an imperfect measure. You will also trigger unintended behavior. People might figure out how you're making these decisions, and then they won't invest in a roof because they think, "Well, maybe if we don't buy a tin roof, then the people who come with free money will give us more." You've got to be really careful of those unintended consequences, and intercommunity envy and conflict are real issues. Making the choice to saturate as a whole significantly reduces those negative externalities. The final one is positive externalities. We did a big RCT, and results came out several years ago looking at the general equilibrium effects of cash transfers, which is another way of saying the economy-wide effects of cash transfers on people who are not just the recipients. They found that for every $1 in cash transfer, there was $2.50 in economic activity in the local community. What that meant was that people who didn't receive cash transfers but lived nearby benefited almost as much as the people who received cash transfers. That's a wild thought that I can send a year and a half's worth of income to you and the person that lives 15 minutes away from you benefits almost as much as you do from that cash transfer. But that's what happened, and that really helped us understand cash transfers not as an individual charitable intervention, but as an economic intervention. We are stimulating the local economy in a way that aims to benefit everyone in that region. When we take that framework, this model of saturating everywhere aims to amplify those economic effects.

SPENCER: It might actually seem impossible to people that you can get $2.50 of economic gain from $1 given. At face value, it doesn't make sense. Is this a kind of money multiplier? Can you elaborate? How does that even happen?

NICK: In some ways, this is another contrast with graduation. Think of the difference between giving a good and giving cash. If I were to go into a community and give you a cow, what do you do with that cow? You start feeding it, you start milking it. Maybe you sell some of that milk, maybe you consume it, but on the whole, you are largely consuming the goods associated with that cow. If, on the other hand, I give you cash and you then go buy that cow in the local community, the person you bought the cow from now has that cash. What do they do with that cash? Maybe they bring on more labor to sell more cows in the future, or maybe they make an investment in another local business. That $2.50 really comes from that money just recirculating within the economy, with people buying and trading goods and hiring more labor. More people have jobs, and those people then have more money to spend themselves. It's this money recirculating that drives that spillover.

SPENCER: It's really interesting to think about how that kind of propagates outward. So you give someone money, they go buy a cow. Now, how much have they gained? Well, they basically got the money for free, so they kind of got a free cow in some sense. And then, you think about that next person that they bought the cow from. Well, how much did that person gain? Well, they gained the difference between what the cow was worth to them and what the money was worth to them, so some kind of surplus value. And then they go buy labor. Well, how much did that person gain? Well, they gained whatever the income of that job is. And you keep working. It's really complicated to think about the effects of that.

NICK: Totally, and it's one of the really interesting things about economic multipliers. They are a really good example of something that is easy to miss when optimizing for the dollar on the margin. Economic multipliers are really hard to measure; they're expensive. The type of RCT that can measure economic multipliers.

SPENCER: Randomized Controlled Trial?

NICK: Yeah, the type of trial that can measure economic multipliers is giant, expensive, and extremely unusual, and so they just don't exist for most interventions. One of the things that ended up substantially dampening interest in microfinance was when you were just measuring benefits of microfinance on an individual recipient level. They looked really good; people would receive a microfinance loan, they would invest that loan, they would then start their business, and then they would grow that business over time. You kind of study that, and you're like, "Oh, this is really cool, this is great. It seems like a good intervention." It wasn't until — I'm oversimplifying this a little bit — but when there were general equilibrium studies that looked at the economy-wide effects of microfinance, you started to see in some of these studies that the person who got the microfinance loan benefited, but what happened is that they then just outcompeted someone else in the community, and so they had access to cheaper capital. They then outcompeted someone else in the community, and so that person had net negative value due to the microfinance loan. The person who received the loan had net positive, and for many contexts — not all, but for many contexts — the net impact was a wash. That is the type of counterintuitive, or it's kind of intuitive once you say it out loud, but many people didn't anticipate that outcome, and it's quite hard to measure. Yes, I think there are many interventions out there where we know far too little about both the positive and negative spillovers, or what the economy-wide effects of them are. In most cost-effective modeling, those numbers are assumptions that are made up and then heavily discounted. For many years in GiveWell's cost-effectiveness analysis of GiveDirectly, there was a net minus 5% negative spillover applied because of the assumption that cash transfers drove community conflict. Not crazy; reasonable, that does happen in some cases.

SPENCER: That's fascinating to me, because if there's anyone who I would think would pick up on it and be aware of it, it would be GiveWell, because they think so rigorously about that, but even they missed it.

NICK: Yeah, and to be clear, it wasn't necessarily unreasonable. They were optimizing for certainty, and they would prefer to be conservative rather than optimistic. When we didn't have the evidence, it would have been crazy to add a really significant positive multiplier, because in many cases that would be far too optimistic, and you wouldn't be confident enough about the outcome. But then recently, as a result of this study that I was describing, GiveWell factored that in, and that was actually the major driver of GiveWell's updated cost-effectiveness analysis of GiveDirectly. That kind of increased the cost-effectiveness by three to four times, all because of those economic spillovers. It went from a net minus 5% kind of addition to — I actually don't know what it is now — it's like plus something significant, and even then that is heavily discounted. They've discounted that by 50% because there's still only one study that does that general equilibrium assessment, and that's a reasonable decision by them. You want to get the evidence to know that this is replicable, to know that this happens in other circumstances, but I actually think this is a really good example of when you're optimizing for certainty and for measurability, you risk missing some of that upside.

SPENCER: I think these second-order effects are a lot more common than people realize because people aren't used to thinking about them. For example, programs that try to help people get a job help them work on their resume. It could seem really valuable because you're taking someone who is impoverished and doesn't have that kind of help, and you're helping them get a job, which could be really beneficial for them. But then you have to say, "Well, aren't there a finite number of jobs at any given moment, and who else would have had that job? Okay, maybe that person's slightly poorer than the person who would have gotten it, but maybe only slightly, and how do we even know that they're poorer than the person? Because maybe the people who are more aware of these programs are actually slightly less poor than the other people applying." So it's really complicated, and in some cases, you can completely nullify the impact.

NICK: Hundred percent. And one really interesting example of this is actually at a systems level. I was recently having conversations with senior ministers in an African country a few months ago, and I was asking them how the current administration compares to the previous administration, the U.S. administration. What is your experience of previously working with USAID to now working with the State Department under the former and the current administrations? What I got back was we 100% prefer the new administration. Why is that? Under USAID, the way that it worked is all health delivery or program delivery was being done directly through NGOs. USAID would contract or grant to individual organizations to then deliver a certain service that could be a vaccine, bed nets, or whatever it is to deliver that. USAID wanted a direct relationship with those organizations to make sure it was being done effectively and accountably, and that they could account for those dollars. They were worried about corruption and so on and so forth, and so that was how USAID worked. How the new State Department is working is that they're forming these big things called global health compacts, where they do a deal at the government level, and they say, "We're going to give you $2 or $3 billion to the government, you need to stump up a bunch of money. Also, you need to invest in the types of things that we think are also important, say in global health, and you need to throw in these mineral rights or whatever it is. They're kind of trying to get a bunch of America First gains out of this as well, and then at the end of the day, you get to decide how to use this money, and you get to administer it as part of the government healthcare system."

SPENCER: Well, who wouldn't love that if you were in the government, right?

NICK: Oh, 100%. Now, what is the net value of a cohesive strengthened healthcare system versus a patchwork of NGOs that aren't talking to each other, all delivering random interventions that on their own make sense, but collectively are a mess, and also collectively are being done outside of the healthcare infrastructure that that country needs to maintain over time? I think you can make this case for a lot of EA-type interventions that they are entirely oblivious to the system that they operate within, with the intent of controlling cost-effectiveness as much as possible, focusing on delivering a service as efficiently and cost-effectively as possible, and ignoring and not engaging with the kind of government infrastructure that plausibly has far more sustainability and far more scalability. Now, there is also a lot more uncertainty associated with it. There's a reason why USAID didn't trust a bunch of governments to deliver things because those governments didn't always have the administrative capacity. Maybe there was corruption, maybe there were other problems, and so there are no black and white answers here.

SPENCER: The tracking is harder, too.

NICK: It makes tracking harder. There are all sorts of reasons why it might be reasonable to not go directly to governments, but the unintended consequences of building out all of these programmatic interventions that bypass national systems mean that money is ultimately not passing through state infrastructure, and that the state doesn't get stronger over time. There are some pretty big potential unintended consequences for the overall health of the health system, and I think they're the types of negative externalities that are never going to get measured. That is an immeasurable potential downside consequence, but another good example of both the upsides as well as the downsides of potentially impactful positive or negative externalities.

SPENCER: This reminds me of the sort of general observation that everything is way more complex than you think, and we only think things are simple because we haven't paid enough attention to them. As soon as you start looking at anything, like doorknobs, it seems simple. Then you start learning about doorknobs and how many different types there are, and all these different considerations they make when they manufacture them. You're like, "Ah, okay." But obviously something like global health is even more complicated than doorknobs. There's a sort of endless complexity when you dig into it.

NICK: Totally, totally,

SPENCER: It can be a bit discouraging, but at the same time, we have the benefit of lots of people who think very carefully about these topics that we can try to learn from.

NICK: Yeah, I think that's right. I think it is totally possible to be overwhelmed by the complexity of it all, and I think this is where more first principles thinking about some of these things can actually end up taking you further than the minutiae of a 130-cell cost-effectiveness spreadsheet.

SPENCER: You mentioned earlier one of your principles is around risk and the unknown. What's that principle you use?

NICK: I think a lot about incentives in systems, and when I think about the incentives inherent in the system that is trying to make the world a better place, like let's call it the nonprofit sector, or maybe it's EA-aligned interventions, or whatever it is, there is a huge incentive against risk; failure is heavily punished. I think it's slightly better in the EA world than it is in other places, but it's still heavily punished. Certainty is heavily encouraged; epistemic humility is encouraged, I think, in EA, but even then, nothing gets rewarded like a sexy RCT that gives you some real data to put into your cost-effectiveness models. Certainty is highly prized, and evidence is highly prized. When I look at the incentives, I think a well-balanced ecosystem of interventions has a portfolio of high-confidence, high-evidence things and high-risk, high-reward things. In the same way, in the finance world, we have venture capital for high-risk, high-return things, we have private equity for higher confidence, but still a bit of risk. You have the public markets, you have bonds, you have land; you have this big portfolio, and I think generally the system as a whole is benefited by having these different risk profiles. I personally choose to prize higher risk, higher uncertainty bets, not because they are inherently good, but because I think they are inherently underweighted in the portfolio of interventions that we have at our disposal. I see more net value and benefit in doing those things that have more uncertainty associated with them, or that have higher risk associated with them, because I trust the system less to incentivize that. I think overall we would be benefited by reweighting it a little bit more towards those lower confidence, higher uncertainty, higher risk interventions.

SPENCER: I tend to have the intuition that things that are really certain to work tend to get saturated relatively quickly. It's not always true; sometimes you get something that's really highly confident, and actually it's not saturated, maybe because it's really large scale, like giving cash. It seems really likely to work, and it's a huge scale; that's why it's not saturated. But at the same time, things that have a one in 10,000 chance of working, I think often a problem with them is that we don't really know if it's one in 10,000 or one in a million. They're such crazy long shots that we can't even really think about them clearly, but the things that maybe have a 10% chance of working, I think that's a lot of really interesting things in that region where they get underexplored, and a lot of startups are in this realm. I think the general consensus is something like 90% of startups fail.

NICK: Yep, I think that's exactly right, and I wouldn't discourage anyone from working on something that's like a 0.1% chance of working out of a hundred, but I certainly personally like to be in that sweet spot of somewhere between 1% and 10% or 1% and 30% probability. Partially that's probably somewhat selfish in the sense that I want to take some shots on goal at work. I want to take as many shots on goal as possible. I don't think I would personally feel great about taking 50.1% shots on goal in my life, and the expected value of that may kind of be net good, but if I got to the end of my life and I was like, "Well, turns out I made the right bets, but none of them paid off, I think I would be pretty bummed by that."

SPENCER: Yeah, but even with 10% bets, if you're only taking one of them, that's pretty risky. I think it's helpful to think of it as part of a bigger process; you're going to take multiple bets in your life, so it's okay to take one 10% bet here and there.

NICK: I 100% agree with that. I think we all get to take several big bets in our life. The biggest bets that we take are about how we allocate our time; that's the most limited resource. Choosing to work on one problem for five years, I generally think that it is only worth working on a problem if you're committing to it for at least five years. That means you can probably take somewhere between five and 10 or 15 shots on goal over the course of a lifetime. Thinking of that over time is a good way to think about that probabilistically, but there are lots of lower-risk shots on goal that we take all the time in our giving, like what we personally choose to give to, who we choose to help. You can help out an organization even if you're not dedicating 100% of your time to it, and I like to think of that as a portfolio and try to have that mix of risk and return as well.

SPENCER: What's wrong with the reasoning that says, "If you're just focused on helping the world and you're really being altruistic, shouldn't you just put everything into the highest expected value bets, no matter how low the probability is?"

NICK: I sometimes hear this argument, and people will say to me, look, if you do your assessment and you say, "Here's one thing that really comes out - all of my principles, all of my cost-effectiveness analyses, all of my napkin math points that this one thing has much more return than something else, because it's neglected, or because I have an unfair advantage on it in some way." So, why wouldn't I put all of my time and all of my money into that one thing? If someone feels that way, go for it. I personally have far too much uncertainty to do that. I think I'm, with some humility, pretty good at many types of social change now. I've spent more than a decade on political campaigning, I've spent more than a decade on different types of philanthropy and capital mobilization, and I've spent more than a decade on technology delivery on all sorts of different things. My confidence in my ability to tell the difference between a 75th percentile giving or work opportunity and a 95th percentile giving or work opportunity is really low, just because of the number of assumptions that need to go into it and the variation that might happen within those assumptions. I don't think it's crazy to go all in on a single bet, but for me, at least, I have too much uncertainty to be confident in any one conclusion that I come from. From a kind of net impact perspective, I much prefer to be like, here are four or five different things that I think all make sense at a similar order of magnitude. I'm going to spread my bets against those because I think that's net better for the world, because it doesn't rely on me being right on everything.

SPENCER: GiveWell had this really interesting observation as they were studying more and more charities early on to try to figure out which are most cost-effective, which is that, if I hopefully state it correctly, those that seemed most impactful were also most likely to regress. In other words, if you think something's a thousand times better than everything else, almost certainly you've made a modeling mistake, or there's some piece of information you're missing, and that number's going to come way down. If you think something's only twice as good, it's less likely to revert way back when you get more information. Now, it probably still will tend to revert somewhat, and this might come about because there's sort of this distribution of doing things in the world, and there are very few things that are really that good. If you think something's really that good, there's a pretty good chance that you're actually missing something, and it's not that it's that good; it's that you have missed something important in your analysis.

NICK: Totally, I think this is spot on. I see this all the time in the research into development, where a new randomized control trial will come out about a particular intervention, and the results will be astonishingly good. It will be a robust randomized control trial, and people will get really excited about it. They'll be like, "Holy shit, we found the new thing that is 20 times, 30 times more cost-effective than anything else. Let's scale the hell out of this." And then, as more trials come in, you get a reversion to the mean. Not that it ends up being null, but it ends up being nowhere near what that first thing showed.

SPENCER: In psychology, the field I work in, that is a big knowledge.

NICK: Yeah, I mean that is more frequent than you would hope in all fields, I think. But I think, why might that be true? There are so many reasons why that might be true. It might be true that the location in which the first randomized control trial was done was uniquely fertile for the intervention, and those same characteristics don't hold in other locations. It could be because quality is hard to scale, that there was some special source about when the founder of this organization was directly managing all of the direct inputs themselves, they had such a good high-quality bar that you got this amazing result from this randomized control trial, but then when you scale it to hundreds of thousands of people and suddenly have random contractors and random countries doing this with no investment in the outcome at all, then the quality declines by half. It's very reasonable that that might happen. Or maybe the overall economic circumstances of the market change. There's just so much that we don't know about, and so I think it's very reasonable to expect that when one really amazing study comes out and you're like, "Wow, this is too good to be true," it's probably too good to be true, and you expect some degree of mean reversion there.

SPENCER: There's also interesting statistical reasons that you get mean reversion. Suppose that you're using a threshold like, "Oh, you need a statistically significant result to publish it," which is often true in journals, and suppose your study just doesn't have very many participants, like it's reasonably small. Well, then what you can show is that a lot of the time, if the effect is just modest, it won't even find it, it will be a false null, and so it won't get published, but if you accidentally overestimate the effect, you'll get over that threshold, and you'll publish it. So, there's a weird statistical bias where, with relatively small studies, the things that get published tend to be significant overestimates of the result.

NICK: Yeah, that makes total sense.

SPENCER: You mentioned implementation. Do you think that people often don't focus enough on how implementation can impact the different results?

NICK: Quality of the team is one of the number one factors that I look for in whether or not I want to take a bet on a gift, supporting as an advisor, joining as an employee, whatever it is. The quality of the team, the quality of the culture, the quality of the operating environment that an organization is in is such a high determinant of success. It is really hard for me to overemphasize how big a factor this is in how successful organizations are over time and how impactful they are over time. I can't even tell you how many organizations that I think have a really credible theory of change and a really good base of evidence, but they're just not well run. It just doesn't matter how good the theory of change is, it just doesn't matter how good the evidence is; if they're not well run, there are a thousand things that can go wrong in any given organization. I do think that this is something that is pretty systematically missed in the way that we assess giving opportunities and working opportunities. I think people probably prioritize assessing it a little bit more for working opportunities because they have some skin in the game for wanting to be part of a good team. If I had my way, in most cost-effectiveness models, there would be some section that looked at the leading indicators of really high-quality execution, but also the ability to scale that execution as an organization grows. I have both made all these mistakes myself and seen that take organizations down. I've seen them be set back by years, and it's sometimes not obvious how that can break impact models. Sometimes it might be in monitoring and evaluation; you're like, "Oh yeah, this seems like a good intervention," but it relies on a level of monitoring and evaluation quality that allows you to then correct things over time. Without having your eye on the ball, you can go years without identifying a problem that could cause millions of dollars of waste. With some humility, GiveDirectly is somewhat guilty of some of this early on in our lives, where early on we were pretty dismissive of doing government relations in the countries that we worked in. We thought this was a bunch of meetings that made everyone feel good, that seemed like a bit of a waste of time, where everyone just exchanged pleasantries, and it didn't seem like a good use of money. We were just going to get on with our job, give people money as efficiently as we can, and then that's it. A number of years ago, that really bit us. This was in Uganda, and it was a volatile political situation. Some rumors started spreading that we were essentially buying votes or influencing the political process by the way that we were distributing cash. No one knew who we were in government. We had no champions, we had no allies, we had no friends. We were banned, we were shut down, and millions of dollars that we had invested and were ready to deploy got stopped in their tracks because we hadn't understood what it took to execute at that level of scale in that context. It took two years for us to dig out of that hole in Uganda. Now we have great relationships, and that's a lesson that we've since learned, and we now apply it to all of the countries that we're in. We no longer think that that's a waste of money; we've learned that lesson. Now we have government relations in all the different countries that we work in, but it's such a good example of how these small execution factors can just totally make or break your ability to achieve an outcome. It's really hard to assess those things from the outside, but again, maybe learning a little bit from the VC model, overwhelmingly VCs, particularly at an early stage, will make decisions based on two things: what's the size of the market and what's the quality of the team. Everything else is a secondary factor. I wish that there was more focus on the quality of the execution and the quality of the team in how we make decisions to allocate resources.

SPENCER: In many fields that deal with interventions, whether it's global health or even psychology, it's really tempting to think about an intervention just as an abstract thing, like, "What's the effect of giving people $100 or whatever, or what's the effect of getting people to meditate?" It's very easy to forget that the implementation of that could have huge consequences in terms of effectiveness, and a really incompetent execution often will find no effect, no matter what. Even if it's the most amazing intervention. You can't really think about the effectiveness of an intervention as an abstract quantity; the implementation will always be part of that.

NICK: Yeah, 100%. I think it's really hard to measure these things. You could find probably dozens of interventions out there that, with the right execution, would have been amazing, but ended up getting null results. In the same way, you can find dozens of interventions that are going to be successful, regardless of the quality of the execution, or are far less dependent on execution quality in their outcome. Building the mental models, to be clear, people who are doing grant allocations and thinking carefully about giving often assess the quality of the team. This is a factor; we need more robust models for doing this, more shared frameworks, and for the casual observer, it's something that is probably underweighted.

SPENCER: I think even more advanced analyzers, when they're looking at effect sizes, find it hard to take into account the quality of the implementation. You just kind of say, "Well, we get the average effect size across different people who applied this, but were they doing a really good job on average," or were many of them not doing a good job? Is it a robust intervention where it doesn't really matter that much how good a job you do?

NICK: Totally, and what do you do with that? At some level, you've still got to gather as much evidence and make well-reasoned decisions as you can. I don't think we should ignore some of those things, but I do think what you just said, in terms of some interventions being more robust to the quality of execution, is important. This is one of the things I was talking about a little bit with graduation versus cash transfers. I think graduation can be incredibly effective, and there are organizations doing amazing graduation work. Raising the Village is a really great example of this; they do really great work. On the whole, I think cash transfers are more robust to execution quality because there are fewer moving pieces, and it relies less on getting every step of that layer cake right to achieve the outcome you're going for.

SPENCER: Before we wrap up, if you'll permit me to make a kind of awkward statement, when I think about Change.org, I don't think of it as a super impactful organization. I could be totally wrong there. I do think GiveDirectly is extremely impactful, but I'm curious how you think about those two organizations, how you compare them to each other, and feel free to sidestep the question of their relative impacts if it's not something you can talk about. I'm just really interested in how you think about that.

NICK: When I started my journey of trying to make the world slightly better, the first meaningful thing I did was work on this political campaign that I talked about. I had far more impact than I expected. We were able to move the needle on domestic foreign policy in a way that I think is very significant. I'm someone who always asks the question, how do you scale that? "That's great. How do we do more?" I spent many years after that working in political campaigning, understanding how to influence policy, how to do government relations well, how to do media campaigning, and how to mobilize people. I got really good at that, and I was a bottleneck. I found it really hard to scale myself because so much of it is context, so much of it is judgment, so much of it is reading a moment and seizing the moment, depending on how that context is changing. I was confident that I was having an impact, and I was excited about the work I was doing, but I did not see a pathway to ten times that, and so that's what led me to change at all, which is, "Okay, how do I ten times political change? How do I a thousand times political change?"

SPENCER: The number of people that have signed the Change.org petitions is staggering. In terms of scale, it's harder to get a bigger scale than that.

NICK: Totally, and when you think carefully about how you scale things, there are often hard trade-offs. In order to get to the scale that I thought was interesting for transformative political change, I think you've got to let go of a lot of control, you've also got to simplify, you've also got to find ways of reaching people that you wouldn't work on yourself or in other ways. At Change.org, the thesis was to radically simplify the way in which people can engage in the political process by providing powerful tools that go far beyond petitions, actually, and more about organizing in all its other forms, whether it's media, lobbying, or various things. We can give a way for people to participate, have influence, and shape the policies that affect them at a local, national, and international level in a way that otherwise wouldn't be possible. I joined Change.org. The first thing I did was open the Australian office, and I didn't care that much about policies in Australia, but what I cared about was, "Can I build a blueprint that allows me to open up new Change.org offices all around the world?" I spent years opening up offices in Indonesia, in India. We ended up having 20 countries where we had a very strong Change.org presence, many of which are in the global south, and I'm so proud of many of the campaigns that came out of some of those places, where we had 30,000, 40,000, 50,000 campaigns being started on the site every month, 50 million, a hundred million users, depending on which month you were looking at, re-engaging on the site every month as well. In Indonesia, we had anti-corruption campaigns, in India we had clean water campaigns, in Argentina we had education quality campaigns. If we had tried to do that in a top-down manner, I know there's just no chance, there's no chance in hell that we would ever come close to the level of potential change that might happen now. What's the net effect of all this? I think there are very good questions that you could ask because there are a bunch of policies that exist in the world because of Change.org, and I was like, "I would not choose that policy." Frequently, people campaigned on things that I passionately disagreed with, but the bet I was making, and I think it was a bet worth making, is that more people having a voice on the issues that matter to them at a local and national level would ultimately going to drive better policy choices, particularly in the global south, and that, in aggregate, was going to add up to extraordinary benefit. That is quite speculative, it's something that you're never going to be able to measure in some robust way, but that was the chain of thought, and that's why I'm pretty proud of that work. I actually think there's a lot of consistency going from Change.org to GiveDirectly because both of them bet on scale enabled by fundamentally empowering people to solve their own problems rather than us coming up with the solutions for themselves.

SPENCER: I think that's a great way to defend it, and insofar as it empowers people to essentially have a vote, so to speak, especially with governments that don't listen to the people that much, that seems really powerful. On the other hand, insofar as it just causes random changes, then maybe not so much. I think it's probably a mix. I think we probably both agree, but I think you made a good defense of it.

NICK: Yeah, I remain uncertain. I think it was absolutely worth the bet. I'm very proud of it.

SPENCER: It's incredibly impressive work. There's no question about that. It's very impressive. There's a massive scale. I think it's really interesting.

NICK: I think it's a good example. I started doing this type of work in 2005, 2006, 2007. This is before EA stuff existed. I was lucky enough to study under Peter Singer, who was one of my lecturers at one point in Australia. In some ways, the effective altruism stuff just kind of passed me by because I had committed myself to a political theory of change for more than a decade. I had this experience of working on aid policy in Australia, seeing that change turn into something powerful, and then being like, "Okay, I think there's something here. I want to understand how to do this a thousand times over this from a scale perspective," and I really dedicated myself to that. I was thinking through a set of principles that were, I'm interested in data, I'm interested in evidence, I'm interested in bets with huge upside but low probability. I was already internalizing a lot of those principles that now form what I do. I was very admiring of a lot of the maximize the marginal dollar work that was happening in parallel to all of this, but I also felt like it never quite perfectly fit how I was seeing the world because I didn't think it took into account political change enough. I'm interested in what the outsized upside bets that we can be taking that really understand what scale is possible. So, I don't know, it's kind of interesting to watch, actually, as over time I've seen this evolution away from high confidence, high certainty to more of a hits-based kind of giving and intervention approach. I think that has been particularly true in field building, as other cause areas have emerged beyond global health and development. I think first in animal welfare and then into AI safety and various things like that, where you kind of by definition have to take a bigger theory of change that takes into account politics and hits-based stuff. That's an evolution that I've been pleased to see because it maps to some of my early experiences.

SPENCER: I think the work you're doing in GiveDirectly is awesome, and I think a lot of people who don't know where to give should consider cash as an option. I think you make a really compelling case about it.

NICK: Thank you. I feel very lucky to do what I do. I have been with GiveDirectly one way or another for 10 years, the first eight of which were as a donor and the last two as CEO. Before I joined as CEO, I spoke to a bunch of GiveDirectly's major donors and asked, "Why is it that you give here? There are so many different options." I remember one donor I spoke to, a biotech investor who made a lot of money, said there are very few bets in the world that are heads you win, tails you don't lose. There are so many interventions out there with large error bars. In the best-case scenario, you do a lot of good, but in the worst-case scenario, it's not just null; maybe you made things worse. He saw cash transfers as a heads you win, tails you don't lose bet. Heads, you transform the sector or unlock an intervention that can credibly move tens of billions of dollars and catalyze tens of millions, hundreds of millions of lives out of poverty. Tails, you don't manage to shift the sector or mobilize the tens of billions, the hundreds of billions of dollars that are possible, but you have very confidently lifted hundreds of thousands or millions of people out of extreme poverty in a sustainable way. There are just so few things that sit at that intersection of a good bet with asymmetric upside, and that's why I feel very lucky to be able to do the work that I do.

SPENCER: It's a really compelling and special kind of intervention for those reasons.

NICK: Yeah.

SPENCER: Nick, thanks so much for coming on the Clearer Thinking Podcast.

NICK: Thanks for having me, Spencer.