SPENCER: Doug, welcome.

DOUG: Hello, Spencer. How are you doing?

SPENCER: There's a famous quote that's often misattributed to Einstein which is, "Not everything that can be counted counts, and not everything that counts can be counted." I think the person who actually said it was William Bruce Cameron. I suspect that you disagree with this quote — that you think that pretty much everything that matters can be measured. Why is that?

DOUG: Well, he said counted, not measured. I could see a difference there. Accounting implies that you know exactly, and that's not actually how the word 'measurements' has been used for about 100 years in science. It's always meant, this entire time, a quantitatively expressed reduction in uncertainty based on observation. So if you just learn more than you knew before, you could have less uncertainty measured probabilistically — that was a measurement. And I would say that anything that counts has observable consequences. And once you figure out how it's observable, you're halfway to measuring it. The rest is trivial math.

SPENCER: So the idea is that, not that you can necessarily measure things directly, but that it will have observable consequences that you can measure. And in that sense, everything that matters is measurable?

DOUG: Yeah, sure. Think about measuring the mass of an electron or the gravitational constant. Anything like that in physics are all indirect measurements. If it weren't for indirect measurements, our scientific methods would be handicapped by quite a lot more. Obviously, lots of measurements are indirect. We can figure things out, we make some observations, do a little bit of math, and then we can make other conclusions from those observations.

SPENCER: Now, I'm trying to get in the mind of my audience. What I imagine they might immediately think is something like, "Well, take something so squishy like love. Could you really measure love?"

DOUG: Well, has anyone ever seen it? I think a lot of people would say, "Yes." And then, I might follow up with, "Have you seen two people that were more in love than some other two people?" They would say, "Well, yeah, of course." And then I would say, "So, what did you see when you saw more love? Describe what you saw." And then they would end up describing some observable consequences. As soon as you're figuring out how you observe it, then really, all you need are some procedures for collection of data in the math world, that's it. It's also very useful to figure out why you need to know it. For example, if you're interested in figuring out whether or not a married couple will stay together because you're a community bank and you give out loans to family-owned businesses, maybe you're interested in how long they would stay together. Well, actually, there is some reason to believe that some near-term observations over a short period of time actually correlate with whether or not they would stay married for a longer period of time. If that's what you're actually interested in, then you've got some near-term observable things that are relevant to something you're trying to forecast. But a lot of reasons that things seem immeasurable, like love, is people haven't really quite figured out what they meant by it in the first place. As soon as they figure out what they see when they see more of it, they're well on the way to the path of measuring something. And the other thing is, if it has consequences at all, if it matters in the least, it has to be observable somehow. And not even just indirectly either it can be directly observable.

SPENCER: Maybe part of what's confusing people is this distinction between measuring the thing itself versus measuring things that are correlated with it. If you measure things that are correlated with it, you are measuring the thing itself in an indirect sense. But maybe, to people, that feels like, "Oh, you're not measuring love. You're just measuring the correlates of love."

DOUG: Yeah, but that's not so different from so many other measurements. You step on a bathroom scale, are you directly observing your weight? No. If you're looking at an old mechanical scale, what you're really observing is how much a particular spring was deformed and then, how that spring deformation changed the needle on the old school kind of scale. And if you're looking at a digital scale, you're just looking at another indirect measure of your weight. You're seeing how much your weight changes resistance on some piezoelectric crystals of some sort, for example. That would be a way that indirectly measures your weight. In fact, all of human experience is indirect signals of the things that we're actually experiencing. Everything you hear, see, or touch. You're just recreating those images in your head. That's all you're doing. Everything is indirect.

SPENCER: That's an interesting point about the scales. But I guess, people might say, "Well, at least with the scale, we have a theory of how to map the deformation of the spring into the quantity we're trying to measure, which is your weight. Whereas something like love, even if we have a bunch of correlates of love, we still don't quite know how to define the thing that we're measuring or how to get a good uniform measurement around it."

DOUG: Yeah, but that's also not too different from stuff like gravity. We know well enough to be able to predict outcomes very well for shooting a satellite into orbit or something like this. But have we really figured out a unified understanding of what gravity is? No, not really. What we do care about are the things that we see and how they relate to the things we expect. That's all human experience is and ever was and probably ever will be. It doesn't even matter what it's about.

SPENCER: Why do you think people are so resistant to trying to measure certain things?

DOUG: I think there's probably multiple explanations. One is that they've just heard over and over again that certain things are immeasurable. I think, to a certain degree, measuring some things sounds threatening, in some sense, because they think it dehumanizes it or something if you measure it. I think measurement and modeling our reality mathematically is as uniquely human as music and language. Language may not be that uniquely human. There's lots of reasons; we've heard other animals communicate in various ways, possibly in complex ways. People aren't generally threatened by, let's say, if I wrote a poem about someone, they don't feel like they're reduced to a poem. But if I describe some attributes of a person quantitatively, they feel like we're reducing a person to numbers. I think it's just a cultural thing. I think that actually varies quite a lot culturally. When you look at books, like Innumeracy by Paulos, that's a cultural thing. That's really more American and somewhat European than it is Indian, for example, or Chinese, to believe that we dehumanize things when we measure stuff. So partly, it's cultural. And I think, to some degree, it's threatening because people don't feel like they understand that stuff really well. So when you're not speaking their language, they feel better by simply declaring it as irrelevant. If somebody's doing something that you don't understand, and you might feel a little intimidated by that, a defense mechanism might be that you feel it's irrelevant. You see what I mean? I think that's part of it. We did a survey for the book How to Measure Anything in Cybersecurity Risk, and we found this direct relationship between people's quantitative aptitudes in statistics and probability — because we asked them a few stats and literacy questions — and their preferences and attitudes towards quantitative methods. No surprise, people who scored worse were more resistant to quantitative methods than people who scored better. There's part of the explanation right there.

SPENCER: I think those make sense as partial explanations. It seems to me there's an additional explanation, which is that people don't necessarily see the value in measuring things. So, why bother measuring things that maybe are not usually measured?

DOUG: Yeah. Well, that's a legitimate criticism. If you feel it can't possibly inform any behavior at all, then, there's no value in the measurement. On the other hand, if a measurement is helping you make a decision under uncertainty that has consequences, then yes, you want to reduce your uncertainty; that improves your chances of making a better bet. If you're going to invest in some big new R&D project, or even just a software project internally, or you're going to hire a new person, you're making a bet, a decision under uncertainty. And if you could make certain measurements, particular measurements, you could reduce that uncertainty, and you're increasing the odds that you're making the right choice. I suppose it's only relevant for people who either don't have uncertainty or none of their decisions are consequential. But if they have any uncertainty at all about any consequential decisions, then there is some value to measurement.

SPENCER: What are some examples where you think people probably should be measuring things based on their own goals, but they typically don't?

DOUG: Well, first off, I heard this just earlier this week: "Nobody ever makes decisions on net present value." And I said, "Well, I'm not sure how you could say that without some planet-wide census of some sort. But I have a few counterexamples, myself, that I know that don't literally mean everybody." I think people even just measure their own performance based on their own judgments. That's something that was clearly wrong. I knew from personal experience that there are people who make decisions quantitatively. And you should doubt your own decision making. That's where a lot of this comes from. Measuring your own skill in decision making is probably one of the most important measurements you can make in your life. It's kind of a meta measurement. It's a measurement of everything else you're doing. Your most important decision is how you make decisions because that's going to tell you what you should measure, among other things. That should tell you, "What are the conditions under which I'm a better decision maker? Are there methods that help me make better decisions?" And that includes things like buying a house or a car, or accepting a job. The reason I brought up the net present value example is because the person said, "As an alternative decision making method, people rely on their guts." Well, people's guts have been measured quite a lot. And it's really not nearly as good as people think. People are consistently better if they rely, at least in part, if not entirely, on some quantitative assistance to their decision making.

SPENCER: What does that quantitative assistance look like?

DOUG: Well, for example, you want to add on your house. You could just say, "Well, my gut tells me I should add on to my house. I'm gonna add two bedrooms here and do this yard work and expand the deck in the back and all this other sort of stuff." Well, what you probably do, though, is you try to figure out how much it would cost. And you might even figure out, "Well, maybe I'm going to sell this place in a couple years, am I really going to get enough value out of this? Is it going to increase the value of the home because if it isn't going to increase the value of the home, maybe I can do without it?" Maybe part of the justification for the decision is that you believe that would increase the resale value of the home. And if it didn't, then you could do without it. Those are kind of assumptions that you should test, especially if these are large decisions for you. If this is a consequential decision, where you're making a big bet, you've saved up a bunch of money and you're going to do these things — well, there are other things you could have done with that money — but you're kind of winging it. People start businesses, people leave their jobs, with some safety to start up a business most of which will fail in two or three years. Most of those small businesses that people leave their jobs to start will fail. Now, a little bit of analysis would actually reduce that failure rate. It's not going to guarantee anything, but some of them are completely avoidable. I think that's an amazing statistic: how often small businesses fail. We've had a couple of opportunities over the years to do an analysis on small business failure rates for clients that have to evaluate small businesses for various reasons, like lines of credit, for example. The statistics in the studies on failure rates of small businesses are amazing. People save up their money in retirement to invest their entire retirement in a restaurant, because that's what they always wanted to do. And then it fails. They're out of everything they've saved, went into it, and it failed. And now they're in debt. That happens. How does that happen as often as it does, because it's not infrequent? Those are important decisions. If you're going to start a business at all, you should at least take the time to do a little bit of math, a little bit of forecasting, gather some data: "What's the market for a new restaurant? Do we know anything about the success rates of restaurants? Can I do a little bit of analysis on my retirement portfolio and figure out whether I could risk this amount? What if it fails?" Those are all things that you should sit down and do the math on. And I'm not talking about calculus or anything like this, or even statistics or probability theory. I'm just talking about just doing the arithmetic on these things. Absolutely, you should try to measure that kind of stuff, and that's just personal decisions. Think about the people who are tasked with making decisions for public policy and large corporate policy decisions where major investments are made. People are deciding to dive into big new infrastructure investments, or big new software for a corporation or merging with another corporation. Some of these have very high failure rates, remarkable failure rates. And they're made on gut. Some of them will try to do a cursory quantitative analysis, but that's more of a formality in some cases. They should really try to doubt their own intuition. We should all doubt our intuition. We have far too much confidence in our gut feelings. We should doubt it. We should test it. We should continuously improve it. And how do you do any of that without actually measuring it?

SPENCER: Are there certain types of important decisions that you think people should not try to quantify? For example, let's say, someone's trying to decide, "Well, should I stay with my current romantic partner or break up with them and look for a new partner?"

DOUG: Well, I guess it depends on your objectives. If the primary objective is just your happiness, that's something where you've got direct personal experience. It's not really something you have to externally observe. You have direct personal experience with that. On the other hand, if part of your assumption about your romantic partner and staying with them is that you think they're going to be a good partner for raising kids and they're financially responsible and things like this, then maybe you want to do a little bit of math there, because you're making forecasts. If you're worried about someone's viability as a long-term partner, if that's your objective, then yeah, you should look into that. You should think about that.

SPENCER: What would it look like doing math in that scenario?

DOUG: It's interesting. Even understanding somebody's credit score is not a bad idea. When you marry somebody, you're mixing your assets. Do they have the same attitudes towards saving and financial responsibility as you do? Because if they don't, that could lead to a big problem. Now, fortunately, those are the sorts of things that people probably figure out while they're dating, and then they make those decisions later on. But when you look at divorces and think about — not that I've done a lot of research in this particular space — how many of them are really based on objectively observable things that were available before they got married. There was an economist, Andrew Oswald, that I cited some research on in the first book. He talked about measuring the value of a happy marriage. He surveyed, I think it was a few thousand people or something like that, and he asked them if they're married, if they're happily married, and so forth. And he just asked them, in general, how happy they are. And he also asked them how much money they made. And he found out that, in order for a single person to be as happy as someone who's married, they had to make almost an extra hundred thousand a year to self-report being as happy as a married person. So a single person could be as happy — they would self-report as being happy — as a married person if they made, on average, about an extra hundred thousand a year. There's a whole page on that one in the book. We can talk about these things. I don't know why people feel insulted about it. But you can write about love, you can observe love, but as soon as you start thinking about love in another very specific human language — mathematics — then sometimes it feels like it's being dehumanized. But not if it's music or poetry; just that particular human language.

SPENCER: We talked about two things you could quantify with a partner. You could quantify their credit score. You could quantify things around your finances with them. Are there other things that you think people could quantify when they're considering whether to stay with a partner or is it really pretty limited in that case?

DOUG: Well. No one's actually hired us yet for this particular measurement, so my background research on this might be limited, other than a couple of other areas, we've done some related research regarding small business propensity to repay loans, and so forth. But, I know that some of the dating sites actually have better algorithms than some of the other dating sites. Think about how most people, for most of human history, actually met. It was actually pretty random. So, I think there's a better chance of actually doing a little bit of math on these things. I'll have to look into the research on that. Like I said, I've no one paid me to build models on that particular but I know there's some research on it.

SPENCER: The reason I asked about that in particular is because it's one of the things that people feel so unquantifiable. So it is kind of an intuition pump for how you might quantify things.

DOUG: I guess it still comes down to, "Well, wait a second. Is it observable? Have you ever seen anybody in love? And have you seen somebody who's more in love than someone else you've seen? Well, what is it that you're seeing?" The beginning of any real rational inquiry in science is just figuring out what we're talking about — just defining your variables. That's all we're doing, really. The beginning of human thought is to figure out what we're talking about.

SPENCER: What's the connection between measurement and assigning probabilities?

DOUG: Measurement is, as I'm talking about here, I think is the most practical use of the term; it is consistent with scientific use of the term and with practical decision making use of the term. Probabilities are about quantifying our uncertainty. So if I say that something is 99% likely to occur, I'm more certain about it than if I said it's 75% likely to occur. Now, I've defined measurement — as I said, consistent with scientific method and practical decision making — as a quantitatively expressed reduction in uncertainty based on observation. So we're changing our state of uncertainty by making some observations. We do a little bit of math and we've updated our uncertainty. That's the connection between probabilities and uncertainties. Probabilities are just the way that we quantify our uncertainty.

SPENCER: So would you advise people to always think in terms of probabilities about their decisions?

DOUG: Yeah. Certainly we make plenty of decisions during the day where I don't make explicit probabilities: I decide where we're going to go out to eat dinner, or which movies we should watch a couple of weekends ago or something like this. No, we don't build models on that. But, my wife is a math professor at a local community college, and we built a pretty detailed model when we started our business and bought a new house. We actually worked out the probability, "What's the chance, given our uncertainties about the coming and going of revenue and projects, that at some point in time in the future, we might not be able to make a mortgage payment?" So we worked out that probability and figured out it was acceptable. And after a while, that was no longer an issue. And then I tell the story — that some people find interesting and some people have the opposite negative reaction to — of how I bought my wife's ring. This was the early 90s, the Internet was kind of getting rolling and I was in the early to mid 90s. I downloaded a bunch of data on wholesale diamond prices, and I built a model that did a pretty good job of predicting the wholesale price of a diamond based on carat, cut, clarity and color. Those are the four attributes you look at. And so, we knew what kind of diamond we were looking for, and so forth. And I could plug this new model, and I could figure out what the wholesale price would be. — There's a small diamond district in Chicago. It's not like New York's but there's a kind of a diamond district, I suppose, where there's a bunch of diamond places right next to each other. — I could tell which ones are kind of charging too high of a margin compared to others. I actually also got utility curves from my wife, but the trade-off was on those four attributes. And she was totally game for it. Now, what's interesting is that when I tell the story to my wife or other people that kind of get us, they understand, "Wow, that's really neat that he put that much effort into it. They really align on this sort of stuff. That's romantic." Other people had exactly the opposite. They think it's the opposite of romantic and I just say, "You don't know my wife."

SPENCER: Well, it sounds like your love language and your wife's love language is measurement and statistical modeling.

DOUG: Well, we're just not threatened by it. We just know that, "Hey, if we're going to be a little bit happier and save a little bit money and spend a little bit more time on our honeymoon, by the way, because we're a little bit smarter about buying a ring or something like this, well then that makes perfect sense."

[promo]

SPENCER: You mentioned earlier that quantification often beats our intuition. How much do we know about the extent to which people's intuition performs worse than doing math?

DOUG: Oh, yeah. So there's this one guy called Meehl, who started looking at studies way back in the 1950s and continued through shortly after 2000. I forget exactly when he passed away, but he had well over 150 studies collected, where he compared human experts to statistical models in a huge variety of fields like prognosis of diseases, outcomes of sporting events, the chance of loan repayment by small businesses, criminal recidivism, and all this kind of stuff. He said that he can only find about six out of all of those studies where the humans did just as well or slightly better. And with those outcomes, you could probably explain that result as well just by chance alone. A few of them had to get lucky. If you did those studies again, it could easily have been the case that the humans didn't do as well, out of that many. Those kinds of studies are replicated a few other ways, so there are other groups of researchers that got into this. And then of course, Philip Tetlock wrote Superforecasting and he ran The Good Judgment Project. It was over a 20 year period. He was tracking 284 experts in political affairs, military affairs, economics and technology trends. He had over 82,000 individual forecasts where people put probabilities on things. And he concluded — I think I had this just about quoted just about right — "It is impossible to find any domain in which humans clearly outperformed crude extrapolation algorithms, less still sophisticated statistical ones." So, he and Meehl were perfectly aligned on this. They couldn't find data where the humans were consistently better. It just never happened. There's so much error that we have in our intuition, that if you remove some of that error, you end up making better decisions. We've seen this in other ways, by the way. There was this guy back in the 1950s, Egon Brunswick, who developed something that eventually became called the Lens Method. That's where you build a model of the human expert making a judgment of some sort. It's based on just that human's judgment. So, you have a human that's, let's say, estimating the duration of a bunch of different project tasks, or you have a human that's estimating how well different applicants for grad school will do in grad school, and what their GPA will be. You're given all this data about all these different scenarios — a big long list of them — and you build a regression model based only on that expert's subjective judgment. All you've done is, you've tracked what the expert would have said subjectively, and then you try to build a model that predicts what the expert would say. So it's not even based on historical data. All it is, is it's trying to predict what the expert would say.

SPENCER: The dependent variable, the thing you're predicting, is what the experts would say. And the kind of variables you're using to predict are just the different factors the expert might be looking at?

DOUG: That's right, exactly. What's interesting, though, is that when Egon Brunswick first did this, he said, "Well, I want to make sure that my models are good because I would like them to be, maybe not as good as a human, but maybe close to as good as a human, like replicating their judgment." He would run these experiments where he would try the human against the model of the human in a real world experiment, observing outcomes and measuring the error between the forecast and the actual outcome. It turned out that not only were the models as good as humans, they were better than the humans. That's a weird thing

SPENCER: It's very perverse, right?

DOUG: Yeah. Why is the model of the human better than the human? And it turns out it's because humans are so inconsistent and applying whatever experience they've learned, that simply removing the error of inconsistency makes their judgments better. And that's a huge one by itself. There's a bunch of other biases and errors in human judgment and decision making, but inconsistency is a huge one. It turns out that if you track expert judgments over a period of time in almost any field, and look at how much their judgments vary in different situations, you can explain about 20% of the variation in their judgments. It's just personal inconsistency. They just would have come up with a different answer next time.

SPENCER: And presumably that's based on sort of hard to track factors like what mood they're in or how much they slept last night, or all kinds of things like that?

DOUG: Oh sure. And some of those actually have been measured in certain contexts. We know, for example, that a person's risk tolerance is affected by a bunch of irrelevant, random external factors like, whether they were exposed to smiling faces (that's a weird one), or whether they were asked to recall and describe an event in their lives when they were angry versus an event in their lives when they were afraid. Those have opposite effects on people's risk tolerance. And you could observe the risk tolerance that they have when they play games of chance. So, you set up an experiment, we've got a test group and a control group. They're both playing some game of chance on a computer. But maybe, for one group, let's say, as part of the background screen on their computer they have a smiling family, and the control group don't. And they actually behave differently. We would love to think that our decisions are rational, and they're not so willy-nilly. But sometimes, you make the decisions that you make because you're more of a risk averse on Tuesday morning than you were Thursday afternoon, and that was it.

SPENCER: It seems to me that there may be an additional reason why the models would be more accurate, even though they're just predicting what the expert would do. And it has to do with some kind of calibration — that the models are sort of guaranteed to be calibrated in a sense. Do you think that's right?

DOUG: Well, it captures all of the other biases that the human has. So, if the human systematically ignores one important variable, or has a recency bias or something like that, or if you just looked at the last variable before you make the decision, if that gets a higher weight or a lower weight, the algorithm is going to pick that up; that's going to have the same errors. And maybe the human might be consistently optimistic or pessimistic in a judgment; the algorithm will have that same problem. However, the algorithm will at least be perfectly consistent — that's really the only difference. In fact, if I wanted to make the algorithm more like a human, I would have to add a random number term just so it gave a different answer every time. Then, it would be more like a person.

SPENCER: Right. Now I'm seeing why this wouldn't be calibrated. Because if it's just predicting what the human would say, if humans aren't calibrated, it would be uncalibrated. You'd have to do an additional step to calibrate it, which will probably make it perform even better, to make sure that when it says a 90% chance, it's right 90% of the time. When it says 50% chances, it is right 50% of the time, which you could do to improve the algorithm further, right?

DOUG: That's right. In fact, that's what you usually do. So just for your listeners, when we talk about calibration in this context, we're talking about calibrated probability assessment. So we're calibrating the human expert, just like you would calibrate any other scientific measurement instrument. You give them a series of problems where you already know the answer, and you compare their responses to the answer on a large number of trials. As you said, we look at all the times that somebody said they were 90% confident, were they right over a large number of trials about 90% of the time? Are they right about 70% of the time when they say they're 70% confident? And so on. That's what being calibrated is. Calibration training addresses that particular issue, where people generally start out overconfident. They put too high a probability on being right compared to their track record, so, calibration affects that. The lens model, the method I was talking about, reduces a different kind of error about inconsistency.

SPENCER: What do we know about how well calibration training works and what kind of calibration training do you think is most effective?

DOUG: Well, we have over 2000 people that we've calibrated over the past 26 years by now. I think it's getting close to 3000 by now, actually; I haven't added them up lately. We've had quite a lot of people go through this. What we have observed is that about 80% of people or so are statistically indistinguishable from a bookie at putting odds on things by the end of the training. It doesn't work for everybody, but almost everybody improves. And about 80% of people improve enough, that for the sample size that we see, they're calibrated. In other words, for the size of the sample that we're seeing, we can't tell the difference between them and a perfectly calibrated person.

SPENCER: And how do you teach them that?

DOUG: There's a few different methods. One is practice. One is you give them a bunch of prompts, they make estimates on them, then you show them their answers, and they find out they're wrong a lot more than they thought. They said they were 90% confident in something in several items, and then they realized, "Oh, I only got half of those right. I was not 90% confident." And sometimes they'll say they're 100% confident and they get it wrong. You don't need a lot of samples for that one. If you've ever said you're 100% confident and you turn out to be wrong, you are overconfident. There's no statistically allowable error for the sample size there; you have to be right every time you say you're 100% confident. So if you do those iteratively, if you have a series of exercises where you assign probabilities, and then you see results, get feedback, and then do it again, that is the most important method. When you think about it, how often do they even state a probability that they track and then get feedback on? They don't do that. When you do these series of exercises, people are getting more feedback on the accuracy of their own predictions than maybe they've gotten their entire careers. When did they ever sit down and assign probabilities to things that they're forecasting, and then check them against observed outcomes? You could go 30 years and never see that. And they're getting all this feedback, one right after the other in these exercises that each take 20 or 30 minutes. And we do them back to back, and then we teach them a few other techniques. We show them how they can set up a problem where they treat each calibration exercise as a kind of a bet. There's some interesting research on this, but if you pretend to bet money, you're better at assessing probabilities. In fact, actually betting money is not much better than pretending to bet money. But apparently, pretending to bet money, just tying personal consequences to it somehow, makes everybody more realistic.

SPENCER: Is it because it kind of puts people in a different mindset where they're really thinking about the likelihood of the thing rather than kind of thinking in vague terms?

DOUG: That's exactly it, yes. They think in terms of personal consequences. If somebody says that they're 80% confident that this statement is true, what then would they rather have: win a thousand dollars if that statement is in fact true, or just spin a dial that gives them an 80% chance of winning thousand dollars and 20% chance of winning nothing. They can choose A or B. Now, if they choose to spin the dial, that means they're not really 80% confident in their claim. They should be indifferent between those two. If they believe they prefer spinning the dial, then apparently, they believe there's actually less than 80% chance of that plane being true — what they thought was 80% chance of being true. That's called the equivalent bet method, and there's a bunch of ways to do that. There's a few other techniques as well, and we teach all of these techniques one right after the other in the series of exercises. So we give an exercise, teach them a couple of techniques, give another exercise, teach a couple more, then by the last couple of exercises, they're basically just practicing and fine tuning.

SPENCER: Beyond the equivalent bet method, what are some other techniques that you teach to help people be more calibrated?

DOUG: One is called Klein's PreMortem. It's an interesting one. The decision psychologists, Gary Klein, came up with this one. He would talk to project managers, and he'd say, "What could possibly go wrong?" And they couldn't think of anything. They figured they got it all figured out. But he just reframed the question. Instead of asking them what could go wrong with their project, he would say, "It's five years from now and the project was a disaster. Explain what went wrong." Apparently, what that does is it frames it as a backward looking narrative for people. People assumed to be willing to explain things that already happened, and not forecast things. You know what I mean? So when he asked, "Assume that the project failed, now tell me what went wrong," then they were a little bit more forthcoming about potential risk. So what we teach people is, we say, "Now, once you've answered this question, assume your answer is wrong. Just assume it's wrong and explain why." Often they'll think of one or two reasons why that might be the case. Now, does that have bearing on your answer? Should you go back and change it? Sometimes it does, so they go back and change it. That's another one. Nice.

SPENCER: Any other techniques that you think would be helpful to share?

DOUG: Oh, yeah. When people are estimating intervals, confidence intervals for example, where you have to put a range on something, like "What year was Napoleon born?" They put an upper bound and lower bound. You practice with trivia questions like this, so that you can answer real world questions like: How long is this project going to take? What's the probability of a cybersecurity attack? What's the frequency of this event? What's our revenue going to be next quarter? All of those things are things that they might estimate on those...not the discrete probabilities, but the ones that are continuous values. They want to express their uncertainty with an interval. If you have more uncertainty, you have a wider and wider interval. You don't know what you're gonna make in the first quarter of this new product, so you say, "Well, I have a wide range; it could be $10 million or $200 million — pretty wide range. And if you had less uncertainty, it'd be a narrower range. And if it was a lot more uncertainty, it could be an even wider range. Well, what we find is that when you apply the equivalent bet, often what people will do is they almost always widen the range after applying the equivalent bet. In other words, they compare their interval to a spin of a dial where there's a 90% chance of winning the same amount of money as what they would win at the actual answers between the upper and lower bound. And they would often prefer the dial, and then realize that they need to widen their range a lot before they're indifferent between betting on the range and betting on the dial. But what we tell people is, instead of starting narrow and widening it, start wide and then use the equivalent bet to start chipping off the tails. So starting wide and coming in seems to — for some reason we haven't quite figured out — work better than starting narrow and trying to widen it. I think it's maybe an anchoring psychology problem or something like that. All we know is that people do seem to get calibrated better with it.

SPENCER: So the idea would be, if you want to estimate when Napoleon was born, you start with a range where you're pretty much certain it's in that. You're like, "Well, I know it's after 1000 and before 2000." And then, you kind of start working in from the edges?

DOUG: Yeah, exactly. As opposed to somebody who comes up with, "Well, I think it was sometime in the 1700s," and then they start sort of inching out in upper and lower bound a little bit at a time. And that's not even close to representing their uncertainty. So, they should start with the 1000 to 2000 and say, "Well, he was around after the Battle of Agincourt," or "He was around after Columbus ran into the Caribbean islands," or something like this and, "Hey, it wasn't the 1950s; it was before television." You start slicing off the ends. And then, you start getting into points where you're saying, "Well, was it before the Civil War? Maybe I'm pretty sure he's born before the Civil War. Yeah, he was born before the Civil War. Was he born before the American War of Independence? Yeah, I think so. Because he came up after, galavanting around Europe in the 1800s before the Civil War." Then, they start getting more uncertain about that, and then they start realizing where they're uncertain about stuff — that's where it matters.

SPENCER: Do you find that it's helpful for people to kind of play devil's advocate for both sides like, "Oh, well, let me make the best argument that it should be expanded. Let me make the best argument I can that I should contract my interval?"

DOUG: Well, with one caveat. It turns out that making an argument for your interval doesn't really help things much. It turns out that the only thing that really matters is being a skeptic. Don't worry about the pro argument, you just assume you started out pro already. What you need is skepticism.

SPENCER: That makes sense. I guess our biases are already towards what our first answer is. Yeah, that makes sense. So for people interested in this topic, I just want to mention two tools. One is the Calibrate Your Judgment tool, which we made at Clearer Thinking. You can get it for free on our website. You can actually practice making probabilistic predictions and confidence interval type predictions to practice your skills. The other is a tool called Fatebook (fatebook.io) that lets you, for your own life, log predictions on what you think is gonna happen. I've actually been using Fatebook and its predecessor, which is called Prediction Book, for a long time, in tracking many predictions from my own life, and I found it really valuable.

DOUG: Since you're mentioning that, we actually also have our own entirely online, pre-recorded asynchronous calibration training with multiple exercise banks. That's a big part of our business, as people sign up for this. It's all online, it interactively produces calibration curves after each test, you see it all online, and we're adding new features to it: after the training, you can continue to use a what we call 'team calibrator.' It blends right into a software as a service platform that we have. And the software as a service platform has algorithms in it for combining the estimates of multiple people. So when you ask yourself, often you're getting estimates from multiple subject matter experts, "How do you combine them?" It's been researched, and some methods for aggregating multiple estimates measurably outperform others. The answer is not obvious. And the best way to look at this is when you have a lot of estimation data, which we do. You even have to figure out not just their past performance when you aggregate them, by the way, you have to look at how correlated they are to each other. If two individuals are perfectly correlated, then obviously you don't add any information by asking the second person. What you want are more or less independent people who are knowledgeable and calibrated. And then you can combine them with this odds ratio method that we're using. That actually ends up being (this is kind of interesting). But if I had three people who all said that some statement has a 70% chance of being true — like we're going to finish on time, or we're going to win this contract, or something like this — if three people independently, without communicating, who are uncorrelated to each other, all said that there's a 70% chance of that claim being true, and this is one claim out of a set of claims where about half are true (let's say they need to state that as a prior), the chance that the claim is actually true is just the average of three 70%s. So it's not just 70%. The chance that it's true is a little bit more than 80%. It's about 81%.

SPENCER: Is that because the odds get multiplied together to do the calculation?

DOUG: Yeah, that's kind of what happens when you do odds ratios. But you also have to adjust for how correlated people are, as I mentioned, and their prior performance. That actually all goes into the algorithm. Our algorithm for this, we call FrankenSME. FrankenSME measurably is better calibrated than your best individual expert.

SPENCER: It's really interesting because it quantifies this idea of the importance of diverse thinking — why you don't get much additional benefit by having people with similar thoughts as each other. But if you look at problems in really different ways, you actually get this incremental benefit by combining their predictions.

DOUG: Yeah. It's sort of, intuitively, we understand that if two very knowledgeable people disagree a lot, when they both agree on something, it feels like it should be more likely to be true, right? In those cases where they both agree. Well, the math agrees with that. The masses, yeah, that is the case.

SPENCER: Where can people find your online calibration training?

DOUG: They can just go to hubbardresearch.com and go to the AIE Academy (Applied Information Economics Academy). You can see all the courses right there. One of them is the calibration course. They can sign up for that and take that at their leisure. It's not just an online training course with videos and so forth. It has those, but you're actually taking the test on this platform so that actually generates the calibration curves, and you get that feedback. And on top of that, we're just about to add a new feature, where if you get calibrated, you can actually get, for a period of time, access to Team Calibrator. We also sell enterprise versions of this where you can have groups of people all get calibrated and they can all collaborate on Team Calibrator. It aggregates their estimates, managers can create and assign estimation tasks and track who's answered, etc.

SPENCER: Some people might not be convinced listening to this that there's much value in being calibrated with their probabilistic predictions. What's the best case that this is something that people should care about?

DOUG: Every time we take people through the benchmark test — and sometimes people are assigned to the training, they didn't necessarily volunteer for it. Their managers told them they had to take it. So they come in, take the first test — I can tell you, across the board, that everybody is terrible at first. They think their intervals have a 90% chance to contain the correct answer. And the correct answer is within their intervals less than half the time — they're stunned. They didn't realize they could be this bad. Now, if somebody's never gone through calibration training, they might go through their entire career just deluded about how good they are at estimating things. Think about it. If you rely just on your own memory, your own recollection for evaluating your performance as a forecaster, what do you recall? You tend to recall when you're right. So, we all remember being above average forecasters. Well, when people go through calibration training, they're in for a surprise. They're not as good as they thought they were, but they can get better. That's the important thing. And by the way, I've done surveys where people build probabilistic models on a number of different IT projects or estimating the sales of new products or anything like this, or even in sports, stuff like this. We tend to find the people who build these Monte Carlo Simulations almost never calibrate anybody. Most of the models had at least some subjective estimates in them, but none were calibrated. The things they had the subjective estimates tended to be some of the most consequential and uncertain variables. That means that their risk models systematically understate uncertainty and, therefore, systematically understate risk across the board. And they're making all these big bets, not just on their personal lives but also on big investments for their companies or big policies for the government agencies they work for, where they affect a lot of people. I really hope they do get calibrated.

SPENCER: You can certainly see the value that those forecasts are accurate and calibrated when people already have to make forecasts about really important projects. But a lot of people don't have to do that kind of thing for their work, or at least they don't in such an explicit way. So what would you say about the value of becoming probabilistically calibrated for someone that's not an explicit part of their job? Yeah,

DOUG: You might have heard the phrases in the past 80s like, "You don't dress for the job you have, you dress for the job you want." Likewise, for skills in general, when people go out and get new skills, they're often not just for the jobs they have, they're for the jobs they want. If they're happy at whatever their current job is, that requires literally no decision making under uncertainty at all, I guess you don't need this. But if they imagine that either now or eventually, they'll be in a position of being trusted for making big decisions, and if it's a decision that's challenging at all, it's got to have uncertainty in it. And if it's most big, uncertain decisions, it's going to rely on some subjective estimates. You should at least be aware of the problem that people are systematically overconfident. And if you're one of the people providing estimates that go into this big consequential decision, you should get calibrated, not just for the sake of yourself, but for all the stakeholders you affect when you make these decisions.

[promo]

SPENCER: One topic that we haven't discussed yet that I think is potentially critical here is value of information. What is value of information? Why should people care about it?

DOUG: I think people understand that it makes sense to spend more time measuring some things than others. I think they understand that intuitively. You don't really measure everything. Some people misquote the title of my first book and say, "How to measure everything." I said, "No, that's not how to measure anything." I actually make the case that you should not be measuring everything because there isn't an economic value to measurement. And when the cost of a measurement exceeds the value of it, you don't do it. That's the main reason not to measure something — it's just the economic constraints. Though, when you actually start computing the value of information, imagine that you have decisions that have (usually) a large number of variables. That's typical. A decision has a large number of variables, whether it's a big infrastructure project or a big software project, launching a new product, just changing a government policy or a corporate policy of some sort, or a merger or acquisition of any kind. If you put together a business case for those things, like a cash flow, you're probably going to have a lot of variables that you are uncertain about. And let's suppose that you might have empirical data on some of those, for which you can use to make inferences about. You can put probabilistic intervals on things. And a lot of the other things you might have to put calibrated estimates on, and those might be very wide. When you compute the value of information, what you're really looking at is, "What's the chance that I'm going to be wrong about this variable? And what's the cost of being wrong?" That's kind of what the value of information is: the chance of being wrong times the cost of being wrong in its simplest form. Now, if you compute the value of information for every uncertain variable in the model, that's basically telling you where you should assign your resources to make a better bet for this bigger decision. When you start computing the value of information for every variable in the model, you get some surprising results. It's like a sensitivity analysis, except it's monetized. And so, often a sensitivity analysis will more or less agree with an information value calculation, though not always. However, the sensitivity analysis doesn't tell you that it's worth $2.5M to measure one variable, $50,000 to measure another, $500 to measure yet another, and that the rest are valued at zero. The sensitivity analysis is not telling you that. Information value calculations — which have been part of game theory and decision theory for many decades now — does tell you the monetary value of those things. When you start computing that, people will often find out that they've been measuring the wrong stuff all along. Like for example, IT projects. What do you think people spend more time measuring, costs or benefits?

SPENCER: I assume costs.

DOUG: Yeah. And which one do you think is more uncertain?

SPENCER: Benefits for sure.

DOUG: Well, that's a measurement inversion. They should really be spending more time measuring benefits than measuring costs, but they're actually spending more time measuring cost. And you can break it down further. You can make it much more granular. They tend to spend more time measuring initial development costs. However, they're more uncertain about long term maintenance and training; and the latter is bigger. Also, they might be measuring more, let's say, automation's specific time consuming activities, but they're actually more uncertain about things like customer retention, or things like that. In fact, the most important things to measure in software are often not even on a business case. Have you ever seen a software project get canceled?

SPENCER: Yeah, bunch of times.

DOUG: Have you ever seen a chance of cancellation on a cash flow or business case for software?

SPENCER: No, I think it's just assumed that they'd never get canceled.

DOUG: That's a major omission of risk. We always include that, by the way. That's routine. We've got lots of data on this sort of stuff, so we always include it in our models. That ends up being, on average (not always), one of the most important variables to measure. And not only is it not measured, it's not even on the business case. The next big one is adoption rate. So, did they include in the business case uncertainty about how quickly the new technology would be adopted by users? They often (sometimes) just assume that, "Well, when we're done, everyone will just start using it, and we get all these benefits." Sometimes they'll consider sort of a ramp up period, where it's kind of rolled out across the organization, but they're not really considering, "Well, people are not going to be as proficient at it at first; it'll get better over time. Some people will resist using it." And sometimes, this is something that you're offering clients. You can't really just mandate their use of it, right? So, it's really all about whether they find value in it. And even if they do include things like technology adoption rate, they're very overconfident about it. They don't allow for enough uncertainty. And like cancellation, adoption affects all the benefits. So if you're uncertain about it, it affects all the benefits. You could have 12 different kinds of benefits, and cancellation and adoption affect all of them. So, those end up being relatively high value measurements. And they're almost never measured. That's the remarkable thing. So that's called the measurement inversion — the general tendency to measure almost exactly the wrong things.

SPENCER: Why would people have a tendency to measure the wrong things? Is it partly that they're resistant to measuring the things that feel more uncertain or feel harder to measure?

DOUG: Yeah, that's part of it. I also think they have a habit of just measuring certain things. And the things they've been measuring all this time are, by their nature, things they're less uncertain about, because they've been measuring them for so long. And so, you kind of have to get them out of that rut. Usually people don't really ask the question, what should they measure, and then figure out how to measure that. They just start with, "Here's what we know how to measure." We never start that way. We don't even necessarily assume that we have a way of measuring something at first. We're just confident we'll figure it out. But first, we figure out what we should be measuring. That's what we do first. And then, we say, "Okay, here's a couple of really high value measurements. It's not really obvious how we should measure those things. But you know what, with information values this high, it's worth thinking about for a while. So let's figure out a plan." That's the approach that you should take. Obviously, there's a cost to measurements too, so you have to take that into account. But we see these really high value measurements that don't get measured. And what's interesting, is that the high value measurements — by virtue of the fact that they tend to also be uncertain — are actually easier to measure because they're uncertain. This seems paradoxical to people. But the more uncertainty you have, the easier it is to reduce that uncertainty with a few observations. If you know almost nothing, almost anything will tell you something. That's what the math says.

SPENCER: So if I already have 10,000 data points or something, even if I add another 10,000 data points, it's not gonna reduce my uncertainty very much. But if I have one data point, then going from one to even five data points, that's like a big difference.

DOUG: Exactly. That's a huge difference. And when you do the math on that, you see how big a difference that really is. Once you get up to 10,000 data points, for example, or even just 40 or 50 data points, you have to quadruple the sample size for each additional 50% reduction in error.

SPENCER: It's because the uncertainty falls like one over the square root of the number of samples, right?

DOUG: That's right. So you have to keep quadrupling it for every fifty percent reduction error. But that's not the rule for very small samples. It's an even quicker uncertainty reduction for the smaller samples. That's why there's a couple of different sampling methods. For example, if you're trying to estimate the mean of a population based on a sample, for samples less than 30, it's recommended that you use the Student's t-test statistic for the small samples. And the Student's t-test statistic converges very rapidly when you start moving from two samples to three samples to four samples. It's a huge reduction.

SPENCER: Could you give us an example of applying the concept of value of information to one's personal life? Like maybe an example where you've had to apply it?

DOUG: I suppose the more obvious example was when we did the simulation of buying a new house for our business. Where should I spend more time measuring things? Well, obviously — and this is sort of more obvious in retrospect — but it wasn't necessarily obvious that really everything about the house itself was relatively certain. You can have lots of uncertainties about a house: maintenance costs, and things like this that might catch up with you, things that may or may not be covered by insurance, things like that. So you might have lots of uncertainty about that. Obviously, the main uncertainty was just business. But another uncertainty that we could reduce quite a lot was, "How much could we get by on if we had to?" And then when we thought about it a little bit more, we're able to tighten that range a little bit more. That one made sense, because that was a little bit easier to measure. When we're uncertain about our viability for future business and so forth? Well, when we first bought this house, it was before I wrote my first book. But we did some analysis, and I worked out, "Alright, we've already gotten a few good leads, and we tend to get new leads at this rate." I did a little bit more analysis, I reduced my uncertainty a little bit, I still had some risk. But you know, when you're starting a small business, you're typically going to have risks. And starting a small business and at the same time approximately buying a house that you're going to raise your kids in. So, that was another uncertainty about that. Now, we didn't buy our second house until the business had grown a bit. Then, we were a little more confident in it. But yeah, those are things we did for real. And again, we don't do this daily. Like I said, we don't do an information value calculation on where we're going to go for dinner. That's a low consequence enough decision. Even if we made a bad decision, regretted it, we just don't go back. It's not like it's an unrecoverable cost or something like that. But for anything, basically a car or bigger, we would do a more detailed analysis. And as we started growing our business, even hiring individuals, that was a function of the actual analysis we've done. We're starting to do more forecasting: What's the likelihood of new business based on what we're seeing so far? What's the likelihood of having somebody on the bench for a while or retraining them, and they're not bringing in revenue? Yeah, we were doing the same things we build for our clients. We're using it internally.

SPENCER: When I think about value of information on a personal level, one example immediately comes to mind for me is buying a mattress. A mattress is a purchase that you have for such a long time, maybe for five years or 10 years. And also, if a mattress is a bit better, if you actually sleep a bit better, that could be potentially worth a lot.

DOUG: That's a good point. Yeah, that's interesting. I hadn't thought of that one. Sure, that's true actually. I think one way for people to just think about it is, first off, just to understand the general theory that in its simplest form, it's just the cost of being wrong times the chance of being wrong. So if they're thinking about a problem, and they think, "Okay, there's multiple possibilities for this value, how could I be wrong? What's the way of being wrong?" Like, I'm going to buy a restaurant. Well, I'm imagining a certain amount of patronage every night for the restaurant — people coming in for dinner. And I have this wide range. It's got to be over x in order to make money. And it looks like I probably am over x, but there's a chance I'm below it on average. How much do I lose if I'm below this breakeven point? And what's the chance of being below that breakeven point? Well, you can do a little bit of math on that and kind of work out the probability weighted average loss, given that you're below the break even point and what the chance of being below the breakeven point is. That's the expected opportunity loss in that context. And in that case, that's the expected value of perfect information. So if you could eliminate uncertainty (the expected value of perfect information), you've eliminated any expected opportunity loss. There's no chance of being wrong if you have perfect certainty. So, you can reduce uncertainty about that. So at a very simple level, you can just think, "How wrong do I have to be in order to not break even? What's the chance of that happening? And how much would I lose?" And that's true, whether you accept or reject an investment, like buying a restaurant. If you reject it, then the opportunity losses would have turned out to be a pretty good bet, and you lost money because of that. So opportunity loss works either way.

SPENCER: One trick that you talk about in your book that I thought was quite cool is the rule of five. Do you want to just tell us what that is and how you could apply it?

DOUG: Yeah. This seems a little counterintuitive to people. I constantly ask people this kind of a little trivia question. I say, "Suppose you have any population at all. It could be the population of all the employees in a large company, and you're trying to estimate how much time they spend in commute or how much time they spend each week filling out the TPS report, or something like this? You randomly sample just five out of the entire company. Let's say the company has 20,000 people or 100,000 people. You randomly sample five, meaning that everybody in the whole company had an equal chance of being selected, but you randomly sample five of them. And when you get the sample of five, you find out that the answer for this period of time (the thing that you're trying to estimate), the lowest one was 15 minutes a week. The next one was, say, 45 minutes a week. Okay, that's the range of their activity that they spend per week in that sense."

SPENCER: The mean of the five and the max of the five?

DOUG: Now, what's the chance that the median of the entire population — all 20,000 or 100,000 or a million, whatever it is — what's the chance the median of the entire population falls between the largest and smallest of that sample of five? Well, it's 93.75%. That seems like too high a probability to some people at first. Some people will say, "Well, I don't know 50/50, or maybe 60%, or 70%. But 93.75% sounds too high. Well, here's the way you think of it. What's the chance of randomly selecting somebody who's below the median? Well, it's 50% by definition, right? Because the median is the point where 50% of the population is below and 50% is above. So, the chance of selecting somebody who's less than the median on that particular parameter is 50%. Now, what are the conditions under which I would select five people where the median was not between the smallest and the largest? Either, I had to pick five people who were all below the median, or five people who were all above the median by chance. That's just like flipping a coin five times and getting either all tails or all heads. Now, can I work out the probability of getting at least one tail and at least one head, as opposed to getting all the same? Yes, there's a 93.75% chance of that. It's a neat thing to experiment with. You can take any big database that you have access to: anything in your personal life, all your financial transactions of the last 10 years, or whatever you got handy, gas prices in the neighborhood, or something like this — something that you've got a large database for where you can compute the exact median. So you've got thousands of data points in here, and you know exactly what the median is. Then you run an experiment where you just randomly select five of them over and over again. And then, start working out the proportion of times that the median, that you knew in advance, fell between the upper and lower bound of that random sample of five. And as you keep running that experiment over and over again, you find out that you're converging to 93.75%.

SPENCER: Final question for you, but it'll be a big question. If someone wants to improve their decision making, what are the top three things you would point to?

DOUG: First off, just skepticism, I would think. Being personally skeptical. Why do I think I'm as good as I am? Do I have reason to believe this? What's the basis for my confidence? In other words, do I really get feedback on my performance? A lot of people don't. In fact, you can't really improve without feedback on your performance. And if we don't get explicit documented feedback, we tend to just recall the most flattering times. We even reengineer our recollection. We'll say things like, "I knew it all along." Well, no, they didn't, actually. They honestly might remember it that way now that they actually knew before what they believed they did, which, in fact, they didn't know if you go back in time that way. So being skeptical is probably the first thing. The second one is asking, "How do I know the methods that I'm using work?" Well, there's been some research on this. We should also be aware that we even fall for things like an analysis placebo. If you go through a process that seems structured and formal, your confidence will inevitably go up. But it'll go up even if the method makes you measurably worse than you were before. So people adopt methods and they say, "Yeah, I think this is the right method. We feel more confident about our decisions." Remember, the objective of a methodology is not to feel more confident in your decisions, it's to make better decisions. If you want to just feel more confident making decisions, I guess, you can just indefinitely delude yourself that you're making great decisions all the time. I think if you're really interested in making better decisions, you have to start with skepticism. And then, you have to start with curiosity about what works — look into what things actually work. And I would say the final thing is just documenting and tracking your performance. Try to make decisions on lots of little things and see how often you're right. Tracking it over time: what did you think was going to happen? What actually did happen? Now, it's kind of hard to do that on a lot of small scale things. But for the big consequential decisions that some of your listeners are probably involved in, yeah, they should probably do that.

SPENCER: Doug, thanks so much for coming on. This is a fascinating conversation.

DOUG: All right. Thanks a lot, Spencer.

[outro]

JOSH: A listener asks, "What suggestions do you have for someone who feels far more dependent on the esteem of others than they would like?"

SPENCER: I think to some extent, caring about the esteem of others is an intrinsic value for many people. It's something that they fundamentally care about, and there may not be anything deeper beneath it; it may be a bedrock value they have. However, that being said, I think it's worth inspecting that and wondering, is it really an intrinsic value? Or could it be an instrumental value? Could it be that you actually care about it as a means to an end and not care about it fundamentally? So I think that's worth inspecting and really asking yourself, "Hm, well, what if I got this esteem, but I didn't get any other benefit? If I only got the esteem, nothing else, would I really want it?" And then I think another interesting thing to interrogate there is to ask yourself, "Well, whose esteem do I actually want?" Because I think sometimes people live for this kind of hidden crowd that they imagine evaluating their life. But who are these people? Are you living your life for your eighth grade friend group and trying to impress them still as an adult, or an anonymous audience on social media? And I think that when people inspect this, they will sometimes realize that actually — they kind of imagine this hidden audience that's evaluating their life. But really, that imaginary audience doesn't matter. But the esteem of your loved ones probably matters. The esteem of people you respect probably matters. And so I think it's just worth reflecting on: whose esteem do you really care about? And if you're living in a way to try to impress people whose esteem you don't truly, deeply care about, then maybe you can orient more towards caring about only certain people's esteem that really matters to you.