SPENCER: Helen, welcome to the Clearer Thinking Podcast.

HELEN: Thank you so much for having me.

SPENCER: There are so many things that we do every day, such as flossing our teeth or using a specific exercise routine, and I think most people assume that someone somewhere has proven definitively that these things work. Is it your view that actually a lot of the stuff we do is not as proven as we might think?

HELEN: Definitely, yes. So, I've spent about the last five years talking with all kinds of people about evidence from science, and that has definitely shown that a lot of conventional wisdom or things that we do are just not tested. This was all in pursuit of the book that I've been writing, which is called Beyond Belief. It shows what really works, and I've got one kind of, I guess, the most compelling example of conventional wisdom and how it can backfire in a really tragic way, which I ended up putting at the front of the book because it was kind of so awful. This all relates to this book, actually, which I got out for you because I knew I'd be talking to you, which became a childcare bible for parents starting in the 50s. It's called Baby and Child Care by this pediatrician Benjamin Spock. In this book, which was very popular with parents, he made a small change in the text of the 1958 edition. Previously, he had been telling parents to put their babies to sleep on their backs, and he changed it, suggesting that parents put their babies to sleep on their fronts. He suggested that maybe babies could choke on their own vomit or they might even end up with a flat head if they were sleeping on their backs. Because this book was so popular, many parents followed this advice and started putting their babies to sleep on their fronts. Meanwhile, there was a rather sinister rise in the number of children dying from sudden infant death syndrome, or SIDS. Initially, there was no connection made between these things. In fact, there was a big debate about why this was happening. Scientists had ideas, but nobody really understood why it was, and what was missing from this debate was actually some science. What scientists needed to do were these things called case-control studies, where you take a group of children who tragically died from SIDS and compare them with children from similar backgrounds and risk factors, and then try to pull out what factors might contribute to these SIDS deaths. Eventually, these studies started to be done. One came out in 1968, and one came out in 1970, and even then it was quite hard to see this connection. But eventually, over time, the evidence built up, and it became clear that the advice to put babies to sleep on their fronts was tragically and lethally incorrect, and this had actually contributed to what was eventually called an epidemic of SIDS. It's a really powerful cautionary tale about the damage that unsubstantiated advice can potentially do.

SPENCER: You can imagine how hard it would be to even figure out that was the cause, because you could run all kinds of studies, but if you didn't specifically ask parents, "Which way are you putting your baby?" you might find all kinds of weird correlations. "Oh, well, parents in this neighborhood have slightly higher deaths," and maybe there's something to that, and that could just be because in that neighborhood, they're more likely to read this book.

HELEN: Completely. Yes, absolutely, and that's why it was initially difficult in these early studies to see what the risk factor was. Because you're just looking among all these possible factors that might distinguish the babies that died from the babies that didn't. The key in those kinds of studies is to try and control for those differences. You do a control; that's why it's called a case-control. You try to get children who are from a similar neighborhood and a similar kind of health background, so you're trying to screen out all the noise that could get in the way. Nevertheless, this is another really interesting reason I wanted to write about this particular example, because it also shows this really important concept in science, which I think often gets overlooked, which is about synthesizing evidence. When these studies were coming out, if you look at one study, the signals are very weak; it's hard to see that there's this sort of correlation going on with the sleeping position. But if the studies had been put together and synthesized, then the signal would have become stronger. Now, that wasn't done in those days, because it just wasn't so common to do things like a systematic review, which is a common way of synthesizing studies now, or a meta-analysis. What scientists did is they later went back and did this kind of historically, and they sort of said, "Okay, well, if we had synthesized the evidence, at what point would it have been clear that there was that link?" They showed that actually it would have been apparent after the first two studies in 1970, and they calculated that if that had been done and the evidence had been synthesized, then approximately 50,000 deaths of children could have been prevented if that evidence had been acted on and put together, because actually what happened was that advice to parents didn't change until the early 1990s. So another reason this is a kind of powerful cautionary tale, not just about doing the studies, but also about bringing them all together.

SPENCER: Wow, yeah. Now, do you think that Spock was actually making a mistake here, or was he just extremely unlucky? Was there actually any way he could have possibly predicted that this would be a problem?

HELEN: I think that's fair. And I don't want to vilify him. Other pediatricians were handing out similar advice. Parenting is still absolutely rife with opinions and often conflicting opinions and conventional wisdom. So, I didn't have the good fortune to interview him. I can't speak to his exact logic of making that change. Yeah, so I think we should probably be generous, but nevertheless, it does show that these unsubstantiated opinions can have tragic consequences that are hard to predict.

SPENCER: Absolutely, there are kind of two notions of the idea of a mistake. There's a mistake like things went badly, and there's a mistake like you should have known better. Poker players talk about this sometimes, where they actually made the perfect play that they could make, the best possible play, and they lost. Many things in life we just don't have super high-quality evidence, so the best we can do is sort of try to make a guess based on a small amount of evidence.

HELEN: Yeah. I think that happens all the time. I mean, that's really how a lot of policy is done. Another field I look at in the book is policies. Governments introduce changes, and they sort of hope that they work. But a lot of things done in policy haven't necessarily been tested. But you're right. At some point, you have to introduce it and then collect the evidence as you go along. Another example I looked at in the book is something that sounds really, I suppose, the lesson is we shouldn't assume things work, and we do want to collect the data as we go along. Another example is around this program called Scared Straight, which is often trotted out as an example of policies that can backfire. Scared Straight is this government program that's been used in the US and in other parts of the world, where young people who are at risk of going on to criminal behavior are taken into jails to see what it's like. The idea is that they're scared straight; they see how awful it is, and then they won't go on and commit crime. A bunch of studies had been done, but there was an early systematic review, that early systematic review in social policy, anyway, that brought together some of these studies on Scared Straight, and it showed not only did it not work, but it actually backfired in that the young people who'd been through this program were more likely to go on and commit crime. I think it's just sort of important that we challenge our assumptions, and my book talks a lot about just collecting evidence and thinking about at least the evidence that is available.

SPENCER: On our website, Clearer Thinking, we have a quiz where we show you different charitable interventions, and you have to try to guess which ones were proven effective, which ones turned out not to work, and which ones turned out to be harmful, based on later randomized controlled trials. The point of the exercise is it's incredibly hard to tell. You read about the charity, and you're like, "Oh, that sounds like a good thing, right?" They all sound like a good thing intuitively, but then some just didn't do anything, and some literally backfired. I think that, because in our daily lives, for a lot of things, intuition works pretty well, we often learn to trust it, and then we get into situations that are really outside of the distribution of stuff that we have good feedback on, and we generalize and assume, "Oh yeah, I could probably tell which charity works just by reading about it."

HELEN: But does intuition work well, though? Often it doesn't. That's kind of one of the lessons of evidence, I think, is that sometimes things which sound really intuitive, like the Scared Straight thing, which sort of sounds like it ought to work, it's the same kind of concept around anecdotes. One of the things that evidence fights against is the power of anecdotes, in that if I say to you, well, it worked for me, that's just really powerful. It resonates with us, doesn't it? It feels emotionally compelling in a way that data on thousands of people from a randomized trial doesn't tend to feel so compelling. I think one of the lessons of evidence is to challenge intuitions. That's where people have gone wrong many times, I think, in the past.

SPENCER: It's an interesting question of when intuition works, and my view is it tends to work really well in situations where you have many examples with tight feedback loops. If you're a boxer and you box every day, your intuition about when to dodge a punch becomes pretty good. It's not good at first, but eventually it's pretty good. If you're a chess player and you play chess every day, your intuition about what a good move is gets better and better, and eventually gets really good. In a lot of our daily life, we have this kind of feedback loop where our intuition eventually learns to do a pretty good job, but then something like picking a charity is different. Most of us don't do it very much, and we don't get a good feedback loop. We give the money away, but how do we know if it turned out to be bad? If we do find out, it might not be for 10 years.

HELEN: Absolutely. That happens a lot in conservation. One of the challenges in using evidence in conservation, or testing things properly, is that ecosystems take a long time to change. Let's say you turn something into a nature reserve. Generally, you might think that's a good thing to do, and probably it mostly is, but the downstream consequences of that might not become clear for decades, so evidence can be quite hard to collect.

SPENCER: You mentioned anecdotes and stories, and how powerful they are. It almost seems that the human mind runs on stories. Even if we try to work with data, we kind of turn it into a story in our mind before we can operate with it. It might be something about the fundamental architecture of our minds. Do you think that's right?

HELEN: Yes, I'm a journalist and a storyteller. One thing I wanted to do when I was writing this book was to explore the fascinating backstory around how people came to use evidence in medicine and other fields, which is a great story in itself. Sometimes I think the people who were big champions of using evidence, whether in medicine or policy, haven't necessarily told their own story in a way that illustrates that. I went about it by talking about people. Often, there have been individuals in particular fields who looked around and thought, isn't it interesting that all of our practices are based on conventional wisdom, opinion, or anecdote? Is there a better way to operate? They realize that there are studies you can do, like randomized controlled trials, which test whether a treatment is effective rather than just assuming it is. They become inspired and inspire change in their field. These wonderful human stories illustrate the power of evidence and can sometimes, like the SIDS example, show the difficulties or pitfalls of following anecdotal advice.

SPENCER: When you look across these different cases, is it typical that they get a lot of pushback when people start saying, "Hey, maybe these things we've been doing for a long time, we should run studies and see if they actually work," or is everyone, "Oh yeah, that sounds like a good idea?"

HELEN: Well, what do you think?

SPENCER: Yeah, my guess is that people are really wedded to whatever they've been doing.

HELEN: Absolutely, absolutely. That nails it. That's been the case across all of these fields. I'll tell you about medicine, because I think it's a really interesting example, and most people don't realize that the use of evidence in medicine is actually quite a recent phenomenon. I talk in the book, just to give you a story of a particular person. I wrote the book because I met a researcher called Ian Chalmers, who has been a really seminal figure in helping medicine convert to a field that uses evidence. He was training in London in the 50s and 60s, working in obstetrics and gynecology in a hospital. He realized that the senior doctors he was working with would hand out different advice to women with the same condition. One doctor would say bed rest is really important during pregnancy, or another doctor might say prenatal vitamins are really important, and then another doctor would say no such thing. He was puzzled by this. In fact, he even said to me that when he was woken in the night to help a woman, the first question he would ask is what's wrong, and the second question he would ask is who's treating her. This didn't seem right to him. He felt there ought to be a better way to do things, and he encountered, through his reading, the idea of randomized trials. Randomized trials, as your listeners probably know, is when you test a treatment by randomly assigning similar people to one group to take the treatment and another group not to, and then you see whether the group that received the treatment improves better. Because you've randomized, you've hopefully canceled out all these other factors that might influence the outcome. Chalmers was inspired by this idea. In fact, lots of people I've interviewed become very passionate about randomized trials and the idea of testing whether something works. He went on and did this incredible exercise in the 80s where he decided to bring together and synthesize all of the trials that had been done on treatments for women in pregnancy and childbirth. He did this pioneering exercise, and it took 10 years. They had to recruit hundreds of people to find these trials and bring them together in hundreds of systematic reviews. As a result of this, it produced these two big books I've got on my shelf, but it showed that some of these conventional practices for women were basically unsupported. Really quite invasive practices, like giving women episiotomies during birth or shaving their pubic hair during labor, weren't supported by any evidence. Countless women had been subjected to these invasive and degrading practices just because senior doctors thought this was the thing to do. Chalmers' work helped change that. To go back to your point about people not liking it, definitely. This story I told you was one part of moving medicine away from this kind of practice, which is sometimes called eminence-based medicine, which is basically doing what the most senior doctor in the room says, towards evidence-based medicine. He was compared to a terrorist for being so bold as to suggest that doctors were doing the wrong thing. That was the case in all these fields I've looked at; people don't like to be told that what they're doing is wrong.

SPENCER: Yeah, it feels a number of different biases can come into play there. One is a kind of appeal to authority. "Oh, well, the eminent people in our field say to do it this way." Another is a kind of commitment factor, like, "I've been doing it this way the whole time," and also to learn that you've been doing it wrong, that could be really painful. Imagine how painful it must have been for Dr. Spock to discover that he recommended something that hurt all these babies. You could imagine the incredible pressure to not admit that you actually contributed to the death of all these babies.

HELEN: Yeah, and I didn't speak to Spock, I did speak to somebody who worked with another pediatrician who'd promoted this idea of front sleeping, and she said that it was very traumatic for him to realize that, of course it would be. But another more amusing way of, obviously that's a terrible example, but somebody said to me, when I was interviewing people for the book, that for doctors, being confronted with evidence showing that your practice is wrong is like being hit in the intellectual testicles, because it's just so incredibly painful to sort of realize and be confronted with evidence which shows that what you've been doing is wrong.

SPENCER: You almost need a different kind of attitude towards what you're doing, instead of saying, "Oh, I'm telling the patient what to do, what's good for them." It's like I'm saying, "What's the best thing that we know of, given the current state of the evidence, which is quite a different way of thinking about your whole process?"

HELEN: Well, yes, and that's how medicine actually operates now. The idea now, with evidence-based medicine, is that evidence is kind of one thing that's brought to the table when you're deciding what's best for a patient. You've got the evidence suggesting what treatments might be effective, and then you've got the expertise of the doctor, who, of course, has a lot of experiential wisdom, which needs to be factored in. Then, of course, you've got the values and the preferences of the patient, because this is quite interesting, that I think some people don't think about, is that the evidence kind of tells you whether something works, but it doesn't tell you what to do. An example of that in medicine is, let's say you've got antibiotics, absolutely proven through trials to clear an infection, but that doesn't necessarily mean you're going to use them. Let's say you've got a patient who's at the end of their life who's got an infection, and actually doesn't want to be treated anymore. Of course, you wouldn't treat them. I think that's really important, thinking about the evidence. Evidence alone is not enough to tell you what to do, and that's the same in policy. If you look at policy, there might be evidence showing that a particular policy is effective, but the politicians making that decision have got to think about the cost and politics, and public opinion, and all these other things when reaching that decision.

SPENCER: At the end of the day, there's a set of values you could have, and different people have different values. You could have evidence that you know x leads to y, but whether you should pursue x to get y is going to depend on the set of values you care about. I think you point out at the end of life, these can come up, but they can come up in all kinds of things, like, "Are the side effects worth it?" Take antidepressants; people came to eventually realize through study after study that some types of antidepressants cause a lot of sexual side effects. It's really common. It's something like, I don't know, 40% of patients, 50%, depends on the study. Is that worth it? You can't answer that in a vacuum. It's going to be different for every patient.

HELEN: Yeah, absolutely. I mean, that's the point. I think about taking into account what the patient wants. Nowadays in medicine they talk about shared decision making, which is bringing the evidence to the table, along with what the patient might want and their individual circumstances. Evidence provides a guide, but not necessarily a final decision. Sometimes people get irritated when politicians talk about following the evidence because it doesn't work that way. It's as if they're trying to deflect the decision and say, "Well, the evidence is going to tell us what to do," when actually you have to factor in all these other important parts of the decision.

SPENCER: Yeah, absolutely. One thing I think people often don't realize is that medicine is less the individual doctor thinking about all the evidence and weighing it all, and working in first principles, etc. It's more of an algorithmic process. For instance, if there's a gastroenterologist and a patient comes in with a belly ache, they're going to think about the symptoms, and then they're going to pattern match that to some process, like, "Okay, I'm going to start them on this medication, and then if that doesn't work, I've got my second line medication," etc. So I think that's more what medicine is really like.

HELEN: Yes, exactly, it is, but what people don't realize, I think, is that underneath this is a kind of bedrock of evidence that has already been put together. In the very early days of evidence-based medicine, which grew out of a particular medical school in Canada at McMaster University, the early pioneers were like, "We're going to train all of these doctors in how to properly analyze a randomized controlled trial and how to do their own systematic reviews." But they quite quickly realized that it takes a lot of training to actually reach that level of expertise, and it was going to be much more efficient to produce what I called in the book pre-processed evidence. So now the way that medicine operates is that doctors will be routinely gathering together randomized trials and other evidence, and producing systematic reviews that suggest which treatments are most effective. From these, clinical guidelines are produced, and that's kind of what you're talking about, that algorithm, because that will have already said, if a patient comes in with this condition, then the evidence suggests that these are the best things to do. The doctor is not going back to the library and the journals, which is what they were doing in the early days to look at the evidence themselves; that's simply not efficient. But I find it quite interesting that there's this underlying layer, this foundation of evidence, which has already been built, guiding those decisions.

SPENCER: Yeah, that's a perfectly sensible process. It doesn't make sense for everyone to go back to the original evidence; it's not efficient. At the same time, I think it can sometimes leave patients in a bit of a lurch where they don't fit the algorithm, they're actually a weird case, and they actually need to go back to first principles, we need to look at the evidence. Yes, there are some doctors that can do that, but I think a lot of doctors, that's just not what they're used to doing, they're not necessarily good at that, and a patient can end up being pushed into a bunch of algorithms that don't really fit their situation.

HELEN: Yeah, and I think it's valid. Many people feel excluded from evidence. There are populations, of course, women historically have not really been included in clinical trials or their medical studies. They haven't been studied as much, and so it's perfectly understandable that some people feel that the evidence doesn't necessarily apply to them, and that goes back to what I was saying, which is part of the decision. Of course, in some ways it's good, because we can all use the internet to research things ourselves; it helps if we have some skills to understand when we're seeing valid information that we might want to discuss with the doctor. I think the playing field is leveled a little bit, allowing us to access information which sometimes only doctors had access to before.

SPENCER: On that point, I think very few laypeople are used to looking at things like meta-analyses and systematic reviews, but I actually think a lot more should look at these things. Although they're technical documents, often there is a little plain language summary these days, so if you're curious about a topic, you can often be like, "Okay, let's see if someone has a meta-analysis on it, and there'll be a little summary of, we looked at 200 studies on this, and here's the overall result: this thing works, this thing doesn't work." I find that actually a pretty useful life hack.

HELEN: Yes, I agree with you. I put that at the back of my book. I put a bunch of evidence hacks at the back, and one of them is exactly that. I actually changed how I work as a science journalist after researching this book because I became so enamored by systematic reviews. They sound so geeky, but they're so important. Not just systematic reviews; there are other types of study called evidence synthesis. It's this really intuitive idea that we shouldn't just look at one study at a time. We should look at what the whole body of knowledge says. The problem with looking at one study at a time is that studies can be wrong by chance, or they were poorly done, and that's why you have this classic sort of thing in the newspaper: red wine is good for you one week, and then the next week another study shows that it's bad. What we need to do is look at the body of evidence as a whole, and I do agree. When I now go into a whole new topic, which, of course, I do all the time as a journalist, I tend to look specifically for systematic reviews. Not all systematic reviews have meta-analyses in them, but some do. There are also some quite useful AI tools which will really help you do that. I'm sure most people know that it's probably not the best idea to ask ChatGPT to synthesize all the evidence on a topic for you because it might make stuff up, but there are some specific science-based tools, one called Consensus, another called Elicit, and others, which will only search within the scientific literature and are less likely to produce hallucinations. I'm finding with those, you can ask a natural language question and then even filter by systematic reviews. Now, I will just say one caution, though, is that, like primary studies, randomized trials, you can also get bad systematic reviews. Just because it's a review doesn't mean it's a great piece of research, and that's where I get a little bit stuck. You can find the systematic review, but then it almost takes another level of speaking to the experts to go, "Well, are there hidden conflicts around this that I need to know about? Was this a rigorous study?"

SPENCER: I feel there's this whole stack of thinking about evidence where you start at the very bottom, you're like, "Oh, well, my friend says this thing works, so it works." It's like, "Okay, that's not very reliable. We need to have studies," and it's like, "Okay, we have an observational study, and it finds the thing works well." Observational study, that's not so great. There could be many confounding variables. "Okay, we need a randomized controlled trial", and it's like, "Okay, that's great, but it's just one study, what if they had issues?" Or, "Okay, I know we need a bunch of randomized controlled trials, and we need someone to synthesize it with a meta-analysis, yeah, that's great, but sometimes meta-analyses have all these problems, like publication bias," and you're just like, "Oh my god," and I think, as you go up this chain, the evidence does actually get better and better and higher and higher quality, it's just that even at the top of the chain you can't necessarily always assume that it's going to get the right answer, even at the top of the chain, there can be serious issues with it.

HELEN: Sadly, yes. That is science. And I think one huge lesson that comes out of looking at evidence across these different fields is almost around humility, around evidence and science. Evidence is always changing, it's often uncertain. It can be flawed, as you said, it can be biased nowadays. It can be AI contaminated, and it's only ever one factor in a decision, and that's always so, there's always this big caveat, we're always reaching for these kinds of really concrete answers, but we can use indicators. It's better if something's been peer-reviewed. It's probably better if it's a systematic review, but yeah, it is hard to kind of find that certainty, and it's interesting. You just talked about such a hierarchy of evidence. There are lots of kind of formal hierarchies of evidence that are used in medicine, with anecdote at the bottom, and then you have case studies, and it's supposed to show the kind of increasing sort of quality and rigor of the evidence, you have case-control studies, observational studies, randomized trials, and then at the very top you often have systematic reviews and meta-analyses, but some people even push back against that, because it sort of suggests that some evidence is more superior to other forms of evidence. When, as we've just said, you can have bad randomized trials or bad systematic reviews, and sometimes you don't need to do a trial, and I think one of the real frontiers around evidence now is how do you bring together numerical evidence, like say from a randomized trial with, let's say, experiential evidence, or life experience that's being collected by an indigenous community, and kind of bring together these really different forms of knowledge, and I'm not sure anyone has quite solved that problem.

SPENCER: Yeah, well, when we have these hierarchies, what we're really trying to do is give some simple heuristic, like, "Oh, a randomized controlled trial is better than an observational trial, right?" But if you had a randomized controlled trial with four people in it, well, that's completely worthless, you know that you learn nothing. So, okay. Well, it's not always true. The randomized controlled trial is better. I'd rather an observational study with 30,000 people than a randomized controlled trial with four people. That being said, I think it really is true that, on average, randomized controlled trials are much better evidence. But if you say, "Well, what really determines the quality of evidence? I don't actually think it's that subjective. It can be hard to figure out, but I don't think it's subjective. I think it actually ultimately comes down to Bayes' rule, which basically allows us to quantify the strength of evidence, so we get from something, and if we kind of try to apply Bayes' rule systematically, we say, "Ah, well, there's a reason why randomized controlled trials tend to give better evidence. It's because the chance of getting this outcome if this thing were not true gives you a much stronger signal, or at least the probability of getting this result if the hypothesis is true, compared to the hypothesis not being true, is much bigger with a randomized controlled trial than it is with another type of study, on average.

HELEN: Yeah, absolutely. I mean, we're using these as heuristics. I completely agree, but in medicine it was recognized long ago that there are some things which you just can't test in a randomized trial, like some people get frustrated when the randomized trials are put on a platform as being superior to all other forms of evidence. You can't test, within a randomized trial, surgical procedures are quite difficult to test in a randomized trial. It is done, but you have to have complicated shams, or you can't test another thing. I reported on, are home births or hospital births safer. This was a long time ago. This was being looked at because you can't randomize women to tell them where to give birth. So, there are lots of situations where we just have to accept that we can't have this perfect gold standard, and that's really true. Again, going back to conservation, you can't randomize birds into a control group and a treatment group, you can't randomize oceans or forests, so there are all these important situations where we want to collect evidence, but we just have to accept that there are different methods and kind of work with the best evidence that we have.

SPENCER: Yeah, it's a good point. Funnily enough, a family member of mine had knee surgery at the time people believed it worked, but there hadn't really been randomized controlled trials. It's all kind of theoretical. I was like, "Oh yeah, it makes sense mechanistically based on the way the knee works, so we should do the surgery, and they've been doing it for many years." And then finally, a randomized controlled trial was done, and they were like, "Well, how do you control that, right?" Well, they did a sham surgery where they actually opened up the knee but didn't actually do the procedure, and it was no better than sham surgery." I think surgery, in particular, often works this way, maybe in part because it feels more mechanistic. You're like, "Ah, well, we can see the thing wrong within, so if we just shave this off, that should correct it." But indeed, there have been quite a few cases where the thing that seems obvious mechanistically just doesn't work.

HELEN: Yeah, that's a really good example. I think I used that one in my book, around routine knee surgery, it's been done on millions of people. There was a long-term study that had followed some of these patients that had that type of routine surgery, and showed that the outcomes were not better, but a huge waste of resources. Again, that goes back to what you were saying at the beginning. It does sound intuitive that this should work, and yet rigorous evidence suggests otherwise.

SPENCER: Let's talk about the power of randomization, because I think you mentioned how some scientists get lulled in by the power of randomization. I think that for good reason, randomization is this incredibly cool idea, but I think a lot of people don't get what's so cool about it.

HELEN: Yeah, so what's cool about it is that you want to try, and if you're comparing, let's say, a medical treatment of the pill, you want to be sure that any differences between the group that gets the pill and the group that doesn't get the pill are due to the pill itself, but there could be other reasons for differences between these two groups, like one of them could have more older people or more women, or some other factor. Randomization basically says, well, we're going to randomly allocate all these other factors to the two groups, so you should end up with roughly equal numbers of people of a particular age or an equal mix of men and women. If there is any difference in the outcome for those two groups, then we can have more confidence. We can't be completely sure, but we can have a lot more confidence that it's due to the treatment itself. I think also people get really fired up because it's quite simple. Somebody described it to me as she was trying to sell this idea to policymakers and explain why we should randomize people to different types of welfare policy. She was saying, look, it's just like randomly assigning people to get toothpaste or not. Some of them are going to get toothpaste and some of them won't, and then we're going to see which group has healthier teeth at the end. It's a very simple idea once you grasp it, and people have gotten fired up. In the book I looked at, there's been randomization of arrest, there's been randomization of beer glasses. I mean, it's just remarkable once you think about it how many things can actually be tested in that type of study.

SPENCER: It's interesting to think about what you had to do without randomization, or what you still have to do when you can't randomize. Because you're like, "Okay, well, it could be that those getting the pill happen to be older. Let's do a correction for that. Oh, wait, but it could also be that the gender is imbalanced. Let's do a correction for that," and you kind of realize not only do you have to do all these corrections, but you have to somehow know everything to correct for, and you have to have collected all that information to allow you to make the correction. So it's almost this nearly impossible task because you never really know if you actually know everything to correct for, and do you have all that information so you can actually correct it? So you're always left with this big question mark in non-random or most types of non-randomized studies.

HELEN: Yes, absolutely. Scientists do try to do those corrections, but it doesn't mean that those studies aren't valid and useful. I mean, there are situations. The classic example where observational studies were absolutely enough is because there was a strong signal, which is around smoking and lung cancer. Classic studies in the 40s and 50s showed this really strong association between smoking and lung cancer. In some senses, a randomized trial wasn't necessary. They weren't that common then, anyway. It probably would have been a bit ethically problematic to randomly assign people to smoke for five years. There are still cases, if you've got enough big observational studies which consistently pull out the same association, then you can start to have more confidence. But yeah, I mean, I'm sure you know, when you see just one study come out and people are trying to say there's causation there from an observational study, then you get a bit cautious, because however much you've tried to control for all these other factors, it might not have been enough.

SPENCER: Right. Obviously, there are cases we genuinely can't randomize, but I think with some cleverness, randomization can be used more than people realize. Even in the smoking case, you could randomize people to not smoke or give them an intervention around reducing smoking. Or even in cases where you're like, well, that really violates someone's autonomy. Let's say mothers giving birth at home versus at the hospital. You could give them a special deal, like randomly assign who gets a special deal for covering more of their costs at the hospital or something like that. So use that as an instrumental variable. I think with some cleverness, we can inject more randomization.

HELEN: The women one sounds slightly ethically dubious to me, because you'd be incentivizing women to do something they might not want to do. But I completely agree with you about smoking. That's exactly what happens in a lot of cases, is that you can't force people to follow a certain path, like give up smoking, but that's exactly what you can randomize. You can randomly assign people to receive programs which encourage them to follow this particular health behavior, like give up smoking or cut down on alcohol or something like that, and that's exactly what scientists try to do.

SPENCER: Let's talk about the mother one, because that might be interesting. Maybe you and I have a genuine disagreement about the ethics of it. So, the study I was imagining was, imagine you have a number of women who are planning on giving birth at home, and you randomly just say half of them picked at random, say, 'Hey, if you do it at the hospital instead, we'll give you this special discount on health care. Is that unethical?

HELEN: Of course, I'm coming from a situation where my health care is free.

SPENCER: In America we don't have that.

HELEN: I did have my kids in America, so I've tried both. I still worry that then you're giving people a financial incentive to make a health choice. I mean, yeah, but this question has to be taken. It would be a good ethical debate. So, I still would have qualms about that, I have to say.

SPENCER: Interesting. What about so sometimes people get really uncomfortable about randomization, like, "Let's go to schools and randomize something about the curricula." A lot of people really get their hackles up, they get very upset about that, and I kind of understand that, but what I don't understand is that they're often okay with just doing something that has never been proven. In other words, they're like, "No, you can't randomize between two things that have never been proven, that's unethical," but you can just give them one thing that's never been proven. So, I think to me there's a bit of a weirdness there.

HELEN: Obviously, I've been speaking to the wrong people, because I haven't found those, but I mean, there are loads and loads of randomized trials in education, which I find a really interesting example where that field has built up this huge body of evidence, evidence-based education, which is a thing. It's not as widely embraced as we see evidence being used, for example, in medicine. One group I wrote about, there's an incredible organization over here called the Education Endowment Foundation, which has synthesized all of the randomized trials that have been done on different education techniques. These might be things like, does particular types of feedback help children learn? Does tutoring in small groups help children learn? They haven't just looked at whether these things are effective, but they've actually compared which one is more effective than another. The idea is now this actually happens in the UK, is that head teachers who've got some money to spare can actually go to this database that they run and say, okay, how can I get the most bang for my buck? How can I give my children the most learning based on the money I have to spend using evidence, which I think is a remarkable resource. So, to go back to what you're saying, in terms of can you randomize things? These things are being randomized. I just don't think many people realize that. I think it's something like, I might get the figure wrong, but it's something like half of English schools have been involved in randomized trials of educational techniques.

SPENCER: Wow.

HELEN: Yeah.

SPENCER: So, that's interesting. So, do you have an encounter with people who are against randomization? I definitely see these people around because they feel uncomfortable with the idea of randomizing, for example, what their kids get.

HELEN: I mean, obviously, I've interviewed a lot of scientists who are very vocal about the opposition they face. For example, I've talked to people who've really pushed the idea of randomizing policies to policymakers in Congress, and the reaction was often just like, well, what's the point? This would be too difficult. How can you randomly assign people to receive different welfare programs, for example? It seems unethical or too costly. So, clearly, yes, there's a lot of objection to it. So, yeah, maybe I don't chat with them all the time, but I certainly hear about those objections.

SPENCER: Do you think the term evidence gets abused a lot?

HELEN: Yes, I think that's a great question. The term is quite slippery, actually, and maybe that's leading to some confusion about it. The dictionary definition of evidence is information that indicates whether a belief or proposition is true or valid, so it's information. The way you and I are talking about it, in the way I did in my book, is I'm talking about scientific evidence, empirical evidence gathered through careful observational studies and experiments. Often, when people are talking about it, they think of evidence as being synonymous with that type of evidence, scientific evidence. But there's evidence in a court of law, which is just information presented in court. There's experiential evidence, which is our lived experience, which is also valid information. Even in science, the term is used in different ways, so we're not always talking about the same thing. Because of that ambiguity, the risk is that it can be abused; people use it to mean some information. It could also be abused in terms of evidence-based claims, like evidence-based homeopathy, where people think that just attaching the term evidence-based to something magically makes it true. It's also a bit problematic.

SPENCER: Yeah, it's interesting. As a mathematician, I can't think of evidence as one thing, regardless of the source, whether it's scientific or anecdotal. It's just a question of how strong it is. An anecdote is evidence; it's just much weaker. But maybe that viewpoint can muddle things and make it more confusing, so I don't know if it's helpful.

HELEN: Yeah, I think that's quite an enlightened view. Some people would never think that an anecdote was evidence. After speaking to all these people for the book, I've reached a quite inclusive position around evidence. It is information used in making decisions. I think we need to be clearer about what we're talking about, so you will find some people who will say research evidence or evidence from research to clarify that we're talking about science here, not just general information.

SPENCER: Yeah, it's funny, I have a little bit of a pet peeve sometimes when scientists conclude their report with, "There is no evidence that x," and I'm like, well, there is evidence, it's just not a randomized controlled trial. They're using the term in a different way; in a way, it's talking in a very formalized way about, well, okay, not only are we only talking about scientific evidence, but we're only talking about scientific evidence meeting a certain bar. I tend to take a broader view of, well, there's all kinds of evidence out there.

HELEN: Yeah, very enlightened, as I said.

SPENCER: It's interesting to think about when anecdotes can be strong evidence because in everyday life, many people make their decisions based on anecdotes. A friend might say, "Oh, I did this thing and it really helped me," and you're like, "Oh, cool, I'll try that." If it's not dangerous and it's a cheap experiment, it's not a crazy thing to do. But it's interesting to think about, are there actually cases when anecdotes could be strong evidence? I think there probably are some when the change was so dramatic and couldn't possibly have been caused by anything else. If someone gives you a new hair dye and says, "This will turn your hair blue," and then you put it on and it turns your hair blue, you're like, "Okay, that works, it turns your hair blue, right?"

HELEN: Yeah, absolutely.

SPENCER: Yeah.

HELEN: In some senses, if you think about it, anecdotes are where a lot of studies start. Somebody observes a side effect from a drug. That will start as an anecdotal observation, but then you might want to go on, if there are enough cases, to actually study it. I wanted to talk a bit about trust in science, which is where this anecdote point comes in, because I'm reporting a bit on that for Nature at the moment. I think one challenge, I'll give you the big picture in a sec, because there's a lot of conversation at the moment about whether trust in science is eroding. Some people talk about a crisis of trust in science, and I've been holding that claim up to scrutiny. When you look at global surveys that ask people whether they have confidence in scientists or trust in science, generally, there's a fairly consistent pattern of high trust. About 80% of people will say they have some trust in scientists, compared to much lower figures for journalists and politicians, who can come right at the bottom. However, there are also problems around trust. One I'm hearing is the growing power of the anecdote, with more people rejecting evidence-based advice. Vaccines, of course, are a classic issue, as is rejecting conventional medical treatments. A lot of scientists are concerned that the voice of science, or the trusted information, is being crowded out in this complicated media ecosystem we're in, where many people are talking about anecdotes. That's what you're encountering on social media: "Well, it worked for me. I think that would be great. Why don't you try it?" I talk to people who argue that it's not that people are losing trust in science; it's just that scientists are losing influence in this world, and that may be the problem that needs to be tackled.

SPENCER: So, are there repeated surveys on trust in science, and have they shown that it actually hasn't gone down over time, as people perceive?

HELEN: Yes. Globally, that is generally the picture; there's been this quite high sustained level. Now, the US, though, is an interesting exception to that. What's been happening in the US is this political polarization around trust in science. There have been some long-term studies that look at confidence in scientists in the US, and they have shown a general dip since the pandemic. Even before that, there was political polarization, with Republicans showing a clear decline in trust in science, whereas Democrats have maintained a high level of trust. The concern among scientists is that the Trump administration is using this distrust to say, "Okay, this isn't a trusted institution, so we can take it apart," using that as one of the justifications for attacking science, which is what's happening at the moment. It's a really complicated picture. Most interesting things are complicated, especially around trust in science. I think those two things — political polarization and scientists losing their influence in the online environment — are the two key problems I've been hearing about.

SPENCER: Do you think it partly relates to viewing scientists as the outgroup? It is true that in academia most people are not even centrist, but actually left of center, and I think the right has become wary and kind of views it as they don't represent us, they're biased, etc.

HELEN: Yeah, I think that's part of it. That's certainly what I've heard from other people, that it's viewed as part of this elite, this sort of academic elite. Many people feel excluded from higher education, it's too expensive, and so I think that is part of the issue. There was an interesting study that looked at trust in science that came out in the UK in April, which also showed that there was a kind of difference that scientists themselves tended to be more left-leaning than the general population, which could feed into what you're talking about. That suggests that there is a difference with scientists, which could potentially undermine trust and confidence in that group, or lead scientists to be out of touch or not understanding of the population they ought to be trying to help.

SPENCER: I feel like there's much more discussion of the politicization of science than of the methodology of science, but from my point of view, most of the issues in science are methodological. I do think political bias can cause some issues on certain topics. I do think science can get corrupted because of politics, but I think much more often it gets corrupted because of bad methodology. I'm curious what you think about that.

HELEN: That's a good question. I'm instinctively interested in methodology. I think, probably like you, just while we're having this conversation, science can never be separated from politics. It is inherently political in the way that it's funded through political decision-making. I think it's problematic to suggest those two things are separate, or that science is some objective truth that doesn't have to engage with politics. That's not right. Clearly, in the US, politics is probably the biggest challenge that science is facing at the moment. But in terms of methods, yes. It also fascinates me. We haven't begun to talk about AI creating a challenge for science as well.

SPENCER: Yeah, that's a great question. I mean, AI is evolving so fast. The AIs of today versus a year ago are already vastly different. Is there information known now on the impacts of AI views on science, or how people are using scientific information?

HELEN: That's such a good example of where we don't really have the evidence that we need to know what to do. Let's look. Actually, I was going to talk about social media, but maybe that's going off the topic of AI. The question is, what? This is another example of where policy is sort of acting ahead of the evidence. All around, what are the impacts of social media and smartphones on adolescent mental health? We've seen the change in policy in Australia, where young people have been excluded, or the law has come in that excludes them from social media. Lots of other countries are talking about the potential harms of social media, but the conversation is running ahead of the evidence. First of all, the evidence on the true impacts of social media and smartphones on adolescent mental health is quite complex and does not necessarily point to a very clear harm from them. Secondly, should you ban them? Nobody really has the answer to that, because we don't actually know the consequences of banning it. There are now some early indicators coming out of Australia, which are finding that if you do ban the students from social media, then something like 70% of them are getting access to these platforms anyway. I was reading something this week about how it's kind of reducing their news consumption, which is a side effect of these policies. Looking at these cases where we just don't have the evidence to know what to do is similar with AI.

SPENCER: Yeah, and so many things in life are that way, unfortunately, where we just don't have that good evidence. Some people have argued on the social media front that even if we can't be that confident, there's a kind of precautionary principle: why are we subjecting kids to this? I suppose you could also argue on the other side, saying there's a precautionary principle in taking it away too; maybe you're taking away something valuable. What do you think about that?

HELEN: Yes, I mean, I haven't fully formed my opinion. I definitely see the argument for taking phones out of schools as a distraction. I think there's been an overplay of the argument that they're causing this adolescent mental health crisis. I wrote a whole piece for Nature about this. There are some benefits to phones. For many people, they are a source of finding important connections and communities that they wouldn't otherwise have. People are gaining information from them, so it's quite easy to look only at the possible harms and talk about the precautionary principle without thinking of these other things that we might lose along the way. I definitely think that is not at all a cut-and-dried case.

SPENCER: My suspicion of why the social media debate is so difficult to resolve is that I think it's very heterogeneous. I don't think a 15-year-old girl spending five hours on Instagram a day is the same as a 30-year-old male spending five hours a day on TikTok. I just don't see any reason to think that those would be similar situations with similar outcomes, and yet we kind of almost homogenize and talk about social media and how to deal with it.

HELEN: I completely agree, and of course, that's also the case among adolescents. There are clearly cases where social media has been linked to harm for adolescents, terribly tragic cases. But the difficulty is extrapolating from that to say this is harmful for all people, and that's where I think the evidence is not really there. Your point about it being individual is absolutely valid. From what I was hearing from the experts I interviewed, the people who might be more at risk might already come to the table with some vulnerabilities or potential susceptibilities to mental health problems. Of course, it's going to be so different, and those types of things also get washed out when you do population studies.

SPENCER: Yeah, I think another really unique challenge for social media, and maybe eventually AI too, is that it can actually change the whole setting and local culture. We could talk about what's the effect of removing one kid from social media, but if all their friends are on it, they're now in a different culture, where not being on social media might actually prevent you from maintaining your relationships or knowing about the cool parties, or whatever. Then it becomes this web where you have to study simultaneously, which I think is much harder than studying individual effects.

HELEN: Yeah, absolutely. Children might feel ostracized or lose their friend groups, so it's a really complex situation. There are some interesting studies starting up now, or that are underway, which are trying to get at some of those things. It's similar to AI. We need to try and collect the evidence, make sensible decisions, but also be collecting the evidence as we go along, and potentially be willing to change course in the future.

SPENCER: There's this funny thing with AI, where it does seem like it tends to be more likely to rely on pretty good sources. If you ask it about conspiracies, it by default doesn't just tell you the conspiracy theories are true. It will challenge them at the same time. It has this other thing it does, which is this sycophancy, where it reinforces whatever you currently think, and I don't know where this is going to net out. On the one hand, maybe it's giving you better sources than if you were to go to YouTube. On the other hand, if you have a false belief, maybe it's going to pick up on that, and because of the way that these AIs are trained to say the things people like, it's going to reinforce your false belief. So, I don't know which way it's going to go.

HELEN: Yeah, I think, to go back to science, obviously AI is changing science so much, but one thing that's happening is just the huge volume of output is going up in terms of the number of scientific papers, which is just soaring. The number that contains AI is just facilitating this. Not all of this is necessarily a complete AI slop. Scientists are also able to use it to speed up their production, but many journals are talking about how they're becoming overwhelmed with papers. Funders are overwhelmed with grants that have been written with AI, so it's fueling this huge growth of information. To talk about a positive way in which I think AI can help, and this was again something I looked at in my book, is around synthesizing evidence. One thing that AI is really good at is making sense of large volumes of information. There's a big conversation going on now among all of these evidence synthesis people that I've spoken with about how we can develop the best tools to use AI to help us do these complex studies, for example, systematic reviews, and ensure that they maintain their rigor. There's a whole community of people trying to build this infrastructure, the dream being that ultimately you and I, rather than wading through Google trying to find evidence, are genuinely able to, almost at the push of a button, get an AI-fueled or AI-powered accurate synthesis of what the current research says that answers our question. That's what we're sort of reaching for, and I, in my optimistic side, think that actually that is an achievable goal.

SPENCER: Yeah, sometimes these processes of going through hundreds of papers could take, literally, researchers can spend a year doing a meta-analysis, or three years sometimes. The idea that maybe an AI could go find all the papers, do a preliminary review, decide which meet the right criteria, and preliminarily extract information is an incredible idea that could be almost meta-analysis on demand.

HELEN: That could really be happening. I would say we're a long way towards doing that. There are already tools being developed and tested. The key is these things need to be evaluated to show that they're good. There's been a huge excitement about producing AI tools like the ones you're talking about. They actually need to be tested, but nevertheless, some of the grunt work of doing a systematic review can already be done by some of these AI tools. You're right, scientists used to think with horror about systematic reviews because it was going to be two years of their life to just do one, and it really shouldn't be that way.

SPENCER: Yeah, and even if the AI one is only 60% as good, like hey, 60% is good in 20 minutes compared to a three-year process.

HELEN: That depends on what you want to do, what you need your evidence synthesis for. When I was interviewing people, I found there were these evidence synthesis purists who were basically like, it has to be a systematic review; it's systematic review or die. It's going to be gold standard or nothing, but in some cases that's right, because if you're doing a systematic review for medicine, it really matters that you found every single study and you've got the perfect meta-analysis, because people's lives can depend on it. Then there are other people who are complete pragmatists who say, my policymaker needs an evidence synthesis tomorrow, so I'm just going to do the best I can in the time available, and that's okay. I think there's sometimes a bit of a tension between these purists and pragmatists in evidence synthesis, but the reality is that different types of synthesis suit different situations.

SPENCER: Yeah, absolutely. Sometimes it's enough to just have a reasonable idea of things, and sometimes it's life or death, and you really need the best answer. We talked earlier about how people often trust anecdotes over science. What's known about how to talk to people who reject evidence or who don't seem willing to listen to scientific evidence?

HELEN: Yeah, I looked a bit at that for a story I wrote for Nature about how to speak to a vaccine skeptic. I've had this myself, but we also felt our readers, who are basically scientists, have these situations where you encounter someone who questions vaccines, and that can be quite confronting as a scientist, because you're like, I don't really know what to say. So I set off to find out. And of course scientists have studied that problem and tried to understand how best to talk to someone who might be questioning vaccines. The answers really were, first of all, don't dismiss them; don't belittle their concerns. Be open, ask curious questions, because many people have very valid questions around vaccines. The other thing was, if you can provide some accurate information. Often your healthcare provider is going to be the best person who might have that information at their fingertips, but if it's you or I, we might not. We might be able to say, hundreds of studies have suggested that the benefits of vaccines generally outweigh the risks, or I spoke to people who said it's okay if you don't have the evidence yourself, but maybe it's okay to just offer your opinion. They've offered you their opinion; you might be able to say, I chose to vaccinate my children because I felt that the benefits outweigh the risks. That was some of the information, and that kind of approach has actually been tested in randomized trials, sometimes called motivational interviewing, where you ask about people's concerns and offer accurate information. It has been shown to be effective in some circumstances. The other thing I've come across quite a lot, talking about trust in science and addressing people with concerns around vaccines, is the importance of acknowledging a bit of transparency, like acknowledging uncertainties in evidence. There are quite a few studies suggesting that people are more likely to trust information or people who are giving them information when we're open about those uncertainties and any flaws in the evidence than when people try to project, just to say, these are the facts.

SPENCER: Right. It seems to people, I think, that often if they give any flaw in their own opinion, then it's going to be giving points to the other side. It's, "Oh, I don't want to give them even one iota," but if you say, "You know, vaccines can hurt people sometimes, for example, with this vaccine, there was an issue, and it was recalled," but then it kind of gives it a concession of, "Oh, okay, I'm not trying to overclaim here, I'm not trying to just beat you, I'm open to seeing multiple sides on this issue, maybe that makes it a little more palatable."

HELEN: Yeah, I think it's interesting that a lot of people I've spoken to would talk about how to get evidence used. It does come down to it's much more relational; there's still a kind of trusting relationship. The data says one thing, but there still has to be a sort of circumstance into which we're communicating as human beings about what that evidence says, and I think being open about things we don't know is just a natural human instinct, isn't it? Maybe when scientists project, as you said, that I have to absolutely know the facts, and this is just how it is. Science communication, I think, has moved beyond that point of that kind of didactic approach; it's much more about having a conversation, being open, being transparent.

SPENCER: Yeah, I noticed this with communication around COVID. In New York, there was a big push to put people on ventilators. There was a lot of talk about ventilators, but it was expressed, at least as far as I remember, pretty openly, like, look, this is what we think is the best thing right now. It wasn't like we know ventilators are a solution, and it turned out ventilators were bad. If anything, maybe they either didn't work or they maybe even harmed people, so it was actually a big misstep, but as far as I could tell, there was very little blowback; people weren't really angry about it, and I think it's because they were just sort of open that this is what we think right now, but there's a lot of uncertainty. Compare that with something like masking, where I think many people in the US actually did just literally get false information about masks that was confidently stated. I was told, "Do not buy masks. They do not work." And I literally got into an argument with someone who worked at the Centers for Disease Control and Prevention, not in the US, but a different country. I literally got into an argument with them on Facebook, and I was like, I don't see how you could say that masks don't work. Look at all this evidence that masks work. What are you talking about? What it was is an attempt to get people not to buy masks so that healthcare workers could get first access to them.

HELEN: Oh, yeah.

SPENCER: But to me that was a major misstep in science communication. But I'm curious what you think about that.

HELEN: I've heard basically exactly that, because again, when reporting on trust in science, a lot of people feel that was a key misstep in COVID. It was overconfident pronouncements of the science at the time, without acknowledging the uncertainties. I was just talking about how important it is to do that. For example, we have a vaccine. It sort of gave the impression we've got a vaccine, it works, we're going to sail out of this, and what people feel now should have been said was, we have a vaccine, it works against this strain, viruses mutate, there are many uncertainties, we might have to update it. So much more open, humble acknowledgment of the uncertainties might have been helpful. Now I do also think it's easy to look back and see those things in hindsight; people were trying to do the best they could at the time. But this point I was talking about, about acknowledging being transparent, acknowledging uncertainties, I think that partly is this hard lesson that was learned from COVID.

SPENCER: Yeah, and you can understand where healthcare folks are coming from if they're like, "Oh, well, if we say the vaccine is not totally effective, but you should get it anyway, maybe fewer people will get it, and lives will be lost." You could see this; even completely well-intentioned people could be like, yeah, maybe we should just be less nuanced on purpose, because it's going to save more lives. I think that was a mistake; I think it actually would have been better if they had expressed it with more nuance and said this is not perfectly productive, but it will reduce your risk by a substantial amount. Yeah, but even years later, I hear people railing against how they lied to us about, they said you can't get COVID when you get the vaccine, or whatever, and it's like, "Ugh."

HELEN: Yeah, well, a hard-earned lesson, and I don't feel like the repercussions of that are going to roll on for a long time. Has that been fully absorbed? I don't know. We'll have to see in the next pandemic.

SPENCER: Yeah, absolutely. On the going back, the question of, okay, let's say someone's not willing to listen to evidence, it seems to me that interpersonal factors end up mattering so much in those kinds of conversations, just building rapport, building trust. We can fixate on making the best arguments, providing them with the strongest evidence, but maybe it doesn't depend on that nearly as much as we'd like to think.

HELEN: Well, yeah, I think that's true. I mean, people have studied what works to get evidence used, the evidence on evidence use, which all gets a bit meta, but one thing that comes out of that, which won't surprise you at all, is that issuing big fat reports with lots of citations is not really the way to get evidence used, because nobody reads them. Building trusted relationships is a really important part of it. There are quite interesting lessons that come out of evidence-based medicine. So, let's look at these different fields; evidence is completely embedded in medicine. I mean, okay, there are lots of poor practices that still happen, but in general, most of modern Western medicine is built on this bedrock of evidence. I think there are two reasons for that, which might explain why evidence struggles in some other fields. One of them is that evidence-based medicine grew up in a medical school; it emerged in a medical school, and students are still taught about evidence in medical school, so it's just part of your training. You go out, you've already learned that this is the way to operate. That doesn't really happen if you look at, say, teachers or police. There's a huge body of evidence that could inform those practices, but they're not necessarily expected to learn that when they're training. For managers in business, there's lots of evidence around good practice and management, for example, but we just don't go through that formal training or have that expectation. That's part of it, but the other part is accountability. If doctors don't follow evidence-based clinical guidelines and somebody gets hurt, then they're going to get sued, people are going to get angry, they might get struck off by their professional body. Whereas, if you look in other situations, if a conservationist doesn't use evidence on what actually works to save Natterjack toads in the sand dunes of the UK, unfortunately, nobody really notices or cares, so there's no accountability there. I think those two things around training and accountability are quite fundamental when it comes to embedding evidence.

SPENCER: Yeah, that's a really good point. Do you think education follows best practices around evidence, or do you think it largely ignores them?

HELEN: Education in schools, you mean?

SPENCER: Yeah,

HELEN: I would say that there's a growing movement in evidence-based education, and it varies from country to country. As I mentioned to you before, it is making some inroads in the UK, and I think part of that is because there is a kind of national system, so the national curriculum and practices are set by a single body, or there's a unified system, whereas in the US it's much more each school district for itself. When you've got that kind of patchwork approach, my impression is that it becomes a little bit more difficult, because you're trying to change one at a time. There is a surprising movement around evidence-based education, which is fantastic, of course, because there are simple practices like offering good feedback for children, which can actually be very effective, and giving them months of extra learning a year.

SPENCER: Yeah, I've seen some of these ideas that seem to be gathering evidence around them, like how do you help kids remember things better? How do you get them more engaged during a classroom? When I was a kid, it was the teacher at the front lecturing you. A lot of kids aren't paying attention, or even if you do pay attention, you don't retain it, because it's just sort of passive. So, I think there's a lot of exciting ideas.

HELEN: Yeah, and there are practices. The Education Endowment Foundation in the UK actually has a ranking where you can say, "Okay, what will produce the most learning for the investment I have?" At the top comes metacognition, which is a simple idea in education of helping children understand their own learning. For example, if you do a math problem and then think about it, and go, you know what, it really helped me to write out all my working, then you're practicing metacognition because you're helping yourself learn. That's one effective practice. Another one is feedback, so particular forms of specific feedback that helps children know how to change their behavior is quite effective. There are other methods that don't seem to have as much evidence behind them. Homework doesn't have as much evidence as you might think, and things like school uniforms are on this list. I found some of these quite amusing. I don't know why we'd expect school uniforms to be particularly effective at helping children learn. Even class size is also in this table of evidence that I've looked at, because a lot of people assume that if you reduce class size, it will improve learning, but actually the evidence around that is not so strong. Partly because in order to have a change in learning, let's say most standard classes are 30 children, it seems what really makes the difference is that if you want to reduce class sizes, you have to reduce it enough that teachers can actually change how they're teaching, and that only happens if you get it down to 15 or below. If you just reduce it by a few pupils, it probably doesn't make much difference, and of course, that's a very expensive way to try and improve learning when there might be much more cost-effective techniques that you could use instead.

SPENCER: That's really interesting. Final thing I wanted to ask you about before we wrap up, deep diving into the evidence. I'm curious, on a day-to-day basis, do you find that you use evidence differently, or look for evidence more often, or think about evidence differently in your own life decisions?

HELEN: Yes, I do. I mentioned before that, professionally, as a science journalist, I am much more likely to start off looking for a systematic review rather than just looking around. I am still going to talk to the experts as well, but the whole evidence synthesis has become much more part of my practice. Also, in personal decisions, if I get a diagnosis and I'm looking around for treatments, I am more likely to look for a systematic review. I have to say, it has also made me perhaps more comfortable than I was before in situations where there isn't evidence, because it made me realize how many areas there are where we are still just coasting on conventional wisdom or anecdotes. I almost feel better about those situations because I understand that the evidence isn't there that I need to make this decision, and I still feel empowered to move forward. Whereas before, I felt there was some hidden information that I ought to have. To go back to the screen time example, I am the same as all other parents. How much screen time should I allow my kids? There is no definitive evidence that's going to give me an answer to that question. I can look at what evidence is there, I can speak to experts, and then I can reach a reasoned decision based on my own circumstances, so it's empowering either way.

SPENCER: It seems people can go different ways. It's like, "Oh, there's no evidence to make this decision. Oh no, what do I do?" Or it can be almost like, "Oh, there's no evidence, so either option's okay, or any option, as far as I can tell, is okay. So I could just use whatever other criteria I have."

HELEN: Yeah, ideally evidence is part of the conversation. It's part of the decision, but we just have to accept that often it is incomplete and it doesn't give us the crisp answers that we want to hear. But that in itself, I think, is empowering.

SPENCER: Helen, thanks so much for coming on the Clearer Thinking Podcast.

HELEN: Thanks so much for your great questions.