SciencePodcast_180831

Transcription

1 00:00 Sarah Crespi: This show is brought to you in part by American innovations, a podcast from Popular Science author, Steven Johnson. If you enjoy keeping up with the science news, hearing about the latest breakthroughs, you'd probably also gonna enjoy a show that gives you a broad context, to put all of these new scientific information into its place in history. This podcast, American innovation, tells of the science behind some of the greatest innovations of the last century, as well as the people and places that lead to those aha moments. 00:31 SC: The newest episode, for example, talks about the early development of AI. I know we're always talking about artificial intelligence, machine learning on this show, but we very rarely get all the way back to when they first started to try to put human intelligence into a machine using chess. Subscribe to The American innovations podcast wherever you're listening to the Science Podcast. [music] 00:57 SC: Welcome to the Science Podcast for August 31st, 2018, I'm Sarah Crespi. In this weeks show we hear from news writer, Kelly Servick, about a replication study, that focused on social science research, published in Science and Nature. How did we do? And how does this study compare with other replication work. And Meagan Cantwell interviews Emily Brodsky about her paper that shows fracking may lead to earthquakes. More than 10 kilometers away from the fracking site. Finally, for the book segment, Jen Golbeck interviews Judea Pearl and Dana Mackenzie the authors of; The Book of Why: The new science of cause and effect. 01:35 Judea Pearl: In statistics causality became taboo. In machine learning, it doesn't exist. 01:44 SC: Now we have Kelly Servick, a staff news writer here at science. Hi, Kelly. 01:48 Kelly Servick: Hi. 01:48 SC: Your story in the magazine this week, is about a new attempt at replicating scientific results. Again, in the field of social science, but this time, it uses papers published in science and nature from 2010 to What was the scale of this study, How many studies were in the study? 02:06 KS: There were only 21 studies in this study, it was an exhaustive list of the... Like you said, the interventional sort of the experimental social science papers that got published in those five years. 02:16 SC: And who did this, who looked across all these papers and replicated them. That's a big effort, right? 02:22 KS: Yeah. This was a collaboration between five labs that ran the actual replications and that included the Center for Open Science, which has been involved in a lot of these replication efforts to date. 10/29/18 Page 2 of 12

2 02:32 SC: So replicating 21 studies. How long did it take and how much did it cost? 02:37 KS: One of the authors estimated that it cost about $250,000 dollars. But you have to think about the fact that these labs were also giving their time and their resources to do these experiments over, and at a scale that was even bigger than the original experiments. 02:52 SC: Right. That's one of the things that really makes this unique, that they recruited more participants than the earlier studies, than when they were trying to replicate. And then they were generous, and they were able to, if things failed on the first attempt, they would just pile on more participants. Did that work out for them? What were their numbers like when they looked at the results of replication? 03:10 KS: Yeah. So ultimately, they were able to successfully replicate 13 of the 21 studies, about 62% of them. But as you've said, this effort to be generous and allow for the possibility that maybe there was just a smaller effect size than it was in the original paper. As a result of that effort two papers that originally counted as non-replications when they expanded a little bit became successful replications. 03:34 SC: We're in the mid-'60s here on replication, which just from a very objective point of view is not great. 03:41 KS: Right. It's funny to hear the response that I got about this when I asked, "Is this good? Is 62% high, is 62% low. One person told me, well, compared to this earlier effort that focused on a 100 psychology papers that I believe came out in 2015, that was 30, 39%, 30 something percent that they were able to replicate. So when you think about that. This is a much more encouraging number, then again, it also suggests that a large number of these papers don't yield the same results when you try to do the exact same method again, so it still suggest there is room for improvement. 04:12 SC: That earlier study focused on research published in a specialty journal, and this... One of the difference is with this one was it looked at general journals. Why did they do that? 04:21 KS: Yep. The psychology project was three journals. That specialize in psychology and this is a look at much broader, much more general journals. I think they were trying to address this question of weather journals that are looking for groundbreaking, eye catching positive results are gonna have a better or a worse track record when it comes to replicability. I don't think this study can be the final word on that. It's really hard to compare across different replication projects' environment. 04:47 SC: Yeah, and this is a very small number. 04:49 KS: It is. 04:50 SC: And then there's also an economics replication project that looked at and got about the same number 61%, but can we take any of these numbers and say, "This is how this field is doing." 10/29/18 Page 3 of 12

3 Or, "This is how science is doing." Or, "This is how journals are doing."? 05:04 KS: I think the people behind this project would say that you really can't condemn a field or compare one field to another based on these numbers, but they do sort of draw attention to this question of; how are we gonna make these studies more repeatable? Because it is making evident all kinds of problems in the system. 05:20 SC: Yeah. I mean, is that what their goal is with this kind of work? Now that we know that this is happening, are they proposing solutions or are they gonna just keep going with their replication efforts? 05:29 KS: In fact, the Center for Open Science is also pushing a system where people pre-register the designs of their study, so they have this sort of platform or these set of tools where people can share data and establish what they're gonna do, how they're gonna analyze their results before they start the experiment, both to keep people from changing that methodology to get a positive result, and also to make people aware even if a study never gets published, it may be because that was a negative result. And you know that because the design for the study is out there. 06:00 SC: And what about this other component of the study that was published this week, which was to get people to guess whether they thought a study would replicate or bet whether they thought a study would replicate. Is that a possible option because they were basically as successful as the replicators? 06:16 KS: Yeah. It's interesting, this approach has been applied a couple of times in the psychology and then economics replication efforts as well, and the result was the same. If you ask people to just read the paper and decide how likely you think it is to replicate, or read the paper and then sort of participate in this market where you can trade stocks that represent different papers. Where their value depends on whether or not they end up replicating, people end up getting really close. And the dividing line between predicted to replicate and predicted not to replicate, falls right about where it should. 06:47 SC: Did they guess the right papers or did they just get the proportion right? 06:49 KS: It got really close. So if you go by the value of the stock of each paper, I think the cutoff was around 54 cents and it should have been 50 cents if it was right in the middle, like a couple of papers fell on the wrong side of the line, but there was a really strong correlation to the likelihood. 07:07 SC: So what does that mean? Obviously, people are closely reading these papers. Are they just not thinking about the right things, like is this replicatable or are they looking at other parts of the paper? Peer review doesn't appear to be working. 07:19 KS: Yeah, it's something that I think a lot of people want to answer in more detail. In this study, they didn't systematically poll people on: What are you picking up on? What inspires your decision here? But sort of anecdotally, we know that people are interested in sample size or the 10/29/18 Page 4 of 12

4 statistical power of the study, and there are a little more dubious of studies that have a really surprising or a counter-intuitive result. So these are things that we know, but that people are now saying, if we can figure out the weight of these things in predicting replicability, maybe there could be some sort of rubric that a peer reviewer could look through and evaluate and just keep irreproducible studies out of the literature early in the process. 07:57 SC: What are people in the field think about this? Do they think this is worth doing? That there needs to be some rectification here so that people trust this material more as it comes out of journals? 08:08 KS: I think there is an ongoing years long effort at this point to recognize problems with replicability and doom certain things to screen the literature for this issue. And I think the question that people have now is the report card, is the track record from these replication projects ever gonna reflect that. And this study... Some of these papers are from 2010, so that sort of pre-dates this push, but that's sort of the next big question that I think the people behind this replication effort wanna answer. Is if you look later in the literature are the journals that are being released now doing better. 08:45 SC: The place where this was published, which was what? Nature human behavior. Yeah. They published alongside this research paper, this replication study, some comments from authors of the original studies, what was their reaction to those results, say if their study was overturned or validated. 09:01 KS: Yeah, I think that several of them pointed out the limitation of doing a project like this, which is that they were trying to repeat one experiment from papers that sometimes had several experiments in them. So it's not like a failure to replicate is over turning the central hypothesis that's in a paper. A lot authors were pointing that out or pointing out little differences in methodology that might account for the differences in results. 09:26 SC: Even though these researchers were often involved in the replication effort. 09:29 KS: In almost all of the cases they over saw the way that the replication effort was designed. 09:35 SC: Yeah. 09:35 KS: And one thing that Brian Nosek, of the Center for Open Science and the University of Virginia who's a part of this project, told me, is that compared to the earlier efforts, people reacted much more positively, didn't seem to be as threatened by this replication and were more encouraged that the field was taking a second look at their work. 09:54 SC: Thank you so much, Kelly. 09:55 KS: Sure, thank you. 09:56 SC: Kelly Servick is a staff writer for science, you can find a link to her story and the related 10/29/18 Page 5 of 12

5 research at sciencemag.org/podcasts. Stay tuned for an interview with Emily Brodsky. She talks with Meagan Cantwell, about how far away fracking induced earthquakes might show up from their source at the fracking site. [music] 10:19 SC: Now we have, Meagan Cantwell, our new podcast host, talking with Emily Brodsky, about her paper on fracking and earthquakes. 10:27 Meagan Cantwell: This week we have Emily Brodsky, a co-author on a research report exploring the mechanisms behind earthquakes that occur as far away as 10 kilometers from injection wells. Hi Emily. 10:38 Emily Brodsky: Hello. 10:39 MC: Humans have been injecting fluids into the earth for a variety of reasons, ranging from hydraulic fracturing to waste water disposal. And this is leading to earthquake activity. When and how did geologist establish this link between fluid injection and earthquakes? 10:55 EB: People have known that humans can generate earthquakes by injecting waste water for almost 50 years. There was a famous experiment in the early 1970s, where the US Geological Survey, turned on and off earthquakes on purpose in a place called Rangely, Colorado, and it was quite clear that when the pressure in wells went up, the earthquakes went up, and when the pressure went down, the earthquakes went down. 11:23 MC: Right. So a pretty easy causal link there. Before we dive into the processes behind these earthquakes. How big and frequent are they? 11:30 EB: Well, they're getting more frequent all the time, as people have discovered new technologies to extract oil and natural gas from the earth. A lot of those technologies have a lot of waste water by-products, currently Oklahoma has quite a lot of earthquakes, I think a lot of people are aware of that, but it's actually throughout the Central and Eastern US. The biggest in Oklahoma, so far, was a 5.8. There are other earthquakes elsewhere within our world, as large magnitude 7s that have been associated with human activity. 12:02 MC: So the previous research has predominantly focused on one driver of these earthquakes, but that only explains the earthquakes that occur right near the injection site. So could you explain the previous mechanism that has been thoroughly researched? 12:15 EB: Well, the best understood mechanism for making earthquakes is that if you inject water straight into the fault, it jacks the fault open and makes it easier for it to slide, and this direct pressure effect has been thought to be the dominant mechanism of human-generated earthquakes from waste water injection. More recently, there's been a thread of research from a number of people that have noted that there's this other effect. When you inject water into a well, you also push on the surrounding rocks, and those rocks are not rigid, those rocks can compress, they'd get 10/29/18 Page 6 of 12

6 squeezed, and that stress can be transmitted elastically to great distances, what we showed in this study, is that that mechanism is actually dominant in many cases. 13:07 MC: So how do these mechanisms result in earthquakes of different magnitudes. So the direct pressure versus the mechanism that you researched? 13:15 EB: So the mechanism we researched which is called poor elasticity, actually, we didn't research any particular mechanism, what we were doing is just empirically looking at data and the mechanisms kind of fell out. The ways that each of these mechanisms makes earthquakes is that it depends on the size of the fault and the elastic energy already available in the earth. If a waste water injection site happens to intersect directly a large fault that might happen to make a large earthquake, if the elastic stresses in the earth, the poor elasticity affects a large region, and that large region happens to contain a large fault, that could make a large earthquake. So there's a certain amount of chance involved. 14:04 MC: Could you talk a little bit about the methods you use to determine that the earthquakes were induced by injection wells? You talked about the large data set that you analyzed. Of that data set, how did you know for sure that those were induced by injection wells? 14:17 EB: Well, we specifically picked a data set where there had been a lot of previously study on these particular sites. And we looked for sites that were very well isolated from any other sites, that they were separated out. So we did not do Oklahoma, we did places where there might be a scientific drilling well, and the timing was immediate after the injection, so there was very little ambiguity. 14:48 MC: What do you think is driving the differences in these processes? Is it more the method of injection, such as waste water or fracking, or is it the type of rock that the fluid is being injected into? 15:00 EB: It's the type of rock. It's very clear that the ones that are more spread out, which are the ones that we associate with poor elasticity rather than direct pressure, those are the sites where the water is being injected into the sedimentary rocks, and the ones where it's getting injected into the basement, are the more compact clouds with the... In general, smaller earthquakes for a given volume of the water injected. 15:25 MC: How does understanding these additional drivers of earthquakes better inform us about mitigation and hazard assessment in the future? 15:33 EB: Well, it's a bit of a word of warning, that we should develop mitigation and plans that include great distances up from wells. We can see induced earthquakes 10 kilometers away from a well, so that means that you need to be thinking maybe not just about this city but the next city, when you develop a well. So there's that aspect to it. And then there's this aspect that, in general, the standard advice to operators had been to avoid basement injection and preferentially if you have to inject waste water, which is always a risky operation, but if you have to do it, do it into the sedimentary rocks, and it's not clear empirically that that's necessarily the right advice in all cases. 10/29/18 Page 7 of 12

7 In fact, if you just look at the data, it seems to be the opposite. The basement injections are making smaller more confined earthquakes than the sedimentary injections, so we really need to think through that advice more carefully. 16:38 MC: Has there been any action to kind of shift this injection toward more crystalline basement units? 16:45 EB: I think we would want to think carefully about changing our actual practice quite so radically, I would hope there would be more than one study that would... [chuckle] 16:56 MC: Yeah. 16:57 EB: Before we take that on our shoulders. But I think the issue is that injecting into the basement, that is where the bigger faults are. So if you are so unlucky as to hit a big fault, you can get fairly large things to happen. It's a little bit of a nuanced piece of advice, that injecting arbitrarily some place on planet earth into the basement is probably a better idea that injecting arbitrarily some place on planet Earth into the sedimentary rocks. However, if you have some geological knowledge about the basement and happen to know there's a fault there, that's not a particularly good idea either. 17:35 MC: That would definitely make sense. After this research, what else do you wanna explore? What do you think wasn't addressed within this paper? 17:43 EB: Well, I think we're dealing with one kind of injection site... Way that people make earthquakes, and there are lots of other ways that people make earthquakes, people can make earthquakes by pulling fluid out of the ground, extraction oil fields, people can make earthquakes with reservoirs, and I think we can apply a lot of the same empirical, examination of the data to those sorts of scenarios and try to expand our understanding, in general of the suite of mechanisms that make earthquakes. 18:15 MC: Alright, great, thank you, Emily. 18:16 EB: Thank you. 18:17 MC: Emily Brodsky is a Professor of Earth and Planetary Science at the University of California, Santa Cruz. She and her co-author write about induced seismicity in this week's issue of Science, you can find a link to their article at sciencemag.org/podcasts. 18:32 SC: Stay tuned for our monthly book segment. When Jen Golbeck, ask the question, why can't we answer why questions? She interviews the authors of The book of Why: The new science of cause and effect. [music] 10/29/18 Page 8 of 12

8 18:51 Jen Golbeck: Hi everyone. I'm Jen Golbeck, and welcome to this month's edition of the book segment of the podcast. This month, we're reading The Book of Why: The new science of cause and effect, by Judea pearl and Dana Mackenzie. Anyone who's taken a statistics class has probably had the mantra; correlation does not imply causation, burned into their minds. And anyone who interacts with science regularly, knows that claiming you've discovered that one thing causes another is met with a lot of skepticism. So much that many scientists avoid it. The foundation for this is a reasonable one, to claim A causes B, you need to know that it's not really C causing both A and B. This was a big issue in the science of smoking and lung cancer, an example discussed in a full chapter of The Book of Why. 19:33 JG: We could observe a strong correlation between smoking and cancer, but that didn't necessarily mean smoking cause cancer. Maybe smokers had a gene that caused the cancer, and also made it more likely that they would be attracted to smoking and becoming addicted to tobacco. Maybe people who got cancer lived in environments where smoking was more common, but the environment not the smoking caused the cancer. Scientist spent years looking at these factors, which we call confounding variables, in proving the causal link. But is this a version to claiming causation correct? Judea Pearl has spent the last couple decades working on the science of causation and establishing a framework for making causal claims. The core ideas that scientists can communicate their common sense assumptions and then state subject to those assumptions, that they've observed a causal relationship. In a very technical book called Causality, published in 2000, Pearl laid out that science. The Book of Why, tries to bring these ideas to an audience interested in the science but who may lack the mathematical background needed to get through that earlier work. 20:31 JG: Today, I'm joined by the authors, Judea Pearl and Dana Mackenzie, to talk about The Book of Why. Thank to you both for joining us. Judea, I'd like to start with you. Can you talk about how the ideas here fit into modern statistics and machine learning, the branch of artificial intelligence that deals with statistical modeling and learning? 20:49 JP: It fits very nicely with the history of statistics, as you mentioned earlier, causality became taboo, in machine learning it doesn't exist. The hope is let's get as much as we can from the data. Not realizing that there are scientific theoretical limitation to what you can get from data. What you can get from data is what statisticians have gotten from data so far, namely correlation. And this awareness, that there are theoretical limitation that you can not do certain function, both impractical and cognitive functions, you cannot do by just looking at the data, you got to have a model of the environment, of the world outside the data, in order to interpret the data. 21:39 JG: Dana MacKenzie, can you give us an example to show why we need this new science of causality? Why isn't correlation enough? 21:46 Dana MacKenzie: There's an example that I really love, it was taken from a medical journal in 1998, of a study that was done on the effects of exercise, of regular walking, on health in older men. They basically found that the men who walked a great deal, more than two miles a day, had twice as low mortality rate as the men who were infrequent walkers, who walked less than a mile day. Now there are lots of honest-full explanations for this that don't involve walking causing better outcomes, for example, it could be that the people who walked two miles a day did it because they 10/29/18 Page 9 of 12

9 could, and when they adjusted for the confounders such as age and other health variables, they still found a strong correlation between walking more and having lower mortality. 22:37 DM: But in the paper, they did not speculate about causation, and in fact, they explicitly said; We cannot talk about the effect of an intervention to walk more. And to me, this is a tragic opportunity wasted, because the whole point of the study is to understand causation. When I read this study I wanna know; is walking more going to help me live longer? And yet they avoid answering that question. And the reason is completely because of the taboo in statistics against talking about causation. And so what we argue in the book, is that if you have identified all of the reasonable confounders, you can draw these in a diagram, if you want, and you can say subject to the assumptions in this diagram, we conclude that walking will improve your life span or walking will reduce your mortality. And they really owe this to the public, to say something like that, because that's why the public is interested. That's why the researchers wanted to study walking in the first place. 23:39 JG: But Judea, I can see the hesitation on the part of these scientists. They did the work, they controlled for every confounding factor they could think of, but it's entirely possible they missed one. There could be some totally unknown science that we're unaware of, and in 10 years that will be discovered and explain this result to show that it wasn't the walking that caused better health outcomes. That hesitation is there because we don't wanna make incorrect claims. 24:01 JP: Yeah. This hesitation is well taken, and this hesitation is a product I believe of their culture, it had reasons to suspect and to hesitate, the reason was they couldn't express their assumption in a communicable way and now we are offering them a method of communicating and making those assumptions clear and let them advertise their existence. The assumptions themselves. It's like communicating with arguments. When people sit and have a discussion about a certain decision. People don't talk in numbers. People talk in terms of strength of argument. I like your argument and I like yours. And that is exactly how science should progress, with arguments, rather than with numbers. 24:49 JG: And Dana, the way that the book explains these assumptions, is communicating them through causal diagrams. Basically, you have a dot for each thing you consider and an arrow pointing to what it may cause. 25:00 DM: Well, when you pick up the book, I think the first thing you'll notice is lots and lots of diagrams, and these simple dot And arrow diagrams communicate our assumptions about which variables are likely to affect... Which variables may affect which other variables, these diagrams communicate qualitative assumptions, and they're usually extremely reasonable. They're often intuitively obvious that one thing could cause another thing or one thing could not. Now, for example, a traffic jam in Taiwan cannot affect the price of beans in Brazil. So, we just intuitively understand that it's too impossible for that to happen. So what the diagrams do is let us communicate our common sense. And Judea likes to say that this book is causing an outbreak of common sense in the world of science. 25:50 JG: Well, if you're interested in how you can integrate Common Sense assumptions into statistical models of the world to claim causation, the Book of Why, will help you take a deep dive 10/29/18 Page 10 of 12

10 into how it can be done when we do it. And how we can use the results to make predictions about the future and to explore alternative outcomes. Thanks, Judea Pearl and Dana MacKenzie for joining me to discuss it. And that's it for August. I hope you all made a dent in your summer reading list. We'll be back next month to add to your stack for fall. And in the meantime, if you have comments, questions or suggestions, we'd love to hear from you on the books blog, Books Et Al, on the Science Magazine website. 26:24 SC: Just a reminder that this week show was brought to you in part by the podcast, American innovations. If you like the Science Podcast if you enjoy learning about current science, you should also think about listening to a podcast about the history of science. The show features Popular Science author, Steven Johnson, and tells of the science behind some of the greatest innovations of the last century, as well, as the people and places that leads to these aha moments. The newest episode is all about the early development of AI and how scientists were able to put human intelligence into a machine. This kind of podcast will help you put what we talk about on the Science Podcast which is very up-to-date, very newsy in a context of the greater art of scientific progress. So please check it out, subscribe to American innovations wherever you're listening to this podcast. 27:15 SC: And that concludes this edition of The Science Podcast. If you have any comments or suggestions for the show, right to We didn't hear from anyone with books questions, so let me know if I should ask about it for next time. You can subscribe to the show on itunes, Stitcher and many other places, or you can listen to us on the Science website, there you will also find links to the research and news stories discussed in the episode. That's Sciencemag.org/podcasts. This show was produced by Sarah Crespi and Meagan Cantwell, and edited by Podigy. Jeffrey Cook composed the music. On behalf of Science Magazine and its publisher AAAS. Thanks for joining us. 10/29/18 Page 11 of 12