Speaking of Psychology: How “open science” is changing psychological research, with Brian Nosek, PhD

Is psychology research in a crisis or a renaissance? Over the past decade, scientists have realized that many published research results, including some classic findings in psychology, don’t always hold up to repeat trials. Brian Nosek, PhD, of the Center for Open Science, discusses how psychologists are leading a movement to address that problem, in psychology and in other scientific fields, by changing the way that research studies get funded, conducted, and published.

Brian Nosek, PhD, is the founder and executive director of the Center for Open Science in Charlottesville, Virginia, a non-profit organization that aims to increase the reproducibility of research by building tools to encourage scientists to share data and conduct their work more transparently. He is also a professor of psychology at the University of Virginia. Much of his lab research has focused on implicit cognition, or thoughts and feelings that occur outside of our conscious control. Broadly, he's interested in the gap between values and practices–between what people intend to do and what they actually do.

Kim Mills: We talk a lot on this podcast about the fascinating research conducted by psychological scientist. Today we're going to do something a little different. We're going to explore how that research gets done and how it might be done better. Over the past decade there's been discussion of what critics have called psychology's reproducibility crisis. The fact that some published research results, including some classic findings in psychology, don't always hold up when other researchers reproduce those experiments. Now, psychologists are leading a movement to address that problem by changing the way that research studies get conducted and published.

So what exactly is reproducibility and why is it important? If the goal of science is to increase our understanding of the world, then is the way science is conducted and scrutinized right now helping researchers achieve that goal? And how much can we as readers and consumers of science information and news trust published research results?

Welcome to Speaking of Psychology, the flagship podcast of the American Psychological Association that examines the links between psychological science and everyday life. I'm Kim Mills. Our guest today is Dr. Brian Nosek, the founder and executive director of the Center for Open Science in Charlottesville, Virginia. The Center for Open Science is a non-profit organization that aims to increase the reproducibility of research by building tools to encourage scientists to share data and conduct their work more transparently. Dr. Nosek is also a professor of psychology at the University of Virginia. Much of his own lab research has focused on implicit cognition, or thoughts and feelings that occur outside of our conscious control. Broadly, he's interested in the gap between values and practices, between what people intend to do and what they actually do. 

Thank you for joining us today, Dr. Nosek.

Brian Nosek, PhD: I'm delighted to be here today. Thank you for having me.

Mills: Let's start with how you became interested in the reproducibility of scientific research. I know there were series of events around 10 years ago that spurred a heightened scrutiny of psychological research and led to a lot of the work you're doing now. I'm thinking in particular of cases such as psychologist Diederik Stapel and Daryl Bem. Can you tell us what happened?

Nosek: Yeah. In the early part of the 2010s, there were a number of eliciting events that brought to fore some issues that have been known for decades but never really had an impact on daily practice. One was the emergence of a massive amount of fraud perpetrated by Diederik Stapel that came to light because of whistleblowers, his own collaborators and students that had worked with him, starting to be suspicious about origins of the data that were behind some of the papers. But, in the end more than 50 papers of his were identified and then ultimately retracted from the literature as being illegitimate in some way, largely invented data. The shocking part of it was that many of these papers were high profile results, results that people knew about, that people were citing and using to inform their own research. And yet, no one over the decades of those papers being written and published, no one had identified any of them as potentially fraudulent, leading to people to say, “Oh my gosh, how did we not know this? How did we not recognize?” and building on that work that actually there was very little basis for at least no actual evidence. Maybe they were true but by chance.

The second event was Daryl Bem, a renowned social psychologist who's done wonderful work throughout his career, got interested later in his career in the potential for ESP and other paranormal types of parapsychological types of things as possibly being true. So he started to study them and he published a paper in one of APA's flagship journals, the Journal of Personality and Social Psychology, showing evidence across multiple experiments of ESP. This created an outcry reaction among psychologists because they said, “How could a leading journal in our field publish something that is so obviously flawed, it can't be true.” Then others on the other side saying, “Well, look at what he reported, it looks like every other paper that we publish in this journal. Follows the same standards of evidence, in fact, it does so in a really remarkably great way compared to many of the papers that are published in this journal.” So, what it created was this tension of either we need to now believe ESP is true because the paper follows all of our rules for what it takes to provide evidence, or we need to question our rules because they allowed something that seems to be a fantastical conclusion to come to light.

The third event that comes with these two, around the same time, was a publication of a paper called False-Positive Psychology by Leif Nelson, Joe Simmons and Uri Simonsohn where they essentially consolidated this many decades of insight of ways that we might create false-positive results, results that aren't actually true, through our practices of how we get our data and how we analyze our data. It's a rhetorically brilliant paper because what it does is, it gives the reader who has some experience with analyzing their own data a sense of, here's something that you might do, which were very common kinds of behaviors. And here's the implications that has for increasing the likelihood that your findings are no longer credible. Just through a sequence of examples, they show that what would ostensibly be something that might create five percent false-positives, seeing signal where there isn't anything, might actually end up being 60% or higher false-positive rates. “Whoa, most of my findings are not actually there because of how I analyze my data?” Each of these created shock waves in its own way of “something is not right here and we need to do something about it.”

Mills: Shouldn't the peer review process prevent such flawed research from being published? And sometimes if a study's conclusion are too convenient or too good to be true, aren't they accorded a more stringent level of scrutiny?

Nosek: It would be great if that were true. And the challenge with why it isn't true are a couple of fold. One, peer review is hard. I am reading what you did in your research and I'm trying to understand it, trying to recognize limitations of it, trying to say, “Oh, here's some things that you might trust and might not trust.” So, just that work itself is hard, because you had to write down what you did, I have to interpret what you did and I have to come to some understanding in the process.

Let's say I'm really good at it as a reviewer, it's also really hard because it isn't transparent. All I have to review of what you did is what you wrote down, so taking that example of these many different analyses that you might do and just report the ones that make the finding come out but actually increase its false-positive rate, if you don't show me all those other analyses that you did for me to say, “Hang on a second, you analyzed this 50 ways, you only showed me the one way, what happened to the other 49 ways?” If I don't see that, there's no way for me to catch it in peer review.

Mills: Right. I didn't like the way those turned out.

Nosek: Yeah. Right. In the Diederik Stapel case, well, he knows how to write a paper. How am I to know that the data aren't there, aren't real, if I can't access the data? So, one of the big limitations of the peer review process, besides not being perfect and that's just part of the system, no system like that will be perfect because science is hard and it takes a while to figure things out. But, one of the big barriers for effective peer review and ultimately self-correction is a lack of transparency. If you can't see what I did, how I did, can't see the data that I generated, then there's very little chance that you can effectively evaluate the quality of my work.

Mills: So, your organization is called the Center for Open Science and I'm hoping you can explain for our listeners, many whom are not psychologists, in fact most of whom are not psychologists, what you mean by that, how does it relate to the reproducibility problem that we're talking about right now?

Nosek: The Center for Open Science is a non-profit technology and culture change organization that has a mission to increase openness, integrity and reproducibility of research. It spun out of my lab in 2013. Jeff Spees and I launched the organization. Our goal was to recognize that this is a culture challenge. This isn't a matter of just teaching people how to do research in a better way, having them read Uri, Leif and Joe's paper and say, “Okay, don't do those things and your research will be fine.” That's helpful. We need education and training about how to do the best research we can, but really as a researcher in the field, I have two major constraints that make it so that the work that I do might be less credible than I want it to be.

One is the limitations of my own mind. I intend to do good work. I intend to find out things that are true, but I have skin in the game. If I find something super exciting, that advances my career. I'm more likely to get a job, I'm more likely to get tenure at my institution, I'm more likely to get awards. And if I find things that don't work a lot and things that are kind of messy, well, that's not going to do as much for my career. I have a conflict of interest with my own results, so I many bring to bear lots of, not reasons, but rationalizations for how it is I can justify reporting this finding rather than that finding. Why it is I would take these two studies that looked really good for my phenomenon and decide that those four studies that didn't find evidence for it were actually just bad studies so I shouldn't report them at all. The bias that I bring to the table I may not recognize in how it is I do my work, so we have that constraint. How do we help ourselves be more credible by being more transparent and confront our own biases. Just like we study biases about how we make judgments of other people in everyday life, implicit bias. Same sorts of principles about scientific evidence.

Second major constraint is the structure of the system, how it is that I'm rewarded. Because I need to get those exciting findings, that constraint, that part of the structure of the system, what universities do to decide who to hire and who to promote. What journals do to decide which findings to publish, what funders do to decide who to give grants to. Those create a system that constrains my behavior. As much as I want to do things well, in order to be in that system I have to align to some degree to those reward systems.

What the Center for Open Science exists to do is to confront both of these and try to facilitate a change in the research culture, and it does so by providing technology for researchers to make it easy to be more transparent. To show you what my plans were in advance, to show all of the things that I found, not putting anything, hiding anything away. To making my data available to you to be analyzed. And, we work on the structure, working with publishers and funders and institutions to try to nudge those policies and incentives so that what researchers are rewarded for is doing rigorous transparent research rather than producing exciting findings that may not be credible.

Mills: Your center encourages several things, open data, open materials and preregistration of studies. What are each of those practices and how are they helping with science being more credible and more transparent?

Nosek: With open data, if you want to be able to analyze my data to see if I found what I said I found, simple reproduction, you can't do that unless you have access to my data. If you'd like to see whether my findings are robust, if you change analysis this way, this analysis that way, all different reasonable choices you might make and see if the same findings occur, you need access to the data. Or, if you want to combine the data I generated with similar kinds of data by many other researchers to see how robust is the phenomenon when you look in Virginia where I am and participants in Iowa and participants in Israel and participants in Russia, then you need to be able to aggregate that data. You can do some of that net analytically with just the outcomes, but it's even more powerful to be able to look at the raw data aggregate. That open data is part of where we started earlier when you asked about the peer review process is, if we want to be able to self-correct effectively and assess credibility, having access to the original data is very useful.

For open materials, you want to see how is it that I generated that data. You want to know what was the exact format of the survey or the behavioral intervention that I did. Or what it is that the people did in the setting when I was trying to elicit their behavior. If you don't have access to all of the actual content of the study, how the study was conducted, it's hard for you to then assess the credibility of the data. It's also hard for you to say, “You know what, I want to know if that replicates in this other context. Sure he showed it in his lab under these very limited conditions, but really if that's meaningful evidence it should also occur in a corporate environment, or it should occur in everyday life in some way, so I want to test it over in this population or that population.” Not having access to materials makes it harder for you to assess how it is I did it and then to redo it, so sharing that, again, provides better effective self-correction extension and building on each other's evidence.

The last one, preregistration, has two important roles. What preregistration means is when I plan to do a study I write down some information of what the study's going to be and how I'm going to analyze the data and I post that in an independent repository. In principle, anyone can discover all of the studies that I plan to do. The two problems that it solves is, first, the publication bias. We talked about earlier where I do 10 studies and I only show you the three that worked. Well, now if I've preregistered all of my studies in advance, you can find the other seven. You can say, “Oh, wait a second, Brian, is the evidence as credible as you say it is? What happened to these?” I say, “Oh, no, no, no, those were bad studies.” And you might agree or you might disagree, but at least you can see the evidence.

The other reason for preregistration is that it makes very clear what was planned beforehand and what I discovered after the fact. Both of those modes of research are super important. We have confirmatory modes of research where it is I have a hypothesis in advance, this theory makes these kinds of predictions and so I'm going to collect some data to test the viability of my theory. See where it holds up, see where it's weak and then revise our confidence in the theory as a consequence. That confirmatory part, it's really important to say in advance “How are you going to test it?” So then afterwards I don't say, “Oh, actually my prediction's a little bit different and I think it has a few holes here and holes there.” So, I don't do that sort of reconstruction after the fact.

The other part of that is that the evidence itself becomes easier to assess because then once I've seen the data, I am free to go into it and explore and discover new things and find out what's in the data that might shape new ideas, might generate hypotheses for me to test next. That's where a lot of the new discovery in science occurs. It's not from what we're testing from existing hypotheses or theories, it's when the data go totally awry and sends us in a whole new direction, “Wow, I didn't think that was going to happen.” That is super important, but it's also very important to recognize that it's more uncertain. When data go in a different direction that what we expected, what we would ordinarily want to do is follow up on it, develop some hypotheses and test those hypotheses for why it went into that new direction. So preregistration helps keep clear when we're in these two different modes of research so that we can respect and value both of them.

Mills: If I preregister a study and I say I'm going to conduct these 10 experiments as part of my study, does this commit me to using all of them or maybe I find that six out of the ten confirm the hypothesis and the other four don't, but I've told everybody that I did it. Can I still submit the paper and leave out those four? How committed am I to including everything?

Nosek: Preregistration doesn't force you to do anything at all. It just makes it so that it's transparent what it is that you did. This happens all the time. I plan a number of studies, I preregister them in our lab and then a couple of them go sideways. Even just implementing the study, I had a plan for how we were going to do it and I can't do it that way, so it is a broken study. But, many times it's more like what you described, it just doesn't turn out the way that I thought. I can say, “Well, these are the ones that I'm going to report,” but the value of preregistration is in, those other ones are also available. They're also accessible for you as a reader or anyone in the community to say, “Well, I want to consider that evidence too.” The goal of preregistration is not to say, “Here are these strict rules that you now have to follow.” It's, “Here is a way to articulate and make plain what your plans were and compare that to what you did.”

I can tell you why I had to make those changes. We planned to do this study with 1,000 six month olds, to measure their attention towards this versus that. Get into the study, it turns out it's hard to do research with six month olds. We thought 1,000, that was crazy. We're going to do it with 20, whatever it is. At least I can then follow up and say, “Here's what we had to change, this is why we had to change it and this is what we think implications it has for how confident you should be in our results.” Then you as a reader gets to say, “I'm glad, I can see that. I'm glad I can understand how it is he got from here to there and I either agree with his explanations and the reasons that he changed or I disagree.” But all of that is enabled by the transparency that preregistration provides. It does not replace reasoning in any way.

Mills: These are pretty radical changes to the way that research is being done. I'm just wondering, are these practices catching on? Are researchers agreeing, going along and saying, “Yeah, this is what we should have been doing all along. I'm all in.”

Nosek: Yeah. You opened saying that the critics have called this a reproducibility crisis and I think the term is wrong. There are many others that have said this as well, that's really it's a renaissance. Leif, Joe and Uri, the ones that did that initial paper, they have a great paper recently called The Psychology Renaissance, and it's a very optimistic paper in that it argues that what these troubling episodes produced was self-reflection in the psychological community about psychological processes, about how humans and the cultures they inhabit are themselves influencing, sometimes for the negative but also potentially for the positive, the quality of science and how we produce and understand findings. As a consequence of that, there's been a very broad engagement in the psychological community by bottom up, researchers themselves saying, “We need to change this, so we're going to start to adopt some of these new behaviors, sharing data, preregistering research and otherwise.” And some top down where publishers and editors, funders and institutions are saying, “We need to change our reward systems, how it is that we encourage the most credible research that we can.”

Now, it's 10 years since some of those elicited events occurred that brought this really to the fore and culture change takes a long time, but a lot has changed. Just as one example, we operate the Open Science Framework, this is part of what the Center for Open Science does, is this technology where researchers can use this as a collaborative management service, the OSF, where you can share your data and your materials. You can preregister your designs, et cetera. The number of users of the OSF, which is open to any discipline, not just psychology, but is more than 300,000 now since it opened in 2012. It's been increasing in exponential rates every year. The number of registered studies is in the many, many thousands at this point. Millions of data files have been shared on the OSF. All of these show this exponential growth and that is largely driven by the psychological community. I think one of the great benefits of psychology confronting this is that really the challenges that are trying to be solved are psychological challenges. Challenges with the individual and their behavior, challenges with the system and how individuals react to that system.

As a consequence of psychology leading the way on a lot of this work, there is a lot of similar sorts of self-reflection and change happening in other disciplines. The reproducibility movement has spanned into biomedicine, into ecology and evolutionary biology, into preclinical disease research, into education research, business research. That, I think, is going to be the next, the next 10 years is really the full embrace of these challenges across different scholarly communities.

Mills: So, replicating experiments that have already been conducted isn't seen as a career enhancing move in the sciences. What, if anything, is being done in an attempt to change this, to encourage people to try to reproduce studies?

Nosek: Yeah. Replication is an important part of science because it's how we establish the reliability of findings. We rarely do a study to make a statement about a historical fact, like did this thing happen or not? That isn't usually what a scientific hypothesis or claim would be. Often scientific claims are about, if these conditions occur again, then this kind of thing will happen again sometime in the future. We're making statements that are general application, make predictions about observations that haven't been made yet. Even if they're about events in the past, we haven't observed them yet, the hypothesis would make an anticipation of what's going to occur. So, replication helps with assessing whether those predictions come true when you try to do them again in similar sorts of context, as the previous research.

But it is entirely devalued because of those reward systems that we've been talking about, focus on innovation and novelty at the expense of verification and credibility. I gain more stature by having a phenomenon with my name on it, the Nosek Phenomenon of X. Whereas the replication is me trying to assess the credibility of somebody else's claim usually, could be my own too. But isn't as valued.

There are some things that are changing. One is just a growing appreciation of the role of replication in research. Journals are changing their policies to be more welcoming of replication studies, recognizing that it is an important contribution to science, that opens the door. It is becoming easier to do replications because researchers are sharing their materials and their data to a greater extent. Now if someone wants to replicate someone else's findings, they can get a leg up on figuring out how they actually did their research more quickly and more easily because of greater sharing.

And now, there are also new publishing models that lower the barrier to trying replication studies. One of them is called registered reports and the idea of registered reports is to really fundamentally change how is it that researchers get that key reward, the publication. In the standard model, I do all of my research and then I write it up and I submit it at the end, here are all my findings, and then the peer reviewers evaluate whether it's credible and whether it should be published. And in registered reports, what I submit to the journal is my plan. Here's the question that I'm going to ask, here's the background material, here's some initial evidence to say that this is worth studying and here's my study design and how I plan to analyze my data when I get it. The peer reviewers evaluate, is that an important question and is the methodology an effective test of that question? If they agree, then the journal commits to publishing my findings regardless of what happens. This fundamentally changes what my goal is as the author. Instead of my goal being make the findings as flashy, as exciting and as innovative as possible, my goal is, ask the most important questions and design the best methods I can to test them and that will get me the publication.

The results are the results, I do the results and I publish whatever happens and that changes a lot of the things we've been talking about. It eliminates publication bias because the studies are going to get out there whether they come the way I hoped they would or not. They're all going to be reported because they've been pre-committed in advance. They're going to get reviewed in advance, so the rigor can be enhanced compared to reviewing it afterward where you say, “Oh, well, you screwed up X, Y and Z.” And I say, “Great, well, the studies are all done. I can't change that now.”

The relevance to replication which is where you started in asking is that it also proposes to do replication studies and get commitments from the journal that they're actually interested in publishing it before I do all of the work to do the replication. And that's a lower barrier entry. I can say, fine, I don't know if they will want the replication but at least I can just write up a proposal for it. And then if they don't want it I don't have to do it, save myself some time. All of those are shifting some rewards for replication but then also overall how it is people are rewarded in general.

Mills: Are some of these open science ideas filtering down to training yet? Are undergraduates and graduate students being taught these new methods?

Nosek: Yes, very much so. That's one of the more exciting elements is that's really how these sorts of things get sustained, is where the training that gets integrated into curricula at the graduate and undergraduate levels really starts to embed these new practices. And that's happening in formal cases like methodology instructors at their universities building them into their course work. It's also happening in informal ways where people who are advocates for open science in the community, it's a very robust lively community, are developing materials and making them available for broad dissemination.

For example, there is a society called The Society for the Improvement of Psychological Science, SIPS, it's improvingpsych.org and the goal of that is to be a very much of a community-based groundswell of ways that we can do better. That might be creating training materials, it might be delivering training, it might be ways to try to change some of these reward systems or anything else. That community has been fabulous at generating lots and lots of different new training materials or education devices or ways of explaining some of these challenges. There are projects like the Fort Project, or the open scholarship knowledge base, that aggregate these materials and then make them more easily available for people to search and discover, bring into their class or to learn themselves. That's really a big part of the engine of what is helping to sustain this change, is the community providing training for itself about these issues.

Mills: Another fascinating thing that you're exploring at the center is faster ways to predict which research results are likely to replicate and which aren't. I know you've looked at something called prediction markets and also at using artificial intelligence to predict reproducibility, can you talk about those tools and how they work?

Nosek: Doing a replication is a lot of work. If we want to know whether a finding is trustworthy or credible, saying, “Okay, we've got to run it again and get another set of participants and spend another year running the study and analyzing it.” Well, that's a lot of effort to assess credibility and that's essential and important and will be continued to be a part of the process for as long as we can envision what science will be, but it would be useful to have other indicators that gave us a more rapid assessment of the credibility of the finding. So peer review is supposed to provide that, but that still is a lot of effort. The reader has to read it all and decide do they think it's a credible finding or not. Wouldn't it be awesome if we could have machines read the papers and decide whether those findings are credible or not. Yeah, that would be great. It would be a lot less work, it would be a lot faster, but of course, can machines actually do that? I don't know.

That is sort of the goal of this program that we are part of, and there's hundreds of collaborators in this program, it's called Score. The idea is to see if it's possible to create algorithms, artificial intelligence, that will score papers and particular findings or claims from papers on their credibility. If that works, then it provides an initial heuristic that readers like you or me can make about papers. If I pass my own paper through the machine to say, what's the scores on it? I might learn something about where the evidence, at least from the machine's perspective, is stronger or weaker. Then that might help guide me to say, “Oh, I need to bolster that with some citation evidence or other evidence that already exists. I might need to do another study there, or I might need to at least look closer.” You as a reviewer might do the same, or an editor might say, “Oh, it would be really useful to make sure that we have an expert on that topic because the machines gave an initial indication of this is an area of high uncertainty.” That's the goal, is to create some tools like that to help facilitate and direct resources for the deeper dives into the evidence credibility.

Why do we think we can accomplish that? Well, there's actually some evidence that it might work. The first bit of evidence is that we in collaboration with a number of others, particularly a great group of economists, have run prediction markets where people bet on whether findings will replicate or not. They give them the money and say, “You can buy shares, buy a yes share, if it succeeds in replication you get paid, if it fails you lose that money. Buy no shares then you're betting against it, et cetera.” Those markets were quite successful in anticipating replication outcomes when done in these systematic studies, suggesting that people have information that can anticipate whether findings are likely to replicate or not. And similar findings are observed with surveys and other kinds of elicitation techniques to assess credibility.

Then, a few different teams have tried with machines, train machines on reading papers, give them a training set and then give them outcomes of whether those successfully replicated or not and then test them on papers that have replications associated with them that the machines had never seen before. They performed similarly to the humans in predicting which papers were likely to produce replicable findings and which were likely to fail to replicate. Those initial studies were sort of the basis of doing this project at scale, the Score Project, which is doing this across the social behavioral sciences and involving thousands of papers and hundreds of replication studies and prediction markets and elicitation of confidence and multiple different artificial intelligence groups all trying to see if we can really create robust automated prediction of credibility. If it works, then it has lots of potential applications and lots of potential risks of over using–appropriate use. It will be a very interesting challenge to navigate of how one would use these appropriately, responsibly and effectively to help address research credibility and improve it.

Mills: Last question. For our listeners who, as I mentioned earlier, are just people who are interested in the science of psychology, not necessarily scientists or researchers themselves, how can they think about the issues that we've discussed today? Is there some way for them to know whether a paper has gone through the process of open data, open science registration, how are you getting the word out there to more of lay audience, particularly people who just get a lot of their science by reading more popular publications, for example, or listening to other podcasts?

Nosek: A big challenge is how do we create effective signals of credibility for different degrees of engagement with the research finding? I am a practicing scientist and I only read a very, very tiny portion of the research literature. Like an everyday person, most of what I know of science, I also get through the same mechanisms, reading the newspaper or otherwise. I can't read all of those papers, there's millions of papers. It is true, not just for the lay person, it is true for every person that we need more effective dissemination of the credibility, the confidence that we should have in different findings.

Part of that is education about how science actually works, which is initial findings are always uncertain. That first study that comes out and says, “Coffee is terrible for you.” You know that next year a study is going to come out and say, “Coffee is the best thing that you could ever drink. Don't stop drinking coffee.” And then on and on and on. Because it's really hard to figure it out. You can't randomly assign people, you're going to drink coffee, it's the only thing you're going to drink for the rest of your life. And you're going to drink not coffee for the rest of your life and then we'll see who lives longer. There's kinds of studies to actually unpack these phenomenon with confidence aren't going to be done and so, the evidence accumulation for a lot of the things that we read about is slow and uncertain and subject to change. And that's fine. In fact, it's ordinary.

The one caution I think that any of us should have is not expecting more from science than it can deliver in the short term. Science works on long time scales and the confidence that we have is accumulation over time and building of evidence. I shouldn't say over time necessarily, it can be done very quickly with lots and lots of resource investment. COVID vaccines is a perfect example of that. The vaccines themselves, the amount of evidence gathered in a short period of time is astonishing. Of course, it's billions of dollars got put into making sure that we got that evidence and got it credibly very quickly. And those particular vaccines are built on 30, 40 years of accumulated research on the basic processes of how it is that we could develop those more quickly. It feels like it happened all of a sudden, but it actually happened over many, many years of accumulating process and insight and evidence. So, I got a little bit off track there, but the basic–who can't think about COVID these days, right?

Mills: Right. Right.

Nosek: But the basic point is that–just is caution. Science is slow and so take evidence as tentative and the more signals that you have about transparency, the better. That doesn't mean that it's right, it does mean that it's possible that you can self-correct or obtain accurate assessments of confidence more easily.

Mills: Well, this has been really interesting. I appreciate your talking to us today, Dr. Nosek and I learned a lot about the renaissance that we're experiencing right now in scientific research. Thank you.

Nosek: Good. Thank you for having me.

Mills: You can find previous episodes of Speaking of Psychology at speakingofpsychology.org. We're on Apple, Stitcher or wherever you get your podcasts. And please leave us a review. If you have comments or ideas for future podcasts, you can email us at speakingofpsychology@apa.org. That's speakingofpsychology, all one word, at apa.org. Speaking of Psychology is produced by Lea Winerman, our sound editor is Chris Condayan. 

Thank you for listening. For The American Psychological Association, I'm Kim Mills.