In his darker moments, University of Toronto social psychologist Michael Inzlicht, PhD, wonders whether he has spent his career "chasing puffs of smoke."
Inzlicht has spent decades studying social psychological effects including ego depletion — the idea that self-control is a limited resource and that exerting willpower in one dimension can sap people's ability to exert it in an unrelated area. He has published several papers on the effect. But in a recent multi-site replication studyopens in new window, 24 independent labs tested more than 2,000 participants and found evidence that the ego depletion effect is either very small or nonexistent.
The study is one of several that have fed a growing debate about whether the standard methods of psychological research and publishing have led the field astray, allowing irreproducible results to accumulate in the literature.
Some psychologists — and many news media outlets — have called it a "replication crisis." Others, however, feel that the concern is overblown, and have critiqued the methods and findings of the replication proponents.
Psychologists are not the only ones grappling with this issue. Many fields of scientific research are facing similar concerns. Take biomedical science. A decade ago, Stanford University epidemiologist and biostatistician John Ioannidis, PhD, published a paperopens in new window about biomedical research titled "Why Most Published Research Findings are False" (PLOS Medicine, 2005) that has become the most cited in the journal's history. Ioannidis is watching the discussion in psychology with great interest.
"Psychology is clearly going through a lot of soul searching and has some of the best ideas and most contested ideas right now," he says. "There are many research practices that people are re-examining, trying to see what they can do better."
From the lab to the front page
In psychology, the replication discussion moved from labs and academic conferences to newspaper pages last summer, when the Open Science Collaboration (OSC) — a group of more than 250 researchers led by University of Virginia psychology professor Brian Nosek, PhD — published the results of their attempt to reproduce 100 studies from three top journals in social and cognitive psychology. The researchers found that although 97 percent of the original studies produced significant results, only 36 percent of the replications did (Science, 2015opens in new window).
In the paper, the OSC researchers did not use the word "crisis" and were careful to say that their findings were meant solely to provide an estimate of reproducibility in psychology, not to definitively prove any of the original research wrong.
"Original studies are not definitive," Nosek says. "And replications are not definitive — no study is definitive. In science, we seek definitiveness and almost never get it. We have to be self-critical and modest with our claims."
Nosek does, however, believe that the OSC results are meaningful, and reflect significant problems in the way that psychological research is conducted and published. Nosek, who spent much of his career studying unconscious biases, has focused increasingly on these methodological issues instead. Three years ago, he founded the Center for Open Science, a nonprofit devoted to increasing the "openness, integrity and reproducibility of scientific research."
He and his colleagues see several problems plaguing psychological research. First, there is the "file-drawer problem." Journals in psychology (and other fields) want to publish positive results. So when researchers find a positive result in the lab, they write it up for publication. When they find a negative result, they shove it in their file drawer — leaving the research literature rife with unchallenged false positives.
And where do those false positives come from? Some will occur at random, of course. But common research practices make them more likely, especially what's sometimes called "p-hacking" — essentially, when researchers fish around in their data and perform multiple analyses until they find effects that reach the standard criterion of significance: p<.05. P-hacking sounds nefarious — like cheating — but in practice it is more complicated and thus more difficult to address than that. Science involves many small decisions at all stages of research — what data to collect, what variables to analyze, how and when — and researchers' small, unconscious biases can affect those decisions at every stage.
So Nosek and his colleagues at the Center for Open Science are advocating for a series of reforms and creating infrastructure to bring negative results out of the file drawer, and to make inadvertent p-hacking less likely. One way to do so, they believe, is for researchers to make their data and research methods publicly available. That would make replication easier and more likely. Another is to encourage researchers to pre-register more studies, essentially "tying their hands" by having them describe their research methods and analysis plan in advance so that post-hoc data analysis is clearly labeled as exploratory. They have written guidelinesopens in new window for journals interested in adopting their ideas, have developed cloud softwareopens in new window to make data sharing easier, and are even offering $1,000 prizes to researchers who pre-register their studies and then publish the results.
Inzlicht, for one, finds the work of Nosek and the other replication efforts convincing. He was particularly taken aback by the results of the ego depletion study, which was part of a new series of registered replication reportsopens in new window supported by the Center for Open Science and published in the journal Perspectives on Psychological Science.
"You could call me a replication convert," Inzlicht says. "I used to think that there were errors [in the way psychology research was conducted], but that the errors were minor and it was fine. But as I started surveying the field, I started thinking we've been making some major mistakes." At this point, he says, "If your faith is not shaken, you're not paying attention or you're in a state of denial."
A replication backlash
But not all researchers are convinced of the OSC paper's validity, or that the state of the research literature is so dire. In March, Harvard University psychologist Daniel Gilbert, PhD, University of Virginia psychologist Timothy Wilson, PhD, and colleagues published a critique of the OSC studyopens in new window, also in Science. In it, they argue that the replication effort had several critical flaws. First, they say, the replicated studies were not chosen at random from the three journals. Instead, some studies were deemed too difficult to replicate and were excluded.
"You have to find a representative sample to replicate if you want to learn anything, and that's not what OSC did," Gilbert says.
He and his co-authors also contend that many of the replication studies were less faithful to the original studies than the Science paper or popular press accounts made clear. Many of the replications were tested on different populations, in different countries and using some different methods than the original studies. The replicators did attempt to vet their methods with the authors of the original studies, but Gilbert points out that the replications that were approved by the original authors replicated at a higher rate than those that were not.
Finally, Gilbert and his co-authors make a statistical argument that the number of "failures to replicate" in the OSC paper is what could be expected by chance alone, given the power of the replication studies and infidelities in replication design.
Along with Gilbert's critique, Science published a "response to the responseopens in new window" from Nosek and his colleagues. The debate continued online and in the blogosphere from there.
The statistical details get complex, and they are important. But taking a broader view, the debate also reflects a basic disagreement over the value of large-scale replication efforts like the OSC.
Gilbert believes that that value is minimal, particularly relative to the costs of conducting the research.
"Finding the exact number [of studies that replicate] is really not important," he says. Regardless of whether it's 40 percent or 70 percent, he says, the field should be discussing research best practices.
"I don't know anyone who thinks open data is a bad idea, but I know a lot of people who believe that pre-registration comes with too many costs" that limit scientific creativity and discovery, he says. "That's an interesting and worthwhile debate to have. But nobody has been able to explain to me why we need this number, other than to give journalists something to misunderstand."
This is not the first time that the replication debate has become heated. Gilbert and other critics point out that for researchers whose work is subject to a replication attempt, the professional stakes are high.
Simone Schnall, PhD, of the University of Cambridge in the United Kingdom, was among the first to experience a replication furor. An earlier Center for Open Science project attempted to replicate a well-known study of hers that found physical cleanliness can influence people's moral judgments.
The replication failed to find a significant effect, but Schnall believes that there were problems with its methods, and says she is aware of other researchers who were able to replicate her finding. She was dismayed that the replication did not go through the same sort of post-study peer review that her original research did.
Two years later, she does not feel that that issue was addressed in the OSC study, because the findings from the 100 individual studies were not evaluated by experts in the same way as the original work. Also a problem, she says, is that for now, studies and research areas are still "cherry picked" for replication. There is no review panel that decides which findings need to be verified, so that can leave individual researchers feeling targeted.
"Being part of a replication project can involve a lot of work, especially if several papers by the same researcher are under scrutiny. Even for somebody with tenure this can be an issue, but it's even more problematic for junior researchers such as postdocs and untenured faculty members," she says. "Merely raising the question about reproducibility can do a tremendous amount of damage, and so that's all the more reason these replications have to be done well."
Other researchers have also had problems with the replications of their work, including Roy Baumeister, PhD, who coined the term "ego depletion" two decades ago. In a comment published with the ego depletion replication study, he argues that the procedure used in the replication was a "new, mostly untested" procedure that did not actually measure ego depletion. He and his colleagues plan to organize another ego depletion replication this year.
Meanwhile, as the debate continues, other researchers are continuing to dig into the details of the OSC study to try to figure out why so many of the replications failed to find significant effects and if there are any common threads that made some less likely to replicate than others. In an analysis in press in the journal PNAS, New York University psychology professor Jay Van Bavel, PhD, found that papers in which the original effect was more "context-sensitive" were less likely to replicate.
Context-sensitive studies are those that examine issues that vary according to time or place, such as race or political attitudes. Context-insensitive studies examine things that, at least in theory, should stay constant in different settings, such as the Stroop effect.
The fact that the context-sensitive studies were less likely to replicate suggests that there are "hidden moderators" at work, says Van Bavel — important contextual differences between the original studies and the replications that, although not obvious to the replicators, contributed to the replication's failure.
Van Bavel does believe that replication studies are valuable, and he sees two takeaways from his study. The first is that researchers and those who want to replicate their work should work more closely together to clarify the theory behind the research and decide exactly what stimuli and experimental designs are likely to elicit the effect they're looking for.
The other takeaway, he says, is that researchers may need to become more clear in their original papers about the limits of their research. "Most of the things we study aren't universal, they are going to have boundary conditions," he says. "So part of these is being modest, and being more explicit about the boundary conditions."
A blow or a boon to psychology?
Gilbert's concern about journalistic misinterpretation hints at another worry for many psychologists: that the replication research is harming psychology's public image.
Nosek believes that the concern is not warranted. "By and large, the actual science journalists have done a good job unpacking this, though nuance gets lost in the headlines," he says.
Crucially, he also believes that questioning some of the basic tenets of psychology will not cause funding agencies to abandon psychological research. "My sense is that funding agencies are actually looking to psychology as a leader in addressing these replication issues," he says. "These are not issues that are limited to psychology."
In fact, among the next projects that the Center for Open Science will undertake are replication studies in cancer biology and tropical ecology.
Within psychology, the replication movement, which so far has mostly focused on social and cognitive psychology, is expanding to other areas of the discipline. For example, Michael Frank, PhD, a developmental psychologist at Stanford University, is organizing a multi-lab pre-registered replication study of a classic finding in developmental psychology: that infants prefer infant-directed speech, or mother-ese, to adult speech.
"The study will serve many purposes," Frank says. "Not just to test the specific effect, but also to allow us to get a handle on some of these methodological issues that make replication tough."
So what about all of the psychologists who, like Inzlicht, are worrying that the seemingly reliable findings on which they have based their research and their careers might turn out to be "puffs of smoke?"
Inzlicht himself sees the silver lining. "We're better off now than we were yesterday, or 10 or 20 years ago, even if it means we overturn findings," he says. "If something is not true, it's obviously better to know. I don't want to waste my time and I don't want to waste my students' time chasing smoke."
From outside the field, John Ioannidis agrees. "I don't expect 100 percent of psychology to be wrong by any means, but there may be a pretty high rate of irreproducibility that will vary across fields," he says. "There will be many classics that hold up, and some will be refuted. Some recent literature with small effects may not hold up. I think this is all very good. It tells us what we know and don't know. Psychology will not disappear, it will become more reliable and more exciting to work on."
