In a classic 1951 experiment conducted by social psychologist Stanley Schachter, PhD, study participants were placed in groups and asked to discuss what to do with a juvenile delinquent they had read about. Unknown to participants, the researcher planted a "deviate" in the groups who strongly opposed the opinions of other group members. After the discussions ended, participants rejected the deviate the most, considering him to be the least desirable of the members and relegating him to the least important tasks, including appointing him to the lowly "correspondence" committee, as opposed to the group's "executive" committee (Journal of Abnormal and Social Psychologyopens in new window).
Schachter's findings on deviation rejection are often referred to by researchers studying group dynamics and normative social influence, but so far there have been only two direct replications of the study — one done by Schachter himself. Last year, psychologists at Illinois State University attempted to reproduce Schachter's study using the original measures. They were able to replicate the original study's results, albeit with a smaller effect size (Social Psychologyopens in new window, May).
"One intriguing possibility is that our study found somewhat weaker reactions to deviates because society is becoming more tolerant of individuals who hold deviant opinions, but only future research can answer this question," the authors say in the article. "We hope that our replication study will inspire other researchers to revisit the deviation-rejection link."
These scientists are among a growing number of researchers who say replications add substantial value to science, says David Dunning, PhD, chair of APA's Publications and Communications Board. Dunning notes that APA and several other national psychology organizations are working to make sure psychological science is based on meticulous, thorough research practices.
"Replication is important not only for preventing misuse of data, but also for furthering science," Dunning says. "Issues related to irreproducibility are not unique to psychological science, and with continued efforts to promote the value of replications, such as publishing them in online versions of APA journals, the necessary attitude shift will begin to take place."
Additional efforts include providing journal space specifically for reproducing classic research and promoting new statistical and open-data guidelines for article submission.
But some psychologists are concerned that the recent commotion around replications may do more harm than good. University of Virginia psychology professor Timothy Wilson, PhD, for example, says he is that worried the movement has put too much emphasis on false positives.
"When an effect fails to replicate, the spotlight of suspicion shines on the original study and the authors who conducted it," Wilson says. "But why should we assume that a failure to replicate is ‘truer' than the original study? Shouldn't the spotlight shine as brightly on the replicators, with a close examination of their research practices, in case they have obtained a false negative?"
Advancing replication
While much of the work to promote the reproducibility of scientific studies has only just begun, the field's enthusiasm — particularly among early career psychologists — is strong, says Brian Nosek, PhD, the University of Virginia social psychologist who directs the Center for Open Scienceopens in new window, which aims to increase openness, integrity and reproducibility of scientific research. Last year, the center received $10 million in grant money from several private foundations, including the Laura and John Arnold Foundation, the Alfred P. Sloan Foundation and the John Templeton Foundation, to support its mission.
"Many people in their early career seem willing to challenge the status quo and find a way to do this better," says Nosek.
Among APA's replicability projects is one that invites researchers to submit replication studies for its Journal of Experimental Psychology: General and Journal of Personality and Social Psychology. These journals encourage authors to submit proposals for replication plans before data collection begins and are interested in replication studies with an interdisciplinary appeal, says JEP: General editor Isabel Gauthier, PhD. Although replication articles will be published only online, the abstracts will go in the print journal.
Other APA journals, including the Psychology of Aesthetics, Creativity, and the Arts, have published special issues on replication. And APA's Archives of Scientific Psychology — the association's first open-access, open-methods, collaborative-data-sharing journal — published its first articles last summer. A top goal of Archives of Scientific Psychology is to increase the number of replication studies by making it easier for researchers to access one another's full methods and data sets.
APA divisions are also advancing data sharing and replication. In its January issue of Personality and Social Psychology Reviewopens in new window, APA Div. 8 (Society for Personality and Social Psychology) summarizes recommendations from its task force on publication and research practices. The task force recommends improving the dependability of research in psychology by using larger, more powerful sample sizes whenever possible, reporting effect sizes and 95 percent confidence intervals, making research materials available to enable others to attempt replications and encouraging the publication of high-quality replication studies.
Other psychology organizations are also working to promote more careful science. In 2013, the Psychonomic Society's Publications Committee and Ethics Committee and the editors of the society's six journals developed new journal guidelines for statistical issues, focusing on the analysis and reporting of quantitative data. Many of the guidelines address vulnerabilities in null hypothesis significance testing, in which the central question is whether or not experimental results differ from what would be expected due to chance.
The guidelines also touch on the importance of having a sufficiently large sample size, since studies with low statistical power produce inherently ambiguous results because they often fail to be replicated. To address this, the editors encourage researchers to attain adequate power and to explain how the number of study participants was determined. The guidelines warn against cherry-picking experiments, conditions, dependent variables and observations, and call for providing readers with the information they need to gain an accurate impression of the reliability and size of the effect in question.
In another effort last year, the Association for Psychological Scienceopens in new window (APS) launched its Registered Replication Reportsopens in new window initiative, an effort to replicate important psychological studies. The project brings together the original authors of an influential paper with a new cohort of scientists who will design a protocol to repeat the original study. Multiple laboratories then conduct new independent studies following the protocol, and the results from all of these studies are published together as a single Registered Replication Report in Perspectives on Psychological Scienceopens in new window. Other journals have also adopted this approach: In May, Social Psychologyopens in new window published an entire issue consisting exclusively of pre-registered replication studies.
In addition, the APS journal Psychological Scienceopens in new window has begun promoting open communication by naming authors who have made their data or research materials public or have pre-registered their design and analysis plans with specific "badges," which appear with the published articles.
Dunning says psychology is in a trial-and-error stage as it works to develop the best ways to replicate important findings. Although it can be challenging since as yet there are no set rules or overarching guidelines, "I think at the end of the day everyone will be very accepting of where the field moves," he says.
But is it necessary?
Not all psychologists agree about how best to extend and replicate each other's work, or about how best to disseminate information about what they've found, says Harvard University psychology professor Daniel Gilbert, PhD. In May, Gilbert spoke out in defense of Simone Schnall, PhD, a University of Cambridge researcher whose 2008 Psychological Science paper, indicating that cleanliness is tied to less severe moral judgments, was selected as one of Social Psychology's pre-registered replication studies. Michigan State University researchers failed to reproduce Schnall's findings, leading to several discussions in blog posts and on Twitter about "replication bullying" and how to improve the handling of replication attempts.
From Gilbert's perspective, replications should be treated as data points, not verdicts, and he notes that no matter how precisely another researcher attempts to execute a study, he or she will always do something slightly differently.
"Why should your failure repudiate my success rather than the other way around?" he says. "Since there are so many ways to screw up a recipe and so few ways to execute it properly, your failure is likely to be less informative than my success."
Wilson agrees, noting that several psychologists he's spoken with on the topic of replication have expressed concern about "false negatives," replications that purport to find no effect but which may have been done poorly, or, for any number of reasons, may not have uncovered the true effect.
"The great concern among replicators is that there are too many false positives out there — studies that purport to show some effect that may be false," Wilson says. "But there can be real dangers to false negatives as well, both to the reputations of the researchers, and to science in general." Wilson also says he's concerned that the increasing focus on replication may inhibit creativity, particularly among younger researchers.
"Almost by definition, new findings and new discoveries will be less replicable because we don't fully understand what's producing the effects," he says.
In an effort to ensure the scientific quality of replication efforts, in May, Princeton University psychologist Daniel Kahneman, PhD, proposed several "etiquette rules" for conducting replications. These include giving the original author a chance to comment on the replication methods and allowing reviewers to read the correspondence between the original team and the replicators.
"The rules are designed to motivate both author and replicator to behave reasonably even when they are thoroughly irritated with each other," Kahneman says in his commentary on the topic.
Other replicablility projects
Meanwhile, Nosek and other psychologists are working to promote replicability as a central tenet of psychology — and all other scientific disciplines.
At the end of last year, Nosek co-led the "Many Labs" Replication Projectopens in new window, a 36-site, 12-country, 6,344-subject effort to try to replicate a variety of findings in psychology. The project tested a variety of effects from studies dating back as far as 1941 and reported successful replications of 10 out of 13 of them. These included classic results such as gain-versus-loss framing (in which people are more prepared to take risks to avoid losses than to make gains) and anchoring, an effect in which the first piece of information a person receives can introduce bias into later decisions. The group is now in the midst of a "Many Labs 2" project to examine several additional effects in psychology.
Nosek is also heading up the Reproducibility Project: Psychologyopens in new window, a large-scale, open collaboration involving more than 250 scientists from around the world. The investigation is dedicated to reproducing as many studies as possible that were published in the 2008 volumes of three prominent journals: the Journal of Personality and Social Psychology, Psychological Science and the Journal of Experimental Psychology: Learning, Memory, and Cognition. The team now has nearly 50 replications completed and another 40 underway, with the goal of completing 100 replications by the year's end.
The goal is to evaluate just how challenging it is to reproduce psychological research. "Having an estimate of that would be very helpful in understanding more about the challenges in reproducibility, the causes for things not being reproducible and whether there's a problem that needs intervention," Nosek says.
The project is backed by the Open Science Frameworkopens in new window, a free, open-source Web application to help researchers organize, archive, document and share their research materials and data. The Open Science Framework was developed by Jeff Spies, PhD, who co-directs the Center for Open Science with Nosek. While the center was originally focused on psychology research, it has now moved on to cancer biology and is exploring collaborations in neurodegenerative diseases, drug addiction and economics.
The National Institutes of Health is also working to address growing concerns from both scientists and lay people about reproducibility in science. The agency now requires its postdoctoral fellows to be trained in responsible research and plans to post information on its website to guide others in rigorous science practices. Several of the NIH's institutes and centers are also testing the use of a common checklist to ensure a more systematic evaluation of grant applications. In this trial system, reviewers are reminded to check that appropriate experimental design features have been addressed, including plans for randomization and blinding. Agency leaders are also considering ways to address the academic incentive system, possibly by providing greater stability for investigators during certain career stages, and providing grants that allow for more flexibility and a longer period than the current average of approximately four years of support per project. And in a move to aid the campaign to make published scientific findings more reliable, in June, a group of about 40 leading medical-journal editors convened by the NIH — including the editors of Science and Nature — endorsed a set of draft guidelines and a general agreement intended to address this issue of reproducibility in their publications.
While these issues are common across the sciences, Nosek notes, psychology is better positioned than most fields to address replication problems and is taking the lead in tackling them.
"The problem of reproducibility is ultimately a psychological problem — it's the incentives, the motivations, the rewards that scientists face, the decisions that they make and the process of how information gets communicated through the journal system," he says. "Psychology has the tool sets to start to think about how to address the culture of science, the motivations that individual scientists face and the impact that that has on the quality of evidence that's produced."
Amy Novotney is a journalist in Chicago.
