When James DuBois, ScD, PhD, launched a training program in 2013 for researchers caught failing to comply with research protocols, plagiarizing, or falsifying and fabricating data, it was controversial, to say the least. The program’s launch was accompanied by a feature article in Nature’s news section, and much of the feedback was incensed (Cressey, D., Vol. 493, No.opens in new window 197).
“Oh, my goodness, the chat for the online story!” DuBois, an applied psychologist at the Washington University School of Medicine in St. Louis, recalled. “There was so much hate.”
It’s no wonder. Misconduct flies in the face of the values of scientific research, which at its heart is about the search for truth. But the reality is that misconduct and its cousin, questionable research practices, occur on a spectrum. The most egregious cases are from outright hucksters who don’t generally qualify for second chances. The majority, though, live in a gray area where an honest mistake, publish-or-perish pressure, and lack of clear norms or quality training can lead to blunders. Anyone, no matter how well intentioned, could be vulnerable.
Psychology as a research field is susceptible to all these pressures, but psychologists are also a major force in solving the problem. With expertise in behavior change, motivations, and incentives, psychological researchers are tackling misconduct at both the individual and structural level. It’s an effort that has already led to big shifts in the field, with more likely to come.
“Psychologists are leading the charge in changing their individual behaviors and the structural components that lead to questionable research practices, including misconduct,” said Brian Nosek, PhD, a social psychologist at the University of Virginia and a proponent of the open science movement as a bulwark for improving credibility. “Most changes have occurred in journals so far. Funders and institutions have begun to change, but there’s still a lot to do.”
The scope of the problem
There are no firm numbers on how much research misconduct goes on. Attempts to quantify the problem usually either count retractions, survey scientists about their own and colleagues’ practices, or statistically reanalyze published papers. A 2019 review of studies using all three methods found that between 0.6% and 2.3% of psychologists surveyed admitted to falsifying data themselves, and between 9.3% and 18.7% said they’d witnessed another researcher doing so (Zeitschrift für Psychologieopens in new window, Vol. 227, No. 1). Statistical reanalyses suggested that there were major inconsistencies in between 12.4% and 20.5% of published studies, but it’s unclear whether these were examples of outright misconduct, questionable research practices, or errors. Finally, a count of retractions found that 64.84% of retractions in PsycINFO were due to misconduct, similar to rates in other scientific fields.
It’s best to take these numbers with a grain of salt, said Armin Günther, PhD, a research associate at the Leibniz Institute for Psychology in Germany who coauthored the review. Misconduct can be tricky to define, and there are many poor practices that can easily slip by unnoticed.
“I’m not sure if it is so important to have a number, to say ‘It’s 3% of articles,’ or ‘It’s 10%,’” Günther said. “It’s more of a qualitative issue, because if you suspect that research and scientific work is basically not trustworthy, that’s a problem.”
Setting aside egregious fraudsters, misconduct can be a slippery slope—and one that is all too often incentivized by the academic structure, says Jennifer Crocker, PhD, the chief editorial advisor at APA and a social psychologist at The Ohio State University. “People’s careers, livelihoods, and reputations depend on their ability to publish in the scientific literature,” Crocker said. “There is incredible pressure for early career researchers to have not only publications but lots of them—first-authored publications, publications in top journals. And that pressure doesn’t go away over the course of a career.”
Participants in DuBois’s course, the P.I. Program at the Washington University in St. Louis, are usually referred by their universities as a requirement for continued employment or research privileges. One common reason for referrals is failure to provide adequate oversight when someone else in a lab falsifies data, DuBois said. Often, the root cause is being overextended and failing to prioritize compliance.
In a study of 120 cases of misconduct pulled from the research literature and from records of federal research oversight agencies, DuBois and his colleagues found that in these cases at least, oversight failures were common (Accountability in Researchopens in new window, Vol. 20, No. 5–6, 2013). Motivated reasoning and narcissistic thinking were common, with primary investigators who fabricated or falsified data often talking about the pressure to publish or get grants in the pursuit of fame. “They’re not falsifying data just to keep their career afloat, but really to become a superstar,” DuBois said.
Changing incentives
It can be fraught to link misconduct directly to publish-or-perish culture, DuBois said; after all, many researchers cope with the pressure without lapses in ethics, and individual psychological traits seem to play a role for those who don’t. Nevertheless, the past decade or so has seen calls for a change in the incentive structure that links top-tier jobs to flashy results in high-impact journals. The San Francisco Declaration on Research Assessment (DORA), first drafted in 2012, recommends that employers and funders “not use journal-based metrics, such as Journal Impact Factors, as a surrogate measure of the quality of individual research articles, to assess an individual scientist’s contributions, or in hiring, promotion, or funding decisions.”
A few universities have tried to put that recommendation into practice. In the Netherlands, a program called Recognition & Rewardsopens in new window aims to inject balance into how academics at participating universities are judged. The goal is to put more focus on aspects outside of publications, including teaching, leadership, and patient care. Even among those who support these ideals, some signatories to DORA have found change to be challenging: In 2019, a chemical systems engineer at ETH Zurich, one of the DORA supporters, had to apologize after including a requirement that postdoctoral applicants must have published in a journal with an impact factor of 10 or higher. And DORA still lacks a critical mass of signatories. Fewer than 50 psychological journals, associations, and programs around the world have endorsed the recommendations.
“Everybody knows that the impact factor is not a good criterion for the scientific work of single persons, but even though everybody knows, it is quite hard to get rid of these kinds of evaluations,” Günther said. “I hope and I think that we will go in the right direction, but it is very slow, like a big ship you have to turn in the ocean.”
If it’s hard to drop the prestige associated with big-name journals altogether, attempts to change the kind of research those journals publish seem to be making more headway. One goal of the open science movement is to ensure that journals don’t just jump on splashy positive results; to better represent the state of science, they should also publish negative results and incremental work. As a result, open science should take the pressure off researchers to produce always flashy, always significant results, reducing one major incentive that leads to questionable research practices.
Open science also elevates transparency as a core value, which makes it more difficult to fabricate or falsify data on the way to a conclusion. Research led by Daniele Fanelli, PhD, a fellow in quantitative methodology at the London School of Economics and Political Science, found that a culture of transparency and communication, paired with strong rules around misconduct, is associated with fewer misconduct-related retractions (PLOS ONEopens in new window, Vol. 10, No. 6, 2015).
“Transparency does a few things,” Nosek said. “As an author, it gives me some additional accountability. As a reader, it gives you a means to evaluate the basis of my claims. And as a community, it gives us shared evidence to debate credibility and whether biases were injected in practice.”
This kind of transparency is likely familiar to many psychological researchers. In November 2020, APA signed on to the Transparency and Openness Promotion (TOP) Guidelines for all its journals, which require certain disclosure of data and research materials and encourage full data transparency when ethically possible.
Research suggests these efforts are having the desired effect. A study led by Anne Scheel, a doctoral candidate at the Eindhoven University of Technology in the Netherlands, found that registered reports in psychology—in which methods and data analysis plans are laid out prior to the study being done, and publication is guaranteed if those plans are met—return findings consistent with their hypotheses about half the time, compared with 95% of journal papers published without preregistration (Advances in Methods and Practices in Psychological Science,opens in new window Vol. 4, No. 2, 2021). This is a good sign, Nosek said, because the registered reports more accurately reflect the reality of research—and they dispense with the unrealistic expectation that researchers publish a stream of positive results.
Individual interventions
Open science is a promising way to change incentives, but it can’t combat fraud alone. Any set of new rules can be abused. And total data transparency is not always possible, Crocker said. Imagine, for example, a longitudinal study of romantic dyads. “If people know that they were in the study and they know their own characteristics and responses, they can find their own data in the data set,” Crocker said. “Then they can find their partner’s data in the data set, so anything the partner says is no longer confidential.”
So other kinds of change matter, too. “Responsible Conduct of Research” courses are now de rigueur in psychology training, but there are still gaps in how much funding and attention research institutions give to research integrity, said Nicholas Steneck, PhD, a historian of science, emeritus professor at the University of Michigan, and founder of the World Conferences on Research Integrity. If anything, Steneck said, pressures are worsening on research faculty at universities: State funding of higher education never fully recovered after the 2008 recession, and federal research dollars have flatlined, meaning steeper competition for jobs and grants. “What’s that going to do?” Steneck said. “It’s going to increase pressure on researchers.”
The research community is pushing for structural changes, but institutions haven’t been as proactive, Steneck said. Many rely on volunteer-only staffs in their research integrity offices, and upper administration is often stretched too thin to monitor the university research climate. One of the main goals of the World Conferences on Research Integrity is to get institutional research leaders involved in identifying the problem and solutions, Steneck said. One of his current projects is disseminating an online research integrity training programopens in new window via Oxford University Press and using it to help university leadership understand potential problem spots in their own programs: In which fields are researchers under extreme pressure to get grants? In which labs are graduate students lacking support from their mentors?
DuBois’s P.I. Program offers a unique opportunity to correct these problems on a case-by-case basis. More than 100 researchers from 57 institutions have attended so far. They’re usually referred by their institutions after a lapse of moderate severity, such as failing to catch a lab member’s data falsification or dropping outliers from their reports without any change to the conclusion.
Evidence suggests that many researchers who make these kinds of errors will not be unceremoniously booted from the field; a study by Kyle L. Galbraith, PhD, a research ethicist now at the Piedmont Athens Regional Medical Center in Georgia, found that of 284 researchers disciplined by the U.S. Office of Research Integrity, at least 47% went on to continue doing research (Journal of Empirical Research on Human Research Ethics, Vol. 12, No. 1, 2017opens in new window). DuBois’s program aims to ensure that these researchers don’t continue making the same mistakes.
The program was founded with a National Institutes of Health (NIH) grant and is also funded by fees paid by participants. The three-day course is grounded in psychology. “Motivational interviewing plays a big role for us,” DuBois said. “People sometimes feel they have been treated unfairly by their institutions and that becomes a real barrier to change. We try to get them to ask themselves a question like, ‘How do you think you’re perceived at your institution? Is that helping your career?’ Pretty soon they are making exactly the changes you want to see without having to concede that it’s all their fault.”
DuBois and his team follow up with clients with a survey after a year. The results are self-reported but suggest that graduates of the program apply lessons they’ve learned in the course, such as implementing standard operating procedures, holding regular lab meetings, and reducing work stressors (Academic Medicine, Vol. 93, No. 4, 2018opens in new window). Although DuBois and his team can’t audit these claims themselves, there has not been a case to his knowledge of a program graduate being disciplined a second time for misconduct.
With an eye toward prevention, one of the coleaders of the P.I. Program is now developing a management and leadership training program for early career researchers. Alison Antes, PhD, an industrial and organizational psychologist at the Washington University School of Medicine in St. Louis, has just received NIH funding to launch an online program that would offer training in these skills for running labs with integrity and compliance.
Most researchers aren’t formally trained in management, even though primary investigators take on the roles of executive management, middle management, and basic supervision all at once, Antes said. “It is really a lot that researchers are asked to manage and deal with while at the same time trying to get grants, trying to get their papers out,” she said.
Antes’s research found that scientists who are rated as exemplary by their peers use seven key practices: holding regular team meetings, providing supervision, encouraging shared ownership, ensuring good training, communicating positive attitudes about compliance, reviewing data and findings, and following standard operating procedures (PLOS ONE, Vol. 14, No. 4, 2019opens in new window). Her new program aims to address failures in these domains. Leadership training goes hand in hand with management training, she added: Leadership involves bigger-picture concepts such as communicating values, setting expectations, and creating an environment that fosters good practices.
The COVID-19 pandemic made clear the costs of public distrust in the scientific process, Steneck said. But public trust must be earned, making dealing with research misconduct an urgent matter.
“If we don’t get our house in order when it comes to integrity, there’s a perfectly good reason for not believing researchers,” Steneck said. “We have to be concerned about that. We have to be above criticism, above fault.”
