Psychology’s reproducibility crisis has become a hot topic in the field. A slew of papers—from epidemiologist and biostatistician John Ioannidis’s 2005 article titled “Why Most Published Research Findings Are False” (PLOS Medicineopens in new window, Vol. 2, No. 8) to an international meta-analysis that examined the reproducibility of 28 highly cited studies in 2018 (Klein, R. A., et al., Advances in Methods and Practices in Psychological Scienceopens in new window, Vol. 1, No. 4)—have called into question the validity of research shared by academic journals. Much of this is due to a flawed scientific process, according to Ioannidis, which can include anything from differing study designs to publication biases to manipulating data analysis to get the desired p value.
As a result, APA, the Center for Open Science (COS), and several other scientific organizations have been working to improve the field’s credibility. Since 2017, APA has awarded “open science badgesopens in new window” to journal articles where studies or analysis plans were submitted to a public registry during the research stage. This process, called preregistration, allows other scientists to verify that the study adheres to the submitted plan, enhancing transparency among all collaborators. And just last year, APA became a signatory of the Transparency and Openness Promotion (TOP) Guidelinesopens in new window, a document created by a group of publishers, funders, and societies under COS that lists eight open science standards for journals to implement—including transparency of a study’s design, analysis, data, and code—and encourages the submission of replication studies of research published in the journal.
Practices like these are essential to furthering equity, diversity, and inclusion in science. For example, providing detailed sample demographics can reveal limitations of drawing broad conclusions from studies done on majority populations. Making more data, materials, and code available also means that more researchers can conduct replication studies or even new analyses, according to Su Yeong Kimopens in new window, PhD, editor in chief of the journal Cultural Diversity & Ethnic Minority Psychology.
But one of the major gaps that remains in the open science movement is that it has not focused enough on the needs of underrepresented scientists, including racial- and ethnic-minority groups and women, according to Annie Hill, editorial director for APA’s 89 journals. Hill and others at APA are coordinating an effort to promote equity in open science. “There are currently open science practices espoused that either disregard the diversity of people practicing psychological science or have unequal effects on participants,” said Rose Sokol-Chang, PhD, publisher of APA’s journals and books. “Our approach is to consider open science alongside equity, diversity, and inclusion.”
With the help of researchers, the editors of APA’s journals, and other scientists, the association is working to address the biases that influence journals’ review and publication processes, such as disparities in research funding and a tendency toward large sample sizes. The goal is to make open science more equitable across psychology.
“I see this overall as a net positive,” said George Banks, PhD, a board member of COS and an associate professor of management at the University of North Carolina at Charlotte. “Open science will continue to move forward from bottom-up efforts of individual scientists openly sharing online and posting their own data and top-down efforts like this from APA.”
Openness in an exclusive field
Many in the research community feel that open science practices are not equally accessible to all researchers because of the deeply rooted power inequities that exist within science. “Factors such as a scientist’s career stage, employment stability, financial circumstances, country of origin or residence, and cultural context (including their race, gender identity, and ethnicity) may all create barriers to specific aspects of open science,” wrote a group of researchers in a piece for American Scientistopens in new window. The movement fails to acknowledge the “systematic barriers that make open science more accessible to some scientists than others, nor any respect for the steps taken to overcome some of these barriers.”
This inequity is evident in the U.S. psychology workforce, where people of color comprise only 16% of the industry population and face a litany of challenges throughout their career. Researchers of color, as well as women, are less likely to receive research funding than their White and male counterparts (Witteman, H. O., et al., The Lancetopens in new window, Vol. 393, No. 10171, 2019; Hur, H., et al., PLOS ONEopens in new window, Vol. 12, No. 2, 2017), which often makes it harder for them to complete their research and hire personnel who can prepare data to share on open science platforms.
Open science practices like the need for a large study sample are also more difficult for researchers who don’t have the proper time, resources, or support, said Hill. Also, research done with smaller, often underrepresented populations will not have a large sample size. “In general, things like power analysis and a huge sample size are a privilege,” she said.
Additionally, for scientists who are studying vulnerable populations, data that contain personal, private, or sensitive information might put these communities at risk of exploitation if shared. A research team with the Open and Collaborative Science in Development Network that surveyed students, women in rural areas, Indigenous communities, and other ethnic groups across the globe found that members of some Indigenous populations had “strong resistance to the idea of openness, as a result of the way in which Indigenous knowledge had been historically misused, appropriated or exploited in the name of colonialism and neoliberalism” (Albornoz, D., & Chan, L., Informação, Memória e Tecnologiaopens in new window, Vol. 4, No. 1, 2018). An example of this type of exploitation is the Tuskegee syphilis studyopens in new window, in which scientists researched the progression of syphilis in Black men without providing them with proper medical care.
Underrepresentation in both scientists and study populations is something that Joanne Chung, PhD, an assistant professor of psychology at the University of Toronto, has witnessed firsthand and focuses on as an instructor and research mentor. When reviewing a study, she has her students ask “Who is being researched? Who is doing the researching? And how do people who are underrepresented fit in there?” said Chung. She shared a preprint of her longitudinal study about the experiences of young Syrian adults in the Netherlandsopens in new window this March on PsyArXivopens in new window, the free preprint service for psychological studies that was created by the Society for the Improvement of Psychological Scienceopens in new window (SIPS) and COS to improve open science in the field. “I want them to be skeptical of the entire process.”
Asking more questions
APA is working to bring this same skepticism into the review and publication process of the association’s journals. In 2019, APA formed an Ad Hoc Committee on Open Science and Methodologyopens in new window, a group of psychologists with expertise in open science that aims to develop best practices for promoting equity in publishing.
“There are a lot of challenges for people doing open science, especially when they are doing open science in communities that are underrepresented,” said Hill, the group’s staff liaison. “We want to encourage researchers to think about people who are beyond their immediate sphere.”
This involves APA mandating that its journals adopt the TOP Guidelinesopens in new window, standards that span from citing data, code, and software properly to being transparent about the team’s research methods to sharing links to preregistered research and analysis plans. “These guidelines are intended to provide aspirational ideal along with very practical implementation opportunities by having eight separate components and stages of adoptions of those components,” said Brian Nosek, PhD, executive director of COS. “The key goal of the open science movement is not to force openness but to make openness the default.”
However, Nosek, Hill, and others understand that not all researchers can follow these guidelines verbatim.
“Sometimes people mistake a call for greater transparency or a call for more open science practices as either not applicable to them or directly contradicting their ethical obligations,” said Katie Corker, PhD, an associate professor of social psychology at Grand Valley State University in Michigan and a member of the ad hoc committee. “Of course, there are going to be exceptions to these new policies; of course, there are legitimate ethical limitations. It’s just a matter of figuring out what these practices look like in a particular subfield or in a particular area where these concerns are more common.” Corker is also an ambassador for COS and executive officer of SIPS.
Additionally, APA’s Office of Publications and Databases is working to add safeguards to the journal review process that prevent the publication of articles that may blatantly or inadvertently spread scientific racism. This could include questions for reviewers such as: Does this research reinforce racial or ethnic stereotypes? Is there a larger inequality highlighted in this study or review? Do these findings contribute to the well-being of all communities?
The issue of equity in science is even more prominent in the era of COVID. According to a report made public on GitHub, a platform that allows people to share and host code last year, the publishing rate of preprint submissions by female scientists on arXivopens in new window and bioRxivopens in new window—two preprint servers that cover many STEM fields—fell relative to the rate for male scientistsopens in new window in 2020. Research shows this is likely because women spent significantly more time doing household chores than menopens in new window, thus making it harder for them to focus on their careers during lockdown.
In a 2020 letter to the editor published in Nature Ecology & Evolutionopens in new window, researchers from around the world also expressed their worry that hiring freezes, remote learning, and other strains on peoples’ lives will gut diversity in science. The authors expressed their concern that the pandemic will disproportionately impact early career scientists from groups that are “disadvantaged and/or discriminated [against] in the fields of environmental sciences, including women, researchers from the Global South and persons with disabilities.”
For this reason, APA’s work to promote equity in open science is more vital than ever. “There is still a lot that we can do in terms of education, in terms of helping people to change their research practices and change the culture of their lab,” said Corker. “Deploying these policy changes at APA—the largest publisher in psychology—has the potential to really improve our practices at a large scale.”
