Finding the missing science

By Harris Cooper, PhD
University of Missouri-Columbia

As behavioral researchers work to draw conclusions from decades of research into human behavior, they must also address a problem that everyone talks about but no one seems prepared to solve. I'm talking about what's commonly called the "file drawer problem"--the tendency for studies that find negative results to remain unpublished, tucked away in researchers' labs. Because these studies never see the light of day, our psychological knowledge remains imprecise and potentially biased.

Battling a bias toward positive results

To visualize the problem, imagine a new journal entitled the Journal of Six Psychology. This journal focuses on research that identifies people who display an abnormally great or an abnormally slight tendency to roll the number six on a pair of dice. A good study submitted to the journal must use fair dice and rigorous statistical procedures to compare abnormal and normal tossers. Thankfully, no such journal exists--the results of all fair dice-throwing studies are due to chance. While the occasional dice-throwing study will find groups of "abnormal" tossers, in the long run, a meta-analysis of all studies will show that study variations can be explained by sampling error alone.

But would the results of studies that appeared in the Journal of Six Psychology come to the same conclusion? Probably not.

First, researchers who found significant differences in dice throwers would be more likely to submit their results for publication than researchers who found nonsignificant differences. In 1997, my colleagues and I (Psychological Methods, Vol. 2,
p. 447-452) found that, of all studies approved by a human subjects review committee, 74 percent of those with significant findings were submitted for publication but only 4 percent of those with nonsignificant results were submitted. Other researchers have found that editors are more likely to accept manuscripts for publication if they contain significant results.

Thus, a meta-analysis of the articles published in the Journal of Six Psychology would conclude that people are more or less likely to throw sixes than they are to throw other numbers.

Complicating things further, researchers might know that the journal editors are biased toward believing that the world favors throwing sixes. Therefore, they'll be more likely to rerun a study finding a disproportionate number of non-six-tossers than a study finding more six-tossers. When reproduced, the non-six-tosser finding would not hold up and would never get published. The six-tosser article would, and the literature would become biased in that direction.

Of course, psychological researchers are smart enough to spend little time studying pure chance phenomena. Generally, the results that appear in our journals are indices of real, systematic relationships.

In addition, not all studies fall out of the publication process because of nonsignificant or uninteresting results. In our survey, we found that researchers also withhold results when they identify study design flaws or undertake a project for pedagogical reasons only.

The solution? Research registers

Still, we have to acknowledge that every test of a hypothesis contains a chance component. Therefore, censoring nonsignificant results and results that might conflict with the prevailing wisdom will influence the strength and perhaps the direction of research results that survive the publication process--in other words, "the file drawer problem."

So, what's to be done? I would not advocate that we abandon peer-reviewed journals. Among other functions, the peer-review process catches flawed studies. Still, publication doesn't ensure high methodological quality and nonpublication doesn't necessarily imply poor quality, so relying solely on published studies to make conclusions about human behavior is not enough. Instead, when we want to assess the state of knowledge on a particular topic, we need access to as much research as possible, both published and unpublished.

One way to facilitate such access is by establishing research registers--databases that catalogue studies from the time they are initiated, rather than from when they are published. For example, in medicine, the International Cancer Research Data Bank contains data of the majority of cancer clinical trials funded internationally.

The human subjects approval forms mandated by law and submitted to institutional review boards could form the basis of a national psychological research register. The costs and problems associated with establishing such a register would be considerable, but such is the case for all ambitious undertakings.

Used in conjunction with the published literature, a research register for psychology would provide those interested in cumulated results with richer and more representative sets of studies upon which to base their conclusions. Ultimately, the credibility of our research claims rests, at least in part, on our ability to find the missing science.

Harris Cooper is professor and chair of the department of psychology at the University of Missouri-Columbia.



Read our privacy statement and Terms of Use

Cover Page for this Issue

PsychNET®
© 1999 American Psychological Association

APA Home Page . Search . Site Map