APA statistics task force prepares to release recommendations for public comment

If implemented, a new set of recommendations for analyzing and reporting data will encourage researchers to be more rigorous and detailed in their reporting, and also open them up to using a broader group of methods and statistical techniques, says Robert Rosenthal, PhD, co-chair of APA's Task Force on Statistical Inference, which penned the recommendations.

The result will be a richer research literature that will allow researchers to come to more useful conclusions more quickly, Rosenthal says.

The task force, created by APA's Board of Scientific Affairs in 1996 to address growing concern among some psychologists that too many researchers over-rely on significance tests to interpret their data, will publish the guidelines in American Psychologist later this year and open them to public comment.

"We really have in the field been doing things fairly badly over a period of time," says Rosenthal, formerly of Harvard University and now at the University of California­Riverside.

Most researchers were basing their conclusions solely on null hypothesis significance testing--a method for testing the odds that a finding is the result of chance. A result is significant, the theory goes, if it reaches or is less than the level of .05, which indicates that the chances of the finding being random is only 5 percent or less.

This over-reliance on significance testing has led to the publication of many irrelevant studies that reached "significance" and the rejection of potentially important studies that didn't reach significance, argue critics.

A small study of an HIV vaccine in six orangutans, for example, would be hard pressed to reach significance, says Rosenthal, even if two out of three vaccinated primates survive.

"You'd be nuts not to consider what you found," says Rosenthal. "But that's what we've been doing and doing badly for many years."

Nuts and bolts

Although some early critics suggested banning significance testing altogether, the task force hasn't recommended banning anything. Instead, its report offers 20 recommendations for the analysis and reporting of research data that allow for a variety of techniques. The overarching theme is that researchers should make their methodology and techniques clear to their readers, acknowledging the limitations of a study's design and providing enough data so that others can judge for themselves whether the findings are likely to generalize to the general population.

Few of the suggestions will surprise researchers, admits Rosenthal. For example, most would agree with recommendations that all studies should carefully define and report the populations being studied and the characteristics of any control or comparison group. Or that researchers should clearly describe their methodology, including the rationale behind the research design selected.

More of a revelation for some will be the idea that researchers should "make friends with their data," says Rosenthal.

"Many of us were trained that we're not supposed to look too carefully at the data," he says. "You come up with a hypothesis, decide on a statistical test, do the test, and if your results are significant at .05, you've supported your hypothesis. If not, you stick it all in a drawer and never look at your data."

But the task force report strongly recommends that researchers inspect their data and even graph it before they formally analyze it. Such preliminary analyses can ferret out glitches that may corrupt study findings, including missing data caused by computer malfunction, attrition problems and outlier data caused by data-entry mistakes.

Other considerations

The recommendations also warn against relying too heavily on complicated computer programs for analyzing data. Researchers should choose the simplest approach to answer the question that interests them. In addition, they should always verify results obtained from computers either through simplified hand calculations or by checking them against the output of another computer program.

Other recommendations include:

* When appropriate, randomly assign participants to study groups using a truly random method. When it's not appropriate, describe the methods used to control for bias.

* Summarize the psychometric properties of all measurement tools.

* Always present effect sizes.

* Be careful when concluding that one variable causes another. Consider carefully alternative explanations.

* When interpreting results, think about whether they're credible, robust and whether they appear to generalize in light of other studies in the area.

After a period of public comment, the task force hopes the final set of recommendations will be used to aid in the next revision of the APA Publication Manual.

If the research community accepts the recommendations, Rosenthal believes researchers will be able to come to more useful conclusions, more quickly because it will open them up to using the best tools for the questions they're asking while also requiring a rigor of reporting and analysis that's been missing from many published studies.

--B. Azar



Read our privacy statement and Terms of Use

Cover Page for this Issue

PsychNET®
© 1999 American Psychological Association

APA Home Page . Search . Site Map