Psychologists receive extensive training in statistical data analysis. The recent crisis around the inability to replicate some of our most prized discoveries about human behavior, however, has revealed that even well-established researchers can make elementary statistical mistakes that undermine the validity of their findings.
In 2016, a Dutch team led by Michèle Nuijten developed a computer program called Statcheck that scans through thousands of psychological research articles, finding inferential statistical tests and their associated p-values. Statcheck then recalculates the p-value for each test to determine if it is correctly reported.
Surprisingly, Statcheck found that nearly half of the published articles scanned — all from top international psychology journals — contained at least one p-value error and, overall, about 10% of the tests were incorrectly reported.
Christopher Green and his team at York University decided to find out whether the same was true for articles published in Canadian Journal of Psychology, Canadian Journal of Experimental Psychology (CJP / CJEP), and Canadian Journal of Behavioural Science (CJBS) between 1985 and 2014.
Using Statcheck on the 1,839 research articles published in those journals during this time span, they found error rates comparable to those discovered by the earlier Dutch team; however, they also found that Statcheck was unable to find or correctly read nearly half of the statistical reports that appeared in the articles.
Furthermore, the program made several errors when the process by which printed articles are converted to digital ("optical character recognition" or OCR) had been poor.
To see whether those anomalies affected the results, Green's team resorted to hand-editing 20% of the articles and comparing these to the original Statcheck results for the same sample. They found that the percentage of statistical errors after hand-editing dropped by nearly half, from 11.1% to 6.3%.
Overall, they found that the rate of errors had increased somewhat over time (1985–2014), from 5.5% to 6.5%. Additionally, the error rate in CJBS was consistently about two percentage points higher than in CJP / CJEP.
The study demonstrated the value of using digital methods on big datasets in order to detect important trends that are otherwise invisible. A surprisingly large number of basic statistical errors were made by accomplished academic researchers who publish in Canada's leading psychological research journals.
Moving forward, this anomaly might be addressed explicitly during the journal review process. The study also demonstrated the risk of generating distorted results when digital methods are not carefully monitored and tested by human eyes.
Thus, human and machine work best when they work hand in hand.
Citation
- Green, C. D., Abbas, S., Belliveau, A., Beribisky, N., Davidson, I. J., DiGiovanni, J.,...Wainewright, L. M. (2018). Statcheck in Canada: What proportion of CPA journal articles contain errors in the reporting of p-values? Canadian Psychology/Psychologie canadienne, 59(3), 203–210. https://dx.doi.org/10.1037/cap0000139
Note: This article is in the Core of Psychology topic area. View more articles in the Core of Psychology topic area.