Top Ten Tips for Graduate Students Who Want to Conduct a Meta-analysis

The Science Student Council is a group of nine graduate students who spend a couple of weekends a year with the Science staff, advising on programs and activities that would benefit graduate students in psychological science. In this column, the students will present useful information that other graduate students need to know! Visit the Science Student Council to learn more about the activities of the SSC.

Top Ten Tips for Graduate Students Who Want to Conduct a Meta-analysis
by Marcella H. Boynton, University of Connecticut

Regardless of your discipline within psychology, the literature in your area is most likely enormous. As a consequence, there are simply too many papers, often with disparate findings, to draw any real conclusions about the topic of interest. Meta-analysis is a way to resolve this issue through the systematic collection and analysis of research studies on a particular topic.

Although meta-analysis is becoming evermore popular, it can be difficult to undertake without the guidance of an experienced meta-analyst. However, if one is motivated and has a good statistics background, even a novice can perform a meta-analysis. Several advantages to conducting a meta-analysis early in your career include familiarizing yourself with an area in which you probably hope to conduct primary-level research, developing an advanced statistical skill, and creating a manuscript that can be submitted for publication. Further, because meta-analysis is a synthesis of data that have already been collected, it is not necessary to obtain external funding or recruit participants. All you need to conduct a meta-analysis is use of one or more comprehensive literature search engines (e.g., Web of Science, PsycINFO, PubMed), access to a basic statistics program such as SPSS or SAS at your disposal, and a good idea for a meta-analysis topic.

For those of you interested in conducting a meta-analysis, here are ten tips for how to get started:

1. Take a course on meta-analysis, whether it is in your department or elsewhere. Having a support system of classmates and an instructor while you work on your meta-analysis can be an invaluable resource.

2. Purchase, borrow, or download a couple of good references for how to conduct a meta-analysis. Statistics can be tough to grasp solely through books or journal articles. In the case of meta-analysis, however, there are several highly accessible texts to help you understand the ins and outs of the process (see listing of some useful resources at the end of this article).

3. Build collaborations with other researchers in your area. Part of your professional development includes forging collaborative relationships. If you have an idea about a meta-analysis you’d like to conduct, think about approaching a colleague to work on the project with you. Undergraduate research assistants, who can often receive course credit for participating in research activities, can also help with the literature search and coding of articles, which are often the most time-consuming parts of the process.

4. Pick a manageable topic. It is vitally important that you select a research literature that is manageable in size and scope. For example, conducting a meta-analysis on the Big Five Personality traits is far too general; however, you could conduct a meta-analysis looking at whether there is a gender difference on the dimension of extraversion.

5. Be comprehensive and systematic in your literature search. The purpose of a meta-analysis is to cover all of the available studies for your topic. This includes not only published research, but also dissertations, studies in non-English language journals, and unpublished studies.

6. Keep detailed records of everything that you do. Precise record-keeping is an absolute necessity in meta-analysis. You need to assiduously track your literature search, which includes recording the date of the search, search engine used, search terms employed, which articles were found, and which articles were excluded. These records will be invaluable when you write the method section of your paper.

7. Design a clear and concise coding form. A coding form is used to record the relevant statistics of interest, as well as information on any of the potential moderator variables for each study included in the final database. Keep your coding form focused strictly on the variables you plan to test.

8. Create a timeline that gives you plenty of time to conduct the meta-analysis. Depending on the size and scope of your project (as well as whether you have collaborators) you will probably need a minimum of several months to complete it. Setting specific due dates will help you stay on track.

9. Don’t be afraid to contact a meta-analysis expert if you get stuck. If you run into a snag, don’t hesitate to e-mail someone who has published a meta-analysis using methods similar to those you wish to employ. Most people are flattered that you read their paper and happy to give helpful advice.

10. Tell a good story. A meta-analysis needs to tell an interesting and useful story about your phenomena of interest. Review the debates within the literature and offer a clear and compelling analysis of the available data. Finally, write a story that makes your reader glad to have picked up your paper in the first place.

Useful Meta-analysis Guides

Cooper, H., & Hedges, L. V. (1994). The handbook of research synthesis. New York: Russell Sage Foundation.

Hunter, J.E. & Schmidt, F.L. (1990). Methods of Meta-Analysis: Correcting Error and Bias in Research Findings, Newbury Park, CA: SAGE Publications.

Johnson, B.T. & Boynton, M.H. (2008). Cumulating Evidence about the Social Animal: Meta-analysis in Social-Personality Psychology, Personality & Social Psychology Compass, 2, 1-25.

Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-analysis. Thousand Oaks, CA: Sage Publications, Inc.

Useful Meta-Analysis Software

Biostat Corporation (2005). Comprehensive Meta-analysis Version 2, Englewood NJ: Author.

Johnson, B. T. (1993). DSTAT 1.10: Software for the meta-analytic review of research literatures. Hillsdale, NJ: Erlbaum

Wilson, D. B. (2006). SPSS, STATA, & SAS macros for performing meta-analytic analyses. Retrieved August 9, 2007 from http://mason.gmu.edu/~dwilsonb/ma.html.