From the Science Student Council
Planning your analyses
By Stephen Mistler
Imagine that you think of the perfect research question for your dissertation. It is important to your field, and you are amazed to find that nobody has published a study on it. You discuss your idea with your advisor, and she loves it. The two of you design a study to test the hypotheses arising from your research question and you spend the next year collecting data. Once the data are collected, you consult a faculty member who is knowledgeable about statistics for help with your analyses. The faculty member informs you that your study design was inappropriate for your hypotheses. You will have to redo the data collection with a new study design, which will take another year.
This hypothetical situation may seem unrealistic, but failure to carefully plan one’s analyses during the conception of a research study can lead to disappointment and headache. Problems encountered following data collection may include: (1) you realize that the study design is inappropriate; (2) you do not know the appropriate analysis; (3) the analyses are underpowered.
The study design is inappropriate. A study design may be labeled as inappropriate if its results cannot be used to test the researcher’s hypotheses. An inappropriate study design is often the result of failing to design one’s study around specific, testable hypotheses. For example, a graduate student who designs a study based on a vague research question (e.g., “therapy A is better than therapy B for treating post-traumatic stress disorder”) is likely to run into problems. The student should instead generate specific hypotheses (e.g., “between pre-test and post-test, participants who receive therapy A will exhibit a larger decrease in anxiety than participants who receive therapy B”). When a student has a set of specific, testable hypotheses, she can design a study accordingly.
You do not know the appropriate analysis. Collecting data before creating an analysis plan can leave a researcher with data that he cannot analyze. To avoid this problem, it is important to create an analysis plan prior to collecting data. Creating an analysis plan can reveal potential analysis problems (e.g., a hypothesis that requires statistical methods with which the researcher is unfamiliar). If a student knows that he is unable to perform an analysis prior to collecting the data, he can change the study design to one that requires simpler analyses. If a complex study design is unavoidable, the student can seek the help of a statistical consultant. If you determine that you require help with your statistics, you should seek out a statistical consultant prior to collecting your data. Professors and students within your department may offer free advice. Many universities have statistical consulting centers that offer low rates to students at their institutions. If you cannot obtain help at your own university, a large number of independent statistical consultants can be found on the internet.
The analyses are underpowered. When conducting a study, it is important to ensure that the study design and the number of observations will provide sufficient power for each analysis. Unfortunately, students often decide on the study design and the number of observations without performing a power analysis. This can lead to results that are filled with false negatives (Type II errors). To avoid this problem, it is important to run a power analysis for every planned analysis before collecting data. Power analysis procedures are available in many major statistical software packages (e.g., SAS®, SPSS and R) and also in stand-alone software (e.g., PASS and G*Power). My personal favorite is G*Power, which is free and easy to use.
Taking a few simple steps can ensure that your next study will be a success. Generate specific, testable hypotheses. Write out an analysis plan that details the specific analysis to be run for each hypothesis. Run a power analysis for every analysis. Consult a statistics expert to review your analysis plan prior to collecting data. Consult a statistics expert again after the data are collected.
Stephen Mistler is the psychological methods representative on the APA Science Student Council. He is a PhD student in the quantitative psychology program at Arizona State University.
PSA is the monthly e-newsletter of the APA Science Directorate. It is read by psychologists, students, academic administrators, journalists, and policymakers in Congress and federal science agencies. Subscribe here.