Conversation with Nick Bowman, PhD

A growing number of journals publish registered reports, for which a preregistration, or protocol, is peer reviewed for its study design and methodology.

Nick Bowman, PhD, associate professor at Texas Tech University and associate editor for registered reports at Technology, Mind, and Behavior (TMB), discusses the key features, misconceptions, and benefits of this unique article type.

Registered reports are research manuscripts that are submitted as part of a two-stage peer review process.

In Stage 1, authors first submit a study proposal that includes a complete literature review, proposed study methodology, and data analysis plan. At this stage, authors are attempting to make the case that their proposed yet-to-be-completed study would advance knowledge regardless of the eventual results—for example, if the study were to produce statistically null resultsopens in new window. In these proposals, authors explain how the proposed work is theoretically sound and speaks to an important question, specify that the research will utilize appropriate methods and analyses to speak to the questions at hand, and so on.

Reviewers then provide formative feedback on the proposal, and editors use this feedback to determine if they will invite authors to move on to the second stage of the process. Authors invited to Stage 2 complete the study, write up the results, and finally submit a full manuscript.

The full manuscript is then reviewed by editors and reviewers, with a primary focus on the appropriateness of the description of the outcomes. Critically, papers that are invited to move on to Stage 2 are guaranteed to be published so long as the procedures that were utilized match those that were proposed and the outcomes and the implications of those outcomes are fairly described.* Acceptance does not depend, for instance, on whether the authors’ hypotheses were supported or falsified. As such, when authors are invited to move on to Stage 2, we refer to this as an “in principle” acceptance. In essence, the journal is indicating that they think the proposed study has value to the field regardless of the outcomes—the review is focused on the science, rather than the statistical results.

*It’s important to note that, contrary to popular belief, it is possible for authors to make changes/additions to their research during Stage 2 as long as they follow a specific process. (Read on to learn about other misconceptions about registered reports.)

In their final published form, registered reports do not look or read all that different from the published manuscripts that most of us are familiar with—they include the literature review, method, results, and discussions sections we’ve all learned from the APA manual. Where the real difference lies is with the Stage 1 review described above.

The model is quite different from more traditional manuscripts, in which authors submit the results of their research rather than a proposal for their research. Yet for most scholars, the registered report process is very similar to the academic thesis or dissertation. That is, an initial study proposal is submitted and then evaluated by a committee and if approved, the candidate completes the research and defends their work as a whole. I like to use this comparison because it reminds people that nearly every one of us has at least some experience with registered reports, either as a degree candidate or a committee member (or both).

These terms are often used interchangeably, in part because all registered reports de facto include a preregistration plan. However, the reverse is not true: scholars can preregister their study predictions/questions, methodology, and data analysis plans as part of a traditional manuscript. The most apparent difference between the two formats is the editorial process.

For registered reports, editorial decisions are based mostly on peer review of the study proposal (i.e., the preregistration materials). For preregistration, editorial decisions are based on peer review of the entire manuscript (i.e., they do not undergo multistage peer review). In a recent editorialopens in new window, I was part of a group of scholars that summarized these differences. I would say that preregistration is still a valuable tool in helping improve the transparency of scientific practiceopens in new window, but registered reports add an additional layer of peer review and feedback to the process.

Within our Editorial Manager softwareopens in new window, submitting a registered report is rather straight-forward as we now include “Registered Report” as a specific article type option. When authors select this option, it will flag the paper as a registered report and this will ensure that the editorial staff and any peer reviewers know to expect a study proposal (and not a completed manuscript). In that way, the submission process is no different than any other article type.

That said and given that registered reports are somewhat new for many scholars, we always encourage our colleagues to reach out directly for specific details—I’ve had several email exchanges and phone calls with authors to help them better understand what to expect in the multi-stage review process, and field questions about how much detail to include in their study proposal. More specific details are outlined on the TMB webpage, which is updated regularly based on community feedback.

There is no single template that registered reports need to follow, beyond the APA style guide of course.

That said, successful registered reports do contain a few common elements and sections such as:

• explicitly stated hypotheses or research questions;
• clear justification for sample sizes and sampling frames, usually based on a power analysis or some other robust logic;
• links to online repositoriesopens in new window for sharing survey language, copies of stimulus material, and other content that could help reviewers understand the study being proposed;
• a “Data Analysis” section, instead of a “Results” section, in which authors provide a detailed discussion of how each study prediction/question will be analyzed (as well as possible contingency plans).

The most common misconception about registered reports is that they are inflexible—that authors are “locked in” to their Stage 1 study proposal and as a result, are discouraged (or even penalized) for any deviations from that accepted proposal. Scientific discoveries are not always so planned and likewise, many of us have had experiences where “on paper” study design was given a harsh reality check by...well, reality.

Authors are in fact allowed to make post hoc changes, but as part of their registered report are required to make clear distinctions between their a priori planning and post hoc discoveries, and to provide a deeper justification for any post hoc changes. We strongly encourage authors to contact the journal editors prior to making changes however, to ensure that the changes can be sufficiently justified.

Another misconception is that registered reports cannot be used for exploratory research. However, even inductive research approaches can and do stem from careful planning and preparation and thus, those plans could be formalized in a Stage 1 review document.

One of the primary benefits to publishing registered reports is that they help authors plan their research agendas in advance, including securing a “publication guarantee” in that planning process. This is especially useful for colleagues working with extramurally funded research or otherwise working to establish a citation record for a series of studies.

Another benefit to registered reports is that authors are able to get formative feedback from peer reviewers and journal editors—it is not uncommon for Stage 1 submissions to receive mild critiques, suggestions, and/or requests that help authors avoid making poor theoretical, methodological, or analytic plans before they become “fatal flaws” later in the process. Related to this, registered reports can also protect us from ourselves, as they somewhat reduce the “researcher degrees of freedomopens in new window” available for a given study.

Submit your registered report to TMB