Guidelines for Reviewing Manuscripts for Training and Education in Professional Psychology®

Guidelines for Reviewing Manuscripts for Training and Education in Professional Psychology

Overview

The Training and Education in Professional Psychology® (TEPP) editorial team has prepared the following guidelines to assist you in reviewing a manuscript submitted to the journal for publication. This statement provides an overview of issues of concern when reviewing manuscripts for TEPP. The TEPP editorial team hopes that these guidelines are useful to you and appreciates your contributions to our journal and profession.

This document was originally developed by Brent Mallinckrodt, PhD, Editor, Journal of Counseling Psychology. It was revised by members of the TEPP Editorial Board and Staff.

Confidential Communication

Please note that the manuscripts you will review are confidential communication. An unpublished manuscript is entitled to copyright protection from the moment it is fixed in tangible form, for example, typed on a page. Therefore, you may not circulate, quote, cite, or refer to the unpublished work, nor may you use information from the manuscript to advance your own work or instruction unless you obtain specific permission for such use from the author.

You must request permission in advance from the Action Editor to share the manuscript with any other person, for example, if you wish to seek assistance from a colleague in preparing your review.

After completing your review, you may retain one copy of the manuscript until such time as the Action Editor sends you a copy of the editorial decision letter and other reviews of the manuscript. Within 48 hours of receiving these materials, you must destroy or delete any remaining copies of the manuscript.

Conflict of Interest
Although TEPP uses a masked review process, if you suspect that one of the manuscript's authors is a person whose relationship to you might present a conflict of interest, for example (but not limited to), a recent collaborator, faculty colleague, or student; or if the acceptance or rejection of this manuscript might result in your own financial gain, you must inform the Action Editor immediately so that the manuscript can be promptly reassigned.
Writing a Review

Writing a review for TEPP involves two relatively independent tasks—acting as a gatekeeper and acting as a consultant. In the first capacity, you are asked to make a recommendation to the Action Editor whether the manuscript should be accepted, rejected, or returned to the authors to revise and resubmit. In the second and equally important capacity, you are asked to provide a detailed, educative narrative evaluation that the Action Editor will send to the authors.

As a gatekeeper, you must render a judgment about whether this manuscript should eventually appear as a published article in TEPP, thereby becoming part of the permanent body of scientific literature that will influence the journal's readers for many years to come. Reviewers must remain constantly mindful of their obligation to future generations of researchers to make these judgments with all the wisdom, fairness, and fidelity that they can bring to the task.

The role of consultant is no less important than that of gatekeeper. The journal cannot accept every manuscript submitted, but we aspire to provide sufficiently high quality feedback so that authors whose manuscripts are rejected are encouraged to continue submitting their best work to the journal, and authors whose work is accepted receive valuable ideas for their next project. We encourage you as a reviewer to see yourself as an anonymous consultant for every author, even (and especially) for manuscripts you believe should be rejected.

Many experienced reviewers find that to render the most objective, rigorous recommendation about the publication value of a manuscript, they must temporarily set aside their consultant role, and conversely, to provide a sufficiently helpful narrative review for the authors, they must temporarily push to the background their gate-keeping role.

Our experience is that occasionally reviewers seem unable to transcend their gate-keeping role. The result can be a review that is infused with a "gotcha" tone, one that points out errors with no suggestions for improvement and shows a lack of respect for the authors as scientific colleagues. Of course, it is a basic task of reviewers to point out flaws in the manuscript, but this should be done with tact and respect for the authors.

Many reviewers find that they must read the manuscript with "two sets of eyes," scanning in one pass with an emphasis on their publication recommendation and in another pass with an emphasis on collecting points that will become the basis for their narrative evaluation.

With experience as a reviewer, you will discover which sequence works best for you and a way to more completely integrate the two roles.

Publication Recommendation

To further differentiate the roles of gatekeeper and consultant (and to make the Action Editor's job easier when quite discrepant recommendations are received), we ask that you communicate your publication recommendation only on the standardized evaluation form, which is not shared with authors. Please do not include an explicit recommendation about acceptance, revision, or rejection in your narrative evaluation that will be shared with the authors.

In making your publication recommendation, please consider these guidelines developed by the APA Publication and Communication Board:

To merit publication each manuscript must make an original, valid, and significant contribution to an area of psychology appropriate for the journal to which it is submitted. That is:
(1) A manuscript cannot have been published, in whole or in part, in another journal or readily available work (see APA Publication Manual, 5th ed., p. 347, for more information on primary, or original, publication).
(2) A manuscript must be accurate, and the conclusions and generalizations must follow from the data.
(3) A manuscript must be more than free of major fault—it must be an important contribution to the literature.
(4) A manuscript must be appropriate for the journal to which it is submitted.

For a manuscript not meeting all those criteria, you will usually recommend rejection, with detailed reasons for your recommendation.

As you consider your publication recommendation to the Action Editor, it may be helpful to answer three sequential questions:

1. Is the topic of the manuscript appropriate for TEPP?
In addition to manuscripts dealing with empirical studies, TEPP is interested in those that extend the theory, research, and practice of education and training through a conceptual/theoretical focus that is grounded in the existing literature and that has clear implications for professional education and training.

Literature reviews that have a strong conceptual focus are of interest. Personal narratives may at times be appropriate, particularly if they are embedded in a paper that focuses on broader conceptual issues.

If the Editor believes that a manuscript is clearly outside the scope of the journal, it can be rejected without peer review. However, you may receive a manuscript to review because the Editor has some question about its appropriateness for TEPP.

2. Can whatever flaws this manuscript has be remedied in a revision?
TEPP editors prefer to invite a revision if there is a reasonable chance that all the serious issues could be successfully addressed and that the revised manuscript has the potential to make a significant contribution to the literature (see the question #3 below).

Conversely, a study that has been brilliantly executed except for one major problem will be rejected if the problem constitutes a "fatal flaw." The crucial point is that your recommendation to reject the manuscript or invite a revision should hinge primarily on your judgment about whether it is possible to address all the major flaws you have found in a revision.

Even if you believe the process would be arduous (perhaps further data collection is required) or that the current version of the manuscript is very far from ideal, consider a recommendation to revise if you believe that there is a reasonable chance these issues could be eventually addressed. On the other hand, no matter how much you see to otherwise commend a manuscript, your recommendation should be to reject if you find a "fatal flaw" that seriously limits the scientific yield of the study, and you do not believe it would be possible to address this flaw in a revision.

3. Does the manuscript make a significant scientific contribution?
The "fatal flaws" discussed in the previous section disqualify a manuscript from publication because the problem sharply limits the scientific contribution of the study. However, TEPP receives a high proportion of studies that pass the first two screening questions. The topic of these studies is appropriate for the journal, and they are free from all major flaws.

For these studies, the final criterion for publication is whether the manuscript describes findings that represent a significant contribution in this area of the literature. If you are a content expert in the topic(s) addressed by this study, your evaluation of this question will be especially valuable to the action editor. As a researcher in this area, do you view these findings as important?

Narrative Evaluation

From the onset, we acknowledge that there are many ways to approach the task of reviewing a manuscript and that individual reviewers may disagree, perhaps strongly, with some of the points below. Our intention is not to impose rigid conformity but rather to provide general suggestions for the task of reviewing.

The "General Style," Organization," and "Introduction" subsections apply to both quantitative and qualitative manuscripts. These sections are followed by guidelines specifically tailored to each of these two general approaches.

General Style

The narrative should be phrased as a communication between you and the Action Editor about the manuscript. Please refer to the authors of the manuscript sparingly and, when doing so, use the third person exclusively. Critical feedback tends to be easier to accept when a review refers to some aspect of "the manuscript" and avoids phrasing in the second person.

Consider the difference in these two examples. Which would you rather receive as an author?

  • The manuscript could be improved considerably by updating the review of literature…The sample needs to be described in more detail so that readers can make a determination about its generalizability…A planned hierarchical multiple regression should be performed instead of a stepwise approach.
  • You need to update your literature review…You haven't described the sample in adequate detail for readers to make a determination about generalizability…Your choice of a stepwise hierarchical regression was inappropriate, instead, use a planned hierarchical regression.

Please note that your review is being sent to editors, the authors, and other reviewers and is a reflection of your professionalism. Please spell check your review and make sure it says what you mean to say before turning it in. Also, please keep in mind the space limitations of our journal; abstracts should be no more than 100 words and the text, not including abstract references, and tables, should not exceed 20 pages.

Organization

Many reviewers begin their narrative with a paragraph summarizing the study. This practice serves two useful functions. First, by providing a brief statement of your essential understanding of the study and its findings, authors are reassured that you have read the manuscript in detail. Second, this paragraph serves to remind you about the study weeks later when you receive copies of the editorial decision letter and other reviews.

Next, many reviewers add a paragraph or two commenting generally on the manuscript. The strengths of the manuscript should be described in some detail. It is important for authors to know what you think they have done well, together with your more critical comments.

Although the guidelines below generally follow the order of the components in a manuscript (i.e., introduction, method, results), after the opening paragraphs your narrative evaluation need not follow the same order, nor is it necessary for your evaluation to comment on each section of the manuscript.

In fact, Action Editors generally find reviews more helpful when all the most serious concerns are collated together into a single series of points early in the review, followed by a clearly separated section of less serious concerns. It is also very helpful if you number each substantive point or request for revision in your narrative. These numbers facilitate our reference to your review in our editorial decision letter.

Introduction

The basic task of this section of the manuscript is to make a persuasive case for the importance of this study.

  • Does the author articulate the goals and the contribution of the study in extending or addressing gaps in the literature?
  • Are research questions stated clearly and derived logically either from theory, conceptual framework, thorough literature review, anecdotal evidence, and/or clinical experience (or some combination thereof)?
  • In quantitative designs, are research questions/hypotheses well grounded in theory and previous research?
  • If hypotheses are proposed, is each one phrased as a falsifiable statement, and is each one logically derived from the theory and previous research presented?
  • If research questions are posed, is this choice justified in your view, or do you believe the state of knowledge in this research area warrants a specific hypothesis instead of a general research question?
  • For qualitative designs, are the research questions logically derived from previous research, theory, and/or experiential evidence?
  • Are the authors clear about whether their study is confirmatory/verification oriented or discovery oriented?
  • If the former, do the authors anchor their research questions in established theory?
  • If the latter, are the authors careful not to let theoretical postulates overly focus the research questions and limit discovery?

Method

One basic task of this section is to provide sufficient information so that future researchers could replicate this work.

In quantitative designs:

  • Have the sampling methods, demographic characteristics, and attrition of the sample been described in sufficient detail so that readers can reach an informed conclusion about what generalizations are possible?
  • Have the proportions of ethnic/racial minority members of the sample been adequately described?
  • When a sample of convenience is used, are the generalizations proposed by the authors reasonable?
  • How have the purposes of the study been described to participants, and how might these procedures influence the results?
  • Did considerations of statistical power contribute to an a priori determination of adequate sample size?
  • In your view, does this sample afford sufficient statistical power and can meaningful generalizations be drawn to the population of interest?
  • Have all variables been appropriately operationalized?
  • Are the measures used to assess them adequately described and appropriate for the participants in this sample?
  • Have appropriate psychometric characteristics (e.g., scoring, dimensionality, reliability, and validity) been reported for all measures and subscales used?

In experimental studies:

  • Have appropriate manipulation checks and experimental controls been included in the design?
  • Has the procedure for assigning subjects to conditions introduced possible confounds?

For qualitative studies:

  • Has the paradigm underpinning the research been clearly articulated; or, if not specified, do the research question, research design (method), data gathering, and analysis reflect a congruent paradigmatic approach?
  • Have variants of the method or analysis approach been cited?
  • Has the research method or design been clearly identified and justified as appropriate for the research purpose?
  • Has the researcher's stance in relation to the participants, community, and phenomenon been articulated?
  • Are the procedures involved in the qualitative method described thoroughly?
  • Are data analysis steps described in detail sufficient for understanding how results were generated and so that the analysis could be replicated?
  • Have the researchers provided for ways to check the adequacy of the analysis (e.g., audits, peer reviewers, triangulation of data, involvement of participants)?

Results, Statistical Analyses, Figures, and Tables

  • Do the results follow closely from the goals described previously; that is, have the researchers studied the questions set forth in the introduction and do they answer the research questions that have been posed?
  • Is each figure and table clear, accurately labeled, and essential, or could the material be presented more efficiently in text?
  • Is the material in each table or figure self-explanatory?
  • Does the text unnecessarily duplicate material that readers can glean more efficiently from the table?

For quantitative studies, manuscripts submitted to TEPP should conform to guidelines for reporting statistical analyses published in 1999 by the APA Task Force on Statistical Inference (American Psychologist, 54, 594–604). For example, reports of statistical significance should be coupled with an appropriate estimate of effect size, which must be reported for each statistical test of each hypothesis or research question. Reports of point estimates should be accompanied by appropriate confidence intervals. Reviewers not familiar with these guidelines should consult this resource.

  • Have the data been screened for coding errors and outliers?
  • Is the treatment of missing data appropriate?
  • Are the quantitative methods used the most appropriate choices for testing the hypotheses or research questions and for the nature of the data (e.g. random vs. fixed effects, longitudinal vs. cross-sectional)?
  • Have the authors demonstrated that requirements and underlying assumptions of each statistical test have been fulfilled by the data?

For qualitative studies

  • Is the Results section consistent with the defining paradigm and approach, producing results consistent with what was anticipated in the Introduction?
  • Do the results seem logical and clear to the reader given the detailed description of the procedures and given the detail and organization of the Results?

Discussion

  • Does the Discussion provide an integration of the findings, referencing literature and theory presented in the Introduction, rather than merely restating the Results?
  • Have the authors noted the unique contributions of this study to theory and method and all the important limitations of the study?
  • Is an integrative summary provided noting how the present study and its operating paradigm have advanced the science given previous research on the topic?
  • Have the authors developed conclusions and recommendations that are justified by the data and results, appropriately limiting and clearly identifying as speculative those inferences or conclusions that go beyond the data?
  • Has the author provided specific useful implications of their study for faculty, trainers and/or students?

Specifically for quantitative studies:

  • Is the discussion confined to interpretation of findings directly relevant to the hypotheses or research questions posed in the introduction, or do sections stray from this focus?
  • Are the interpretations justified by the findings?
  • Is language implying causal relationships, if it is used at all, used appropriately?
  • Are these results compared sufficiently with the results of other studies?
  • When results of this study differ from previous research, are plausible explanations offered?

Specifically for qualitative studies:

  • Has the author been clear in the results and discussion about the proper uses of qualitative findings (e.g., that findings are not generalizable)
  • Has the author avoided using generalized statements about people who have experienced the phenomenon (thus implying generalizability)?
  • Has the author distinguished between limits due to qualitative methods (e.g., smaller sample sizes, researcher subjectivity, etc., which are legitimate components of a study) and limitations of the study?