Also in this Issue

NIH Modifications to Peer Review Moving Forward

A new Implementation Plan calls for NIH to acknowledge more formally the efforts of all reviewers, and make review experience more intellectually rewarding by focusing discussion on impact/originality of proposals; ranking proposals at meeting’s conclusion; and engaging study sections to pilot many of these interventions.

By Patricia Kobor

Lawrence Tabak, Director of the National Institutes of Health (NIH) National Institute for Dental and Craniofacial Research and Co-Chair of the NIH Peer Review Working Group, presented the NIH Implementation Plan on Peer Review to the Advisory Council of the NIH Director (the ‘ACD’) on Friday, June 6, 2008. The plan he presented was the most explicit distillation yet seen of suggestions taken from the committee’s discussions with outside scientific groups and internal NIH stakeholders. For a bit of history on APA’s involvement with this issue, please see the May issue of SPIN Science Policy Insider News. [now APA Science Policy News]

Below are details from the Implementation Plan. Tabak and Co-Chair Jeremy Berg, of the National Institute of General Medical Sciences, have been discussing this draft plan with groups including Study Section Chairs, the Peer Review Advisory Committee, and the Institute and Center directors.

Four core priorities emerged from all the discussions, and the goals are based upon them. NIH should:

  1. Engage the best reviewers;

  2. Improve the quality and transparency of reviews;

  3. Ensure balanced and fair review across scientific fields and career stages, and reduce burden on the applicants; and

  4. Develop a permanent process for continuous review of peer review.

Goals regarding the first priority include increasing flexibility of service to better accommodate reviewers. NIH will likely spread a 12-session review commitment over a longer period of four years or so to make study section service less onerous, and allow duty-sharing by colleagues as appropriate. Also, NIH will expand and pilot flexible submission deadlines. Regarding the need to recruit additional reviewers, some categories of grants will be awarded with the expectation or requirement of review service (e.g., Merit/Javits Awards, Pioneer Awards, and cases in which a Principal Investigator [PI] is named as PI on 3 or more awards).

The plan calls for NIH to acknowledge more formally the efforts of all reviewers, and make review experience more intellectually rewarding by focusing discussion on impact/originality of proposals; ranking proposals at meeting’s conclusion; and engaging study sections to pilot many of these interventions.

The plan acknowledges that there was considerable support in the public comments for compensating the time and effort of reviewers who serve for 18 full study section meetings as chartered members, or equivalent service. This compensation may take the form of allowing reviewers to apply for an administrative supplement of up to $250,000 for their own grants and allowing reviewers to request that they be considered for Merit/Javits awards on a competitive basis.

Changes in Scoring and Criteria

Regarding the need to improve quality and transparency of reviews, the plan acknowledges research findings of psychometricians that reviewers cannot cognitively manage a 41-point scale; reliability would improve with a more manageable 7-point scale.

NIH will modify the rating system to focus on specific review criteria with less emphasis on methodological details and more on scientific impact. The five specific review criteria will be: impact, investigators, innovation/originality, project plan/feasibility, and environment. Each application will be scored for each criterion, plus receive a global score.

NIH will restructure the summary statement accompanying each review to align with explicit rating criteria. It will provide an optional field for reviewers who wish to provide applicants with additional advice (e.g., an opinion as to whether applicant should resubmit). Tabak pointed out that this section would be optional and not part of the actual review. One comment from an ACD member was that now there is often the expectation that if the reviewer’s advice is followed, the score will improve on resubmission, but that doesn’t always happen and shouldn’t be expected.

Another modification is that NIH will shorten and redesign applications to align with updated review criteria. The application will be 12 pages for R01s, with an appendix of up to 8 pages for specific items, like elements of a clinical trial.

The implementation plan emphasizes that peer review should not disadvantage early career investigators. It should apply appropriate weighting for past performance and future potential. It should be designed to minimize need for repeated or multiple applications. It should encourage “transformative” research.

It was noted that the average age of new R01 investigators in 1980 was 37.2, while in 2006 it was 42.2. From 2002 to 2007, the number of scored NIH applications remained roughly the same, but 535 fewer applications from first time investigators were scored.

Ideas to treat Early Stage Investigators (ESI) more fairly in peer review include: Cluster review, discussion, scoring and ranking of ESI applications within a study section; pilot percentiling ESI across all study sections; and work to ensure that the number of fully discussed proposals from ESI is not disproportionately reduced.

For more experienced investigators, the plan calls for placing equal emphasis on retrospective assessment of accomplishments and a prospective assessment of what’s being proposed. It also calls for clustering review, discussion, scoring and ranking of clinical research applications within a study section.

The plan proposes that NIH encourage ‘transformative’ research by encouraging and expanding on Pioneer, EUREKA and New Innovator awards.

To help reduce the burden on grant applicants, reviewers and NIH staff, the plan proposes that NIH reduce the need for resubmissions by applicants that are likely to be funded ultimately, and by those who are unlikely to be funded ultimately. It calls on NIH to establish policies to carefully balance success rates among A0 (first submission), A1 (first revision) and A2 (second revision) submissions to increase system efficiency.

Continuous Review of Peer Review

The plan calls for NIH to establish an infrastructure to continuously review the peer review system to ensure it adequately supports cutting-edge science. NIH would pilot and evaluate new models of review, (e.g., 2-stage review -- the so-called ‘editorial board’ model) and pilot and evaluate different methods for ranking the relative merit of applications. It also calls for NIH to develop metrics for monitoring performance of review.

How soon will these changes be implemented? An ad hoc Peer Review Task Force, to be chaired by NIH Deputy Director Raynard Kington, will be formed to develop detailed plans and oversee initial implementation. A new entity will be formed within NIH to oversee the Continuous Review of Peer Review.

The Science Government Relations Office will keep you updated about any new developments via Psychological Science Agenda and SPIN, so we encourage you to open those emails!