Peer-review overhaul nears completion

Social and behavioral researchers face a new group of review panels next winter.

By Beth Azar
Monitor staff

After six years of anxious anticipation, come this winter, social and behavioral researchers looking for money from the National Institutes of Health (NIH) will have their grant applications evaluated by a new group of review panels.

Gone will be NIH?s eight social and behavioral review groups?commonly known as study sections?as well as most of the in-house study sections of the National Institute of Mental Health (NIMH) and the National Institute on Drug Abuse (NIDA).

Over the next six months, an NIH working group will finalize a new set of study sections based on a template created in June by a panel of 60 social and behavioral researchers. After the working group hones the template, it will post the proposed study sections on the Internet for comment by the general scientific community (look for it in late August at http://www.csr.nih.gov). Based on those comments, the working group will create a final set of study sections this fall, in time for researchers writing grant proposals for the February deadline.

The restructuring of social and behavioral peer review stems from the need to integrate NIMH and NIDA grant applications into NIH?s main review system (see sidebar for details), which is managed by the Center for Scientific Review (CSR). And most researchers have approached the process with concern and skepticism, says Paula Trubisky, the APA public policy officer tracking the changes. But all of the researchers polled about their experience at the June meeting say they walked away with their fears soothed and their doubts suspended.

'I thought that [the process] was remarkably fair,' says psychologist Warren Bickel, PhD, professor and vice chair of research in the department of psychiatry at the University of Vermont. 'NIH brought together a bunch of scientists and said, ?Go create review committees.? There was no agenda other than that.'

'There was a really remarkable coherence of interest,' adds psychologist Charles Snowdon, PhD, of the University of Wisconsin?Madison. 'We all worked very well together and made a good faith effort to cooperate, even across a wide range of approaches to the study of behavior. And, in the end, there was no conflict we weren?t able to work out.'

Everyone gets a home

NIH began the behavioral and social sciences review restructuring by forming an internal working group representing each of the 12 institutes that fund social and behavioral research as well as CSR and the NIH Office of Behavioral and Social Sciences Research (OBSSR). The group will formulate the final set of study sections based primarily on recommendations from researchers.

In particular, a group of 60 researchers met at NIH June 10?12. Their goal was to examine 1,000 social and behavioral grant application abstracts from proposals submitted in a single round of review last year and cull them into what they considered appropriate categories. From there, they created study sections to handle each category, including a description of the research each would review and the composition of the researchers who would ideally sit on the panel. The only rules were to avoid institute-specific study sections by organizing review groups around research methods or general social or behavioral topics; and to build some overlap and redundancy into the study sections so that at least two could review most grant applications.

The result was 16 proposed study sections, with a handful of subsections, including 'cognition and perception,' 'social motivation, emotion and social cognition' and 'developmental disorders and child psychopathology.'

Ensuring a fair review

NIH is reviewing the suggestions, reducing overlap and filling in any obvious gaps in coverage. Later this summer, it will take another set of grant applications and sort them into the revised study sections to ensure all can be placed and fairly reviewed.

All types of social and behavioral research should be well represented in the new review groups, says a randomly polled sample of researchers who attended the June meeting.

'I had a sense that everyone there had a home at the end of the day,' says Bickel, who adds that the working group included researchers from a remarkably wide range of research areas.

Indeed, just among the psychologists?who represented more than half the panel?were Jeffrey Alberts, PhD, who studies prenatal influences on rat pups; Bickel, who represents behavioral economics, with work on animal models of reinforcement; James Blumenthal, PhD, who works in behavioral cardiology; Joseph Campos, PhD, who studies emotional development in children; David Dinges who studies sleep chronobiology; and Sharon Landesman Ramey, PhD, who studies interventions to improve cognitive skills in young children.

Anyone on the panel who had a concern that a certain line of research was poorly represented or spread too thin across several study sections worked with the group to develop a solution, says panel member K. Daniel O?Leary, PhD, distinguished professor of psychology at the State University of New York at Stony Brook.

For example, researchers who study animal behavior and learning banned together to successfully argue for inclusion of a study section that concentrates on 'behavioral processes and integrative systems' in humans and nonhumans, says Snowdon.

Other researchers recommended special recognition for new and emerging areas of research. 'Some areas are not known to enough people to have competent reviews by a general-interest review panel,' says O?Leary.

To address this issue, 'we will work out a way to cluster some areas so there?s sufficient expertise in at least one study section,' says Virginia Cain, PhD, special assistant to OBSSR?s director and a lead player in the reorganization. 'Violence, for example, will have people in two or three study sections so we get a critical mass.'

Another area the researchers singled out for special treatment was research on statistical models and methodologies. Traditionally, such research is tacked onto other grants and reviewed by topic-specific study sections. But the researchers at the NIH meeting voted overwhelmingly to create a special study section for methodology research, even though none of the applications they had to sort would have fallen in that area.

'Not many people submit applications to evaluate statistical modeling techniques,' says O?Leary. 'But as these techniques become more complex, there may be a need to fund pure methodology research.'

The panel hopes that researchers will send in applications once they realize there?s a place to have them reviewed.

End notes

Although NIMH and NIDA have retained some internal review groups?including treatment and health services research?by the end of the year, most behavioral and social sciences researchers will have their grant applications processed through CSR.

As soon as the composition of the new review groups is final this fall, CSR will post their descriptions on the Internet, along with any information it has on who the reviewers will be.

These descriptions will be critical for researchers writing grant proposals for the February deadline, and even more so because NIH is now allowing researchers to select which study section they would like to review their applications, says Anita Miller Sostek, PhD, chief of CSR?s biobehavioral and social sciences initial review group. Review staff will honor a researcher?s request unless it simply isn?t appropriate.

Until then, Bickel encourages researchers who might be concerned about the changes at NIH to get involved in the dialogue by examining the proposed new study sections when they?re posted on the Internet later this month.

'The train is leaving the station, and those who aren?t on board will be left behind,' he says.

Reorganizing social and behavioral grant review at the National Institutes of Health

1992

The National Institutes of Mental Health (NIMH), Drug Abuse (NIDA) and Alcohol Abuse and Alcoholism (NIAAA) join the National Institutes of Health (NIH). A legislative mandate gives them until October 1996 to integrate their grant review into the NIH Division of Research Grants (DRG)?now the Center for Scientific Review (CSR).

Early 1995

NIMH and NIDA conduct some studies and find that a 1995large percentage of their grants would not fit into existing NIH review groups.

NIAAA forges ahead with talks with DRG.

June 1996

NIAAA combines three of its four grant-review groups with three DRG review groups.

Winter 1996

NIMH and NIDA begin formal discussions with DRG about the integration process. An NIH Peer Review Oversight Group is formed. The NIH Neuroscience Integration Working Group begins examining ways to restructure NIH neuroscience review to include NIMH and NIDA grant applications.

Fall 1997

The Neuroscience Integration Working Group creates a new group of neuroscience study sections based on months of discussions and input from the scientific community.

NIH creates a Behavioral and Social Sciences Review Integration Working Group.

February 1998

Twenty-four researchers meet at NIH for a preliminary meeting on how to reorganize the behavioral and social sciences study sections to incorporate NIMH and NIDA grant applications.

June 1998

Sixty researchers meet at NIH and formulate a working draft of the new behavioral and social sciences review groups.

The new neuroscience study sections review their first round of grant applications.

August 1998*

The NIH Behavioral and Social Sciences Review Integration Working Group will draft a revised version of the review groups formulated in June and test them with an experiental sorting of applications from last year.

CSR will post on its web page a revised version of the behavioral and social sciences review groups for public comment (http://www.csr.nih.gov).

Fall CSR will post the final version of the review groups.

February/March 1999*

Reseachers will submit the first set of grants that will be reviewed by the new study sections.

June 1999*

The new behavioral and social sciences study sections will review their first round of grant applications.

*predicted