Inappropriate and Appropriate Selection of 'Peers' in Grant Review
(JAMA. 1994;272:114-116)
Stanton A. Glantz, PhD, Lisa A. Bero, PhD
Objective.--To assess the members of the California
Tobacco Related Diseases Research Program Behavioral and Public Health
Research on Tobacco Study Section and those of the Agency for Health
Care Policy and Research (AHCPR) Dissemination Study Section as
"peers" to review tobacco policy research. Both study sections
reviewed a similar grant application on tobacco policy research written
by one of us (S.A.G.).
Design.--Search of MEDLINE for 1989 through 1993 with the
keyword tobacco for Tobacco Related Diseases Research Program
and AHCPR reviewers. As a control, the National Institutes of Health
Cardiovascular Study Section, which reviewed a ventricular function
grant submitted by the same author with the keyword heart, was
analyzed.
Setting.--Not applicable.
Patients or Other Participants.--Study section members.
Interventions.--None.
Main Outcome Measures.--Publications by study section
members in areas germane to the proposal being reviewed.
Results.--Six (33%) of 18 Tobacco Related Diseases Research
Program reviewers had no "tobacco" publications (median, two
publications; interquartile range, zero to four). The members'
"tobacco" publications concentrated on well-controlled experimental
interventions on smoking cessation and prevention strategies, not
tobacco policy. Only one member had primary expertise in tobacco policy
research. None of the AHCPR reviewers had "tobacco" publications.
All 31 (100%) of the National Institutes of Health reviewers had
"heart" publications (median, nine publications; interquartile
range, seven to 19). Five members had a primary interest in the subject
of the National Heart, Lung, and Blood Institute application.
Conclusions.--Study section members' professional interests
play a critical role in the level of interest and enthusiasm they will
have for a proposal, which affects the priority score. In contrast to
the study section that reviewed the heart grant, the study sections
that reviewed the tobacco control grant were not "peers." The
membership of these review committees has effectively precluded
research on tobacco control policy.
(JAMA. 1994;272:114-116)
THERE IS A strong consensus among tobacco control
professionals that public policy interventions hold great potential
for reducing the burden of tobacco-induced disease and
death.[1] As a result, both the State of California
Tobacco Related Disease Research Program
(TRDRP)[2] and the National Cancer Institute (NCI)[3] have solicited research on
tobacco control policy. Despite this expressed interest, these
organizations have not actually supported much research in this area.
Peer review is the centerpiece of the modern scientific review process,
including decisions regarding grant funding. The cornerstone of the
peer review process is that work is reviewed by peers. The dictionary
definition of peer is "a person or thing of the same rank,
value, quality, ability, etc.; equal; specifically, an equal before the
law."[4] In scientific terms, a "peer" would be an
individual with expertise and knowledge in the area in question that
meets or exceeds that of the applicant. To function appropriately, the
people selected as peers must not only have methodologic depth, but
also a broad understanding of the field in question, particularly the
questions that are important to address. Without this breadth,
innovative work is often passed over purely on the basis of
methodologic considerations.[5] Thus, the composition of the
study section becomes the key issue in assessing whether a grant
receives peer review. Review of the membership of the TRDRP Behavioral
and Public Health Research on Tobacco Study Section demonstrates that,
while suitable for reviewing experimental trials of smoking cessation,
it does not represent a committee of peers for research in the area of
tobacco control policy research. As a result, the study section applies
inappropriate experimental standards to policy research and underrates
its importance. A similar situation exists at the National Institutes
of Health (NIH).
MATERIALS AND METHODS
Publications of Study Section Members
This report presents the results of the analysis of the peers selected
to review three grants submitted by one of us (S.A.G.). Two of these
proposals, one submitted to TRDRP and one submitted to the NCI, were in
the area of tobacco policy research. The NCI proposal was referred to
the Agency for Health Care Policy and Research (AHCPR) for review. One
proposal, submitted to the National Heart, Lung, and Blood Institute,
was in the area of basic studies of cardiac mechanics. None of these
proposals initially received priority scores high enough to receive
funding. (Both proposals were subsequently funded after revision and
resubmission.)
A widely accepted, although imperfect, measure of scientific expertise
is publication in the peer-reviewed scientific
literature.[6] [7] To assess the qualifications of the members
of the study sections as peers in the area of tobacco policy research,
we searched the National Library of Medicine MEDLINE database for the
period from January 1, 1989, through May 26, 1993, for members of the
TRDRP Behavioral and Public Health Research on Tobacco Study Section
and AHCPR Dissemination Study Section. This database contains
1,541,832 citations from medical and health sciences
journals. The searches were performed by means of the Uni- versity of
California Melvyl MEDLINE system with the following command: Find pa
[personal author] StudySectionMember and kw [keyword]
tobacco. Although this search was not exhaustive, it could be
expected to give some indication of the areas related to tobacco in
which members of the study section qualified as experts.
As a control concerning the qualifications of the members of these
study section, we compared them with the NIH Cardiovascular Study
Section, which reviews the applicant's NIH grant for basic research in
ventricular function, substituting the term heart for
tobacco in the search.
Nature of TRDRP Awarded Grants
To investigate whether biases inherent in the selection of study
section membership influences the overall portfolio of grants awarded
by TRDRP, we reviewed all grants awarded for three annual funding
cycles (fiscal years 1990, 1991, and 1992) and categorized each one
into one of the following mutually exclusive categories, by means of
the published abstracts[8]
[9]
[10]
of the grants: policy:
grants that have direct impact on the formulation or measurement of
tobacco control policies, such as studies of taxation or implementation
of indoor air regulations; environmental tobacco smoke (ETS):
grants that study the health effects or the physical distribution of
ETS (these include basic science studies, clinical studies, and
engineering studies); epidemiology: studies that use
epidemiologic techniques (eg, case control, cohort studies) to examine
the relationship between active smoking and disease or factors that
contribute to smoking (studies of ETS epidemiology were categorized as
ETS); prevention/cessation: evaluation of tobacco prevention
or cessation programs, implementation of prevention or cessation
programs, and studies that identify factors that contribute to the
prevention or cessation of smoking (eg, a study of social factors among
Asian Americans that contributed to the cessation of smoking);
nicotine/addiction: grants that study the physiologic or
pharmacologic actions of nicotine, including cellular pathways (eg,
comparison of nicotine and caffeine and models of addiction; this
category included basic science studies and clinical studies); and
disease: all biomedical research on diseases that is related
to tobacco, but that might not be directly linked to tobacco use (eg,
general mechanisms of cancer genesis, but not necessarily lung cancer).
This category is divided into two subcategories: clinical and basic
science. Clinical research includes studies in humans that look at the
effects of tobacco smoking on the pathophysiologic characteristics of
certain diseases or the effect of pharmacologic interventions to treat
disease. Basic science studies include animal, in vitro, cellular, or
molecular studies that explore the mechanisms of action underlying some
of the disease processes that have been related to smoking, such as
seeking gene markers for atherosclerosis or cancer.
RESULTS
Publications of Study Section Members
Of the 18 members of the TRDRP Behavioral and Public Health Research
on Tobacco Study Section, six (33%) had no publications indexed on
tobacco, and two had only one citation. The median number of
citations was two, and the interquartile range was zero to four.
Examination of the publications of those members who had publications
indexed with the term tobacco disclosed that these
publications were concentrated in clinical and school-based smoking
cessation and prevention and smokeless tobacco use. The primary focus
was on well-controlled experimental interventions, such as randomized
controlled trials of smoking cessation and prevention strategies. This
expertise and publications in well-controlled experimental
interventions are not particularly relevant for assessing the methods
of a tobacco policy application. Only one member had primary expertise
in tobacco policy research (two publications). During the same period,
the applicant had seven publications indexed on the keyword
tobacco; only one member of the study section had more than
seven publications indexed on tobacco.
Of the 19 eligible members of the AHCPR Research Dissemination Study
Section, none had citations indexed on the term tobacco.
Of the 31 members of the NIH Cardiovascular Study Section, all (100%)
had publications related to the heart. The median number of citations
was nine, with an interquartile range of seven to 19. Five members had
a primary interest in the area of ventricular function, the subject of
the applicant's National Heart, Lung, and Blood Institute application.
During the same period, the applicant had 11 publications indexed on
the keyword heart; 13 (42%) of the Cardiovascular Study
Section members had more than 11 publications. Thus, all members of the
study section had demonstrated expertise in heart research, with a
number of publications similar to that of the applicant whose work they
were reviewing. Nearly half of the members of the study section had
more publications than the applicant. Five members had direct
experience in the area of the application.
Nature of TRDRP-Funded Grants
As shown in the Table, funds allocated to areas directly related to tobacco (policy, ETS, epidemiology,
prevention/cessation, and nicotine/addiction) have steadily fallen from
54% of all funds in cycle 1 to 30% of all funds in cycle 3, with
funding for general biomedical research increasing from 47% to 70% of
funds. The decline in resources allocated to policy has been
particularly precipitous, from 8% of funds to only 1%. Similar,
although slightly less dramatic, changes have taken place when the
analysis is based on the number of grants.
COMMENT
The differences between the study sections that reviewed the
tobacco policy grants and the Cardiovascular Study Section are
extremely important in ensuring a real peer review.[11] The
professional interests of the members of a study section play a
critical role in the level of interest and enthusiasm they will have
for an investigation of a given question. These judgments affect how
important the reviewers consider the questions to be investigated, and
this judgment directly affects the priority score. By this standard,
the NIH Cardiovascular Study Section members are "peers" for the
reviews of basic investigations of ventricular mechanics. In contrast,
TRDRP had the tobacco policy grant reviewed by a study section whose
primary area of expertise was smoking cessation and prevention
interventions with a strong experimental orientation; the presence of a
single member whose primary interest was in tobacco control policy is
not enough to overcome the biases intrinsic in this membership. It is
interesting that in the first year, TRDRP had a separate study section
devoted to tobacco policy grants; in that year a much higher portion of
funds went to tobacco policy grants than in the second or third years.
A similar situation exists at the NIH, including the AHCPR study
section that reviewed the grant for the NCI. Most tobacco grants go to
the Behavioral Medicine Study Section. This study section is renowned
as a graveyard for tobacco control research. (Because of this
reputation, the applicant requested that the Behavioral Medicine
Study Section not review the tobacco control grant.) According to the
Division of Cancer Prevention and Control Board of Scientific Advisors,
of the past 41 cancer prevention and control grant applications
submitted to the Behavioral Medicine Study Section, not one has been
funded.[12] The board found (written communication from M.
Alfred Hayes, MD, chairman, Board of Scientific Counselors, Division of
Cancer Prevention and Control, NCI, to Bernadine Healy, MD, director,
NIH, June 19, 1993) that this problem extended beyond tobacco control
policy research to prevention issues in general and concluded that
"at this time there is no true peer review of extramural
applications for community-based cancer prevention and control
research. This serious omission in the NIH peer review system is
serving to bar an entire field of research from NIH funding, therefore
retarding the advancement of knowledge in an area critical to improving
the nation's health."
Although assignment of a grant proposal to a study section
composed of peers is crucial for a quality review of the proposal, the
selection of the study section by the Division of Research Grants at
the NIH appears to be somewhat arbitrary. Both of us responded to the
NCI program announcement seeking proposals on tobacco control policy
research.[3] The application by one of us (S.A.G.) was to
study the political activity of the tobacco industry in response to
tobacco control efforts, particularly as they relate to clean indoor
air legislation designed to reduce ETS exposure by the health
community. The proposal by the other of us (L.A.B.) was to examine the
quality and use of tobacco industry-sponsored research on ETS.
Although the specific topics of the two proposals were different, they
both focused on the activities of the tobacco industry to influence
policy regarding ETS. It would have been appropriate for both grants to
have been reviewed by similar peer reviewers. However, S.A.G.'s grant
was assigned for review by the AHCPR Research Dissemination Study
Section, and the Division of Research Grants declined to review
L.A.B.'s grant. After a formal protest and a 6-month delay, the grant
was assigned to the AHCPR Health Services Development Grant Study
Section. Like the Research Dissemination Study Section, none of the 18
members of the Health Services Development Grant Study Section had any
publications indexed on tobacco. The assignment of the two
grants to different study sections, neither of which had expertise in
tobacco policy research, suggests that the qualifications of the study
section members as peers to review the proposals were not considered by
the Division of Research Grants.
The membership of the Behavioral Medicine Study Section is similar to
that of the TRDRP study section. The selection of reviewers for TRDRP
and resulting biases in the peer review process have led to an
important lost opportunity. We used a generous criterion for assessing
expertise in tobacco, by using tobacco rather than
tobacco policy as the criterion for evaluating publications of
study section members. While every reviewer need not be an expert on
tobacco, a reasonable number of members should have a broad
understanding of the field in question, not just methodologic
expertise. Even with this loose standard, few members of the TRDRP
study section, and none of the AHCPR study section members, were
"peers" in the area of tobacco-related research in general, much
less tobacco control policy research in particular. The TRDRP is the
only research program in the world devoted to tobacco. As such, it was
in a unique position to nurture a strong research environment on
tobacco-related problems. Instead, TRDRP has been evolving into simply
one more place for biomedical scientists to seek funding for their work
because tobacco causes so many diseases.
In spring 1993, the California legislature recognized the problems with
the TRDRP and amended the authorizing legislation to require TRDRP to
give high priority to research in tobacco control policy.[13]
In response, TRDRP issued a request for proposals in tobacco policy
research and convened a new study section with expertise in tobacco
policy to review these applications. The governor vetoed this bill on
September 25, 1993, saying that the program should be reevaluated in
the context of all tobacco control programs funded by the California
tobacco tax.
AFTERWORD
One of us (S.A.G.) revised and resubmitted both grants described in
this article to the NIH. The ventricular mechanics grant, which
initially received a 16th percentile priority score and was not funded,
received an 11th percentile score and was subsequently funded. The
tobacco control policy grant was resubmitted together with a letter
making essentially the same arguments concerning problems with the peer
review it received. The priority score of the original grant was 92nd
percentile; the revised grant received a ninth percentile score and was
funded. An appeal was filed with TRDRP making essentially the same
arguments concerning problems with the peer review process; TRDRP
rejected this appeal and did not grant reconsideration of the proposal.
From the Department of Medicine, Institute for Health Policy Studies
(Drs Glantz and Bero), Cardiovascular Research Institute (Dr Glantz),
and Division of Clinical Pharmacy (Dr Bero), University of
California-San Francisco.
Presented in part at the Second International Congress on Peer Review
in Biomedical Publication, Chicago, Ill, September 10, 1993.
This study was supported by funds provided by the Cigarette and
Tobacco Surtax Fund of the State of California through the
Tobacco-Related Disease Research Program of the University of
California (awards IRT 520 and 2KT 072).
Reprint requests to Division of Cardiology, Department of Medicine, Box
0124, University of California, San Francisco, CA 94143-0124 (Dr Glantz).
References
1. Chapman S. Upgrading the academic respectability of
advocacy studies. Tobacco Control. 1993;1:81-83.
2. University of California Tobacco Related Diseases
Research Program, Call for Proposals. Oakland: University of
California; 1992.
3. Division of Cancer Prevention and Control Research
Program Announcement (PA-92-61). In: NIH Guide for Grants and
Contracts. Bethesda, Md: National Institutes of Health; March 27,
1992.
4. Neufeldt V, Guralnik DB, eds. Webster's New World
Dictionary. New York, NY: Webster's New World Dictionaries;
1986:996.
5. Horrobin DF. The philosophical basis of peer review and
the suppression of innovation. JAMA. 1990;263:1438-1441.
6. Knoll E. The communities of scientists and journal peer
review. JAMA. 1990;263:1330-1332.
7. Garfunkel JM, Lawson EE, Hamrick HJ, Ulshen MH. Effect
of acceptance or rejection on the author's evaluation of peer review
of medical manuscripts. JAMA. 1990;263:1376-1378.
8. Tobacco Related Diseases Research Program. Annual
Report to the State of California Legislature, 1991. Oakland:
University of California Office of Health Affairs; 1991.
9. Tobacco Related Diseases Research Program. Annual
Report to the State of California Legislature, 1992. Oakland:
University of California Office of Health Affairs; 1992.
10. Tobacco Related Diseases Research Program. Annual
Report to the State of California Legislature, 1992. Oakland:
University of California Office of Health Affairs; 1993.
11. Cohen J. Study sections: does a superb system need a
tune-up? Science. 1993;261:1678-1679.
12. DCPC advisors to consider seeking prevention and
control study section. Cancer Lett. 1993;19(20):5.
13. Senate Bill 1088, as amended, August 15, 1993.
Table of Contents
