Ethics, Economics, and the Publication Policies of Major Medical
Journals
(JAMA. 1994;272:154-156)
Kevin Schulman, MD; Daniel P. Sulmasy, OFM, MD; Deborah Roney
Objective.--To evaluate aspects of the publication
process that may affect the quality of the literature in clinical
economics and biomedical ethics, and to learn about the policies of
medical journals regarding disclosure of relationships between
investigators and research sponsors.
Design.--Mail survey.
Participants.--Editors in chief of 15 major medical
journals.
Results.--Twelve editors responded to the survey. Ten
reported having statisticians among their editors, while only two had
health economists and none had ethicists. Clinicians in the specialty
field were almost always the primary reviewers of submissions, while
methodologists (statisticians, health economists, or ethicists) were
involved less frequently. Journals reported little knowledge of the
training of their reviewers in these fields. While nine journals
requested disclosure of the financial relationship between author and
sponsor, only one inquired whether the sponsor's written approval was
required prior to manuscript submission, and only one knew whether
there was an independent steering committee for the study.
Conclusions.--These findings suggest that the peer review
process can be strengthened to improve the quality of the medical
literature in clinical economics and biomedical ethics. Journal editors
also need to better understand the terms of research sponsorship
agreements.
(JAMA. 1994;272:154-156)
TWO TYPES of contributions are becoming more common
in medical literature--economic analysis and biomedical ethics. Since
these fields are developing rapidly, aspects of the peer review process
can dramatically affect the development of these fields by influencing
the publication of manuscripts submitted in these areas. Editors
without a firm grounding in these topic areas may inadvertently assign
the manuscript to peer reviewers who are not truly skilled in these
methodological disciplines. This could result in publication of
articles of inconsistent quality.[1]
[2]
[3]
Publication policies
regarding disclosure of the relationship between the investigator and
the sponsor can also influence the development of these
disciplines.[4]
[5]
[6]
Economic analyses are increasing in frequency in medical literature as
the rational use of medical resources has become a pressing issue in
most countries. However, the quality of this literature has been
questioned.[1,2] Published reviews have shown that many
articles lack some aspect of a methodologically sound economic
analysis. [1,2] One report suggested that only three of 77
articles published between 1978 and 1980 and between 1985 and 1987
adhered to six principles of economic analysis. [1] A review
of economic analyses of randomized clinical trials from 1966 through
1988 reported low ratings for quality and completeness. [2]
Reports such as these raise questions about the ability of the peer
review process to critically assess economic analyses of medical care.
Ethics articles are also becoming more prevalent in medical literature.
Medical journals, accustomed to reviewing primarily quantitative work
in the clinical and basic sciences, may not know the standards by which
they ought to judge the quality of original scholarly contributions
that use philosophical methods.[3] Philosophical analyses
differ substantially from editorials in terms of their contribution to
the field of medical ethics as an academic discipline. The field will
suffer if these different contributions cannot be appropriately
identified in the peer review process.
Many different types of relationships may exist between sponsors
of research studies and independent investigators. A sponsored research
agreement allows the author to retain control over the publication of
study results. A consulting agreement may not give explicit publication
rights to the investigator or author. Consulting agreements can also be
designed to require the author to submit the manuscript for publication
only after the sponsor reviews the manuscript and provides written
permission for publication. Generally, sponsored-research agreements
are designed to protect the academic integrity of the study. While the
research produced under these agreements may not appear different to
the journal or to the reader, the sponsorship agreement may serve to
limit the publication of findings that are "negative" from the
sponsor's perspective.[5] Relationships between investigators
and sponsors have been of special concern in the economic evaluation of
new medical technologies. [5,6]
We present herein the findings of a survey we developed and implemented
to assess aspects of the publication process that may affect the
quality of the literature in both clinical economics and biomedical
ethics.
METHODS
We set out to determine the peer review and publication policies
of the leading medical journals with regard to submitted economic
analyses and biomedical ethics manuscripts. This study had the
following three objectives: (1) to assess the expertise of journal
editors in assigning the peer reviewers to evaluate submissions in
clinical biomedicine, clinical economics, and biomedical ethics; (2) to
assess the expertise of the peer reviewers of submissions in clinical
biomedicine, clinical economics, and biomedical ethics; and (3) to
assess the degree of disclosure required by each journal for all
submitted manuscripts, including the relationship of the investigator
and the sponsor under the sponsorship agreement.
We developed a survey to address these issues in the peer review
process. This instrument was divided into six sections: review of
clinical manuscripts, review of economic manuscripts, review of ethics
manuscripts, qualifications of journal editors, qualifications of peer
reviewers, and the degree of disclosure of the terms of
sponsored-research agreements.
To assess the review process, we asked the journals about the
person responsible for assigning categories to manuscripts and about
the frequency with which different types of reviewers were assigned to
each manuscript category (that is, clinicians, methodologists,
statisticians, economists, or ethicists, and the frequency of joint
review with clinicians and methodologists). Respondents were asked to
score the frequency using a Likert scale, with 1 representing never and
5 representing always. In addition, for ethics manuscripts, we asked
editors to classify two sample abstracts as either opinion/editorial or
original contribution pieces. The correct classification of these
abstracts was based on assignments by three medical ethicists (Edmund
Pellegrino, MD, Carol Taylor, RN, MSW, and Kevin Wildes, SJ, PhD).
To assess the qualifications of journal editors, we inquired about the
presence of statisticians, economists, and ethicists on editorial
boards. To assess the qualifications of peer reviewers, we asked
editors to describe the qualifications of peer reviewers in statistics,
economics, and ethics using four descriptive categories: (1) MD only;
(2) MD with some training or experience in the field; (3) advanced
training (master's or PhD) in the field, with or without an MD; or (4)
advanced training (master's or PhD) in an area other than statistics,
economics, or ethics, with or without an MD.
To assess the degree of disclosure of the terms of sponsored-research
agreements, we asked specific questions about whether a journal
requires disclosure of funding sources, disclosure of the nature of the
relationship between the sponsor and the investigator (consulting or
sponsored-research agreement), disclosure of any requirement for
written approval by the sponsor, and/or disclosure of whether the study
had an independent steering committee. We also asked about the general
publication policies of the journal with respect to publication of
primary or secondary data collection studies conducted under consulting
agreements.
The survey was pilot-tested with an editor at one journal and then
revised. The final version of the instrument is available from the
authors on request.
The survey was sent to the editors of 15 leading English-language
medical journals, 12 in the United States, one in Canada, and two in
England. Seven journals were selected from representation on the
International Committee of Medical Journal Editors,[7] with
the remainder selected from among editors of specialty and subspecialty
medical journals. The survey was sent out to the editors in chief of
the journals. A second mailing was sent out 2 weeks after the first
mailing. Results are reported for responding journals and are presented
as means (+/-SDs).
RESULTS
Twelve of the 15 journals responded to our survey.
Clinicians in the specialty field were almost always the primary
reviewers of submissions, with mean scores of 4.6 (+/-0.5) for clinical
biomedical submissions, 4.3 (+/-0.9) for clinical economics, and 4.4
(+/-0.5) for bioethics. Methodologists (statisticians, health
economists, or ethicists, respectively) were involved less
frequently, with mean scores of 3.3 (+/-1.0) for biomedical
submissions, 3.7 (+/-1.1) for economics submissions, and 4.1
(+/-0.9) for ethics submissions.
When describing reviewers, journal editors reported little
understanding of the formal training of their reviewers in these
fields. Ranges for the proportion of reviewers with specific training
(master's or PhD in the field), with or without an MD, fell between
16% and 100% for ethicists, between 10% and 100% for health
economists, and between 50% and 100% for statisticians. Two journals
reported they had no data to address this issue or could not respond to
the question.
When describing the journal's editorial staff, respondents reported
that the journal had editors who specialized in the field (whose
background may or may not have included formal training) 67% of the
time for biomedical articles, 42% of the time for economics articles,
and 42% of the time for ethics articles. We also found that while 10
journals had statisticians among their editors to address concerns in
clinical manuscripts, only two had health economists among their
editors, and none had ethicists among their editors.
As an example of the impact of this lack of special expertise in these
areas, only one journal was able to correctly classify an abstract of a
philosophical ethics article as an original contribution rather than an
opinion/editorial piece.
We found that nine journals requested disclosure of the financial
arrangement between author and sponsor. However, in each case the
journal's understanding of the details of the relationship between
sponsor and investigator was limited. Only four inquired about
publication rights of the author; only one knew whether the sponsorship
agreement required written approval of the manuscript by the sponsor
before submission; only one knew whether the sponsor had the ability to
delay manuscript submission for other than patent reasons; and only one
knew whether there was an independent steering committee for the study.
Ten journals would accept a manuscript reporting results of either
primary or secondary data collection performed under a consulting
agreement, with two unsure as to their policies toward these
manuscripts.
COMMENT
Our study shows that while biomedical manuscripts being
considered for publication in major medical journals are reviewed by
experts in the field, economic analyses and biomedical ethics
manuscripts are not always treated in a similar fashion. Journal
editors often do not know the degree of formal training of peer
reviewers in the rapidly evolving fields of economics and biomedical
ethics. This issue is compounded when editors who do not have specific
training in these fields are responsible for assigning reviewers for
these manuscripts. If the appropriate reviewers are not chosen,
economics articles that lack some aspect of a critical analysis may be
published, detracting from the quality of these types of contributions
or adversely influencing resource allocation decisions made by health
policymakers on the basis of the reported study. Ethics articles that
are not properly reviewed may be unrecognized or miscast.
One major barrier to the development of medical ethics as an
academic endeavor is a lack of appreciation for the formal discipline
that underlies the field. Philosophical ethics has a methodology just
as rigorous as that of other academic disciplines, employs a technical
vocabulary, and requires particular reasoning skills. Philosophically
trained ethicists can offer conceptual analysis and expose inadequate
arguments.[8] They can offer valuable criticism
about purely empirical work in ethics as well.[9]
While there is no officially accepted definition of a medical ethicist, and
nonethicists can make important contributions to the field, we would suggest that
a medical ethicist is someone with advanced training in ethics--a PhD or
MA, or equivalent training--who spends or has spent substantial time
doing research and writing on ethical issues related to medicine. We
would argue that having more such persons among the editors and peer
reviewers of medical journals would help to improve the quality of
scholarship in medical ethics. Editors trained as physicians and
scientists, even if they maintain an interest in ethical issues, could
not be expected to be able to discriminate between simple opinion and
substantive work in medical ethics. The results of our survey support
this. Further, even though we suspect that the editors responding to
our survey held a looser definition of an ethicist than the one we
propose, these editors still reported that very few ethicists could be
numbered among their current staff.
Disclosure of the terms of sponsorship agreements governing
research studies enables editors and readers to better understand
potential conflicts of interest on the part of investigators and to
clearly understand the circumstances through which the manuscript was
submitted for publication. Disclosure of these agreements is critical
to addressing issues related to the academic freedom of investigators
and critical to the development of objective data to aid clinical and
health policy decision making. The issue of publication rights of
investigators is at the forefront of universities' concerns when they
review sponsored-research agreements. However, many research
sponsorship agreements may not be subject to this review. For example,
clinical investigator agreements for clinical sites in multicenter
studies do not always have all of the publication terms for the study
disclosed in the agreements with each university or investigator.
This study demonstrated medical editors' profound lack of
understanding of the nature of the relationships between investigators
and sponsors. The information we requested regarding disclosure of
relationships pertained to all manuscripts, not just economics or
biomedical ethics submissions. The lack of reporting of sponsorship
agreements for the conduct of medical research, by private,
philanthropic, or government sponsors, suggests the potential for
significant restrictions on the academic freedom of investigators and
an opportunity for potential manipulation of research findings by
sponsors.
Investigators who abide by the principles of requiring academic freedom
in sponsorship agreements may find that they are unable to compete with
investigators who do not comply with these restrictive investigation
rules. Since the journals do not inquire about such arrangements,
sponsors and investigators will not encounter any publication pressure
to comply with these standards to maintain the objectivity of research
findings. This may be especially true with respect to the many new
"entrepreneurial" organizations sponsoring clinical research. These
organizations have a large financial stake in the outcomes of the
research and may have little tradition of academic objectivity in
conducting or overseeing their clinical studies.
This study, based on a survey of editors in chief at selected
medical journals, may have some limitations. It is unclear how
generalizable our findings may be to other journals. However, we have
attempted to survey some of the largest and most prestigious
English-language medical journals, and we believe the issues we address
may be even more significant for other journals. This survey was based
on self-report of medical editors and was not validated against actual
peer review records. Further research could help clarify several of the
issues we address.
CONCLUSION
The findings of this study suggest that there are several ways in which
the peer review process can be strengthened to improve the quality of
medical literature in general, as well as literature in the areas of
clinical economics and biomedical ethics. Journal editors need to have
a better understanding of the contractual arrangements surrounding
manuscript submission to protect the integrity of medical literature, a
finding not limited to contributions on ethics or economics. Journal
editors need to better understand the "basic sciences" of clinical
economics and biomedical ethics to help promote the development of
these fields as academic disciplines.
From the Division of General Internal Medicine (Drs Schulman
and Sulmasy and Ms Roney), the Clinical Economics Research Unit (Dr
Schulman and Ms Roney), and the Center for Clinical Bioethics (Dr
Sulmasy), Georgetown University Medical Center, Washington, DC.
Presented in part at the Second International Congress on Peer Review
in Biomedical Publication, Chicago, Ill, September 10, 1993.
Reprint requests to Clinical Economics Research Unit, Georgetown
University Medical Center, 2233 Wisconsin Ave NW, Suite 525,
Washington, DC 20007 (Dr Schulman).
References
1. Udvarhelyi S, Colditz GA, Rai A, Epstein
AM. Cost-effectiveness and cost-benefit analysis in the medical literature:
are the methods being used correctly? Ann Intern Med.
1992;116:238-244.
2. Adams M, McCall N, Gray D, Orza MJ, Chalmers TC. Economic analysis in randomized control trials. Med Care.
1992;30:231-244.
3. Brody B. The quality of scholarship in
bioethics. J Med Philos. 1990;15:161-178.
4. Southgate MT. Conflict of interest and the
peer review process. JAMA. 1987;258:1375.
5. Hillman A, Eisenberg JM, Pauly M, et al.
Avoiding bias in the conduct and reporting of cost-effectiveness research
sponsored by pharmaceutical companies. N Engl J Med.
1991;324:1362-1365.
6. Krahn M, Detsky AS. Should Canada and the
United States universally vaccinate infants against hepatitis B? a
cost-effectiveness analysis. Med Decis Making. 1993;13:4-20.
7. International Committee of Medical Journal Editors.
Conflict of interest. Ann Intern Med. 1993;118:646-647.
8. Beauchamp TL. What philosophers can offer. Hastings
Cent Rep. June 1982;12:13-14.
9. Brody BA. Assessing empirical research in bioethics. Theor Med. 1993;14:211-219.
Table of Contents
