Evaluating the Quality of Articles Published in Journal Supplements Compared With the Quality of Those Published in the Parent Journal
(JAMA. 1994;272:108-113)
Paula A. Rochon, MD, MPH, FRCPC; Jerry H. Gurwitz, MD; C. Mark
Cheung, MD, FRCPC; Jason A. Hayes; Thomas C. Chalmers, MD
Objectives.--To determine the relationship between the
quality of articles and whether they were published in a supplement or
in the parent journal.
Data Sources and Study Selection.--All randomized control
trials of drug therapies in adults published in the American
Journal of Cardiology, the American Journal of Medicine,
and the American Heart Journal from January 1990 and obtained
in November 1992 by means of a MEDLINE search. A total of 318 abstracts
appeared to meet our inclusion criteria, and these articles were
obtained and reviewed in further detail. An additional 76 were
excluded.
Data Extraction.--Three reviewers who were "blinded" and
thus unaware of supplement status independently assessed the quality of
each of the remaining 242 articles according to a standard quality
scoring system.
Data Synthesis.--Overall, 67 (27.7%) of the articles were
published in journal supplements. Article quality scores ranged from
4.2% to 87.5%, with a mean (+/-SD) score of 37.2%+/-13.1%. Quality
scores were lower in articles published in journal supplements than in
those published in the parent journal (t [240]=2.61,
P=.01). The mean quality score for articles published in
journal supplements was 33.6%+/-12.8% compared with a score of
38.5%+/-13.1% for articles published in the parent journal.
Supplement articles included in their final analysis a smaller
proportion of the patients initially randomized
(t [75]=2.8, P=.007).
Conclusion.--Our findings suggest that randomized control
trials published in journal supplements are generally of inferior
quality compared with articles published in the parent journal. The
review process surrounding the publication of journal supplements
should be consistent with that of the parent journal.
(JAMA. 1994;272:108-113)
Concern has been raised about scientific articles
published in journal supplements.[1]
[2]
[3]
[4]
[5]
Supplements are
frequently underwritten by pharmaceutical manufacturers and are less
likely to undergo the peer review process.[1,2,5] Despite
these concerns, articles published in journal supplements are not
easily or clearly differentiated from peer-reviewed articles.
Supplement articles are listed in databases, such as MEDLINE, housed in
medical libraries, included in meta-analyses, referenced in reviews,
and distributed as reprints. They are also routinely cited in
promotional literature distributed by pharmaceutical manufacturers.
To evaluate the quality of articles published in journal supplements,
we reviewed all randomized control trials of drug therapies published
between 1990 and 1992 in three journals that frequently publish
supplements[1]: the American Heart Journal, the
American Journal of Cardiology, and the American Journal
of Medicine. Our objective was to rigorously assess the quality of
scientific articles published in journal supplements compared with
those published in the parent journal, by means of a standard quality
scoring system.
MATERIALS AND METHODS
All randomized control trials of drug therapies listed in MEDLINE from
January 1990 and obtained in November 1992 in the American Journal
of Cardiology, the American Journal of Medicine, and the
American Heart Journal were identified by the following
MEDLINE search strategy: CLINICAL TRIALS or PROSPECTIVE STUDIES or
explode RESEARCH DESIGN (includes DOUBLE-BLIND METHOD; RANDOM
ALLOCATION) or (text root word) random (with any ending, eg,
randomized, randomly) and HUMAN and ADULT. This search identified a
total of 504 articles.
The abstracts for these 504 articles were reviewed by one of us
(P.A.R.). All 318 articles that appeared to be of possible relevance
were retrieved and independently reviewed in detail by three of us
(C.M.C., J.H.G., and P.A.R.). We included all randomized control trials
that reported the results of drug trials in adults, excluding drug
washout studies and secondary analyses of previously published studies.
After a detailed review of the articles, an additional 76 were excluded
for the following reasons: nonrandomized control trial (n=41),
secondary analysis (n=20), drug washout (n=7), non-drug related (n=4),
pediatric study population (n=2), and unobtainable (n=2). The remaining
242 articles comprised our study sample.
Supplement status was determined by verifying whether articles in the
study sample were published in a free- standing supplement issue or in
the parent journal issue.
Characteristics of Articles in the Sample
Information was obtained on (1) the drug therapies evaluated, (2)
source of acknowledged financial support, (3) the number of patients
randomized in the trials, and (4) the frequency with which the
reference list included articles published in journal supplements.
Financial support was defined as one of the following four mutually
exclusive categories: (1) pharmaceutical manufacturer, (2) government
or foundation, (3) both pharmaceutical manufacturer and government or
foundation, and (4) no support acknowledged. Pharmaceutical
manufacturer support included (1) acknowledged grant support from the
pharmaceutical company identified either on the article or on the cover
sheet of the supplement issue; (2) an employee of the manufacturer
listed as an author; (3) drug supplied by the manufacturer; and (4)
other types of pharmaceutical company support (eg, coordinating the
study, conducting laboratory analyses, etc).
Each article was assessed for the discrepancy between the number of
subjects randomized and the number subsequently analyzed in the study.
Quality Scoring
Three trained reviewers (C.M.C., J.H.G., and P.A.R.), "blinded" to
supplement status, independently collected information related to
quality for each of the 242 articles. Formal assessment of the article
quality was based on a modified version of a quality scoring
system[6] [7] that evaluated the study protocol and the data
analysis as outlined below. The questions selected from this quality
scoring system to generate the quality score were those deemed to be
most applicable to clinical drug trials. The quality scores assigned
were percentages (total score/total possible score) because not all
items were always applicable to the study under review. The total
quality score assigned for each article was the average of the scores
of the respective reviewers.
Control Appearance and/or Regimen (3, Same; 0,
Different/Unstated).--Full credit can be attained only when the
appearance of the control regimen is identical to the treatment. In
trials involving parenteral medication, this should include vehicles
and mode of administration as well as physical appearance of the drug.
In any case, the dosage regimens must be identical for credit. This
item is scored "not applicable" when both patients and observers
are not blinded.
Randomization Blinding: Was it Blind? (10, Yes; 5, Partial; 0,
No/Unknown.-- The crux of this issue lies in determining whether
the investigators could discern which treatment was next in line.
Blinding of randomization is ensured by assignment via telephone
communication, preferably by an individual not involved in the actual
treatment, or with indistinguishable treatments randomly precoded by
the pharmacy. Computer programs are available that require insertion of
the name and identification number of the patient before treatment
assignment is determined, recorded, and revealed. Use of patient chart
numbers, birth dates, dates of admission, coin flips, or alternate
(odd-even) allocation are all examples of unblinded,
quasi-randomization methods and should receive no credit. It is
necessary to reaffirm the imperfections of open tables of random
numbers or sealed envelopes (regardless of opacity). In general, if it
is impossible to predict the next treatment in line, full credit should
be given. If there is a small chance of the next treatment being
predicted (eg, sealed envelopes), partial credit should be given.
Mention of random numbers and coin flips could receive half credit if
they are used by someone other than the person enrolling study
participants. No credit for randomization blinding is given if
pharmacy-supplied tablets are labeled A and B and any leak is
possible.
Patients Blinded (8, Complete; 4, Partial; 0, None).--The
confusion on this issue lies in determining whether a trial is blinded
merely by its statement, regardless of contradictory evidence.
Techniques of blinding must be fully reported; there should be little
chance of patients guessing which treatment they are getting. In the
specific case where different dosages of the same drug are given
without "dummies" to make up the difference, no credit can be
given. When side effects clearly reveal the patient's group, the study
should be considered partially blinded. Single-blindedness, unless
otherwise specified, should be assumed to refer to patients'
blinding. A statement that the trial was double blinded without
supporting evidence receives partial points.
Observers Blinded to Treatment (8, Complete; 4, Partial; 0,
None).--The same conditions for patients blinded to treatment also
apply to observers. A study is classified by column as to whether or
not all observers could have been blinded, some only, or none. Then
these questions are answered as to whether complete, partial, or no
blinding was actually employed. A stand-alone statement that a study
was double blind, without any details allowing judgment about the
thoroughness, will receive only four points each for this question and
the preceding one.
Observers Blinded to Results (10, Yes; 5, Partial; 0,
No/Unknown).--Full credit is given solely when there is a separate
data-monitoring person(s). Partial credit is given when less
satisfactory attempts are made to blind the observers as to the
results. A score of "partial" is also given when a coordinating or
data-monitoring committee is mentioned but there is no specific
statement about individual centers or observers being blinded to
ongoing results. No credit is given when no effort is made to blind the
observers as to the results. (Note: This item refers only to the
ongoing results of the study, not to any individual patient's
outcome.)
Previous Estimate of Numbers (3, Yes; 0,
No/Unknown).--There should be evidence that the numbers of
patients required were previously estimated. An excellent article lists
in its "Methods" section the expected rate of control outcomes, the
largest improvement of clinical interest that should not be missed, the
chosen levels of error (alpha and beta), and the numbers anticipated.
Testing Compliance (3, Adequate; 1.5, Fair; 0,
Inadequate).--This item is scored "not applicable" for
in-hospital studies except when a test of compliance (eg consulted
nurses' notes) is specifically mentioned, in which case full credit is
given. In addition, a score of "not applicable" is given for trails
that use parenteral medications or surgical procedures. Full credit is
given for outpatient studies that use pill counts or other similar
tests of compliance. A score of "fair" reflects a flawed attempt to
assess compliance (eg, asking the patients if they took their
medication). Measurements of biologic equivalence will, in some cases,
satisfy this item.
Results of Randomization on Pretreatment Variables and Inclusion
in Analysis (3, Adequate; 1.5, Fair; 0, Inadequate).--Analyses
done to assess the baseline comparability of the study groups may give the interpreter of the results of the trial reasons for being cautious or even suspicious. Thus, the issue here is
whether the prognostic variables have been analyzed well and whether
the author(s) considered their impact on the results. Either
statistical analysis or a statement that the groups are too similar to
require statistical analysis is necessary for a score of adequate.
Major End Points (4, Test and P; 1,
P, No Test; 1, Test, No P; 0,
Neither).--When significance is reported, it should be given
so that the reader can make the actual calculations. Both the test
statistic (where appplicable) and the observed P values should
be stated. If the observed P level is present but not the test
statistic, or the test statistic is given without the P level,
the reader may have trouble verifying the statistical conclusions. The
absence of both is unacceptable.
Post-beta Estimate (Negative Trials Only) (3,
Adequate; 1.5, Fair; 0, Inadequate; 9, Not Applicable).--If the
difference between the compared treatments is not statistically
significant, then a discussion of the type II error and its probability
should be included in the "Results" or the "Discussion" section
of the trial. Actually estimating a posterior beta for a clinically
interesting difference gets a score of "adequate," as does a
comment about the confidence intervals around the difference.
Mentioning the problem and admitting a necessity for more patients
("problem needs further study to be sure a difference is not being
missed") gets a score of "fair." Trials in which all differences
measured are statistically significant warrant a score of "not
applicable."
Confidence Limits (3, Yes; 1.5, Partial; 0, No).--Confidence
limits should be provided for differences in the proportions, rates, or
means used as trial end points. Reporting SD or SE of end points by
group gains 1.5 points.
Statistical Analyses (4, Excellent; 2, Good; 1, Fair; 0,
Poor).--This is an overall assessment of the statistical methods
that are used in the trial. When all appropriate analyses have been
performed and clearly and precisely reported, a score of
"excellent" should be given. If many, but not all, analyses have
been performed, or not all results of these analyses are reported, ie,
the test statistic or the name of the test (where no test statistic
exists) and the P value, a score of "good" is appropriate.
A score of "fair" should be given when more serious deficiencies
exist. (Note that the answer to this question is highly subjective.)
Withdrawals After Randomization (3, None; 1.5,
Listed With Reason; 0, Listed Without Reason No List/Unknown or
>15%).--Withdrawals are defined as patients who were randomized
but did not receive the allocated treatment or complete the specified
observation period, or who were removed from the study during the
analysis period. If investigators specify the numbers of patients who
withdrew from the trial and the reasons why, this would represent an
adequate "list." If the numbers are listed without the reasons, no
points are given. If there are no withdrawals, and it is so stated in
the article, this item would be scored as "none." "No
list/unknown" means that no information is given. If more that 15%
of the randomized patients withdraw from a study or are removed, the
results are suspect and no points are given.
Side Effects Discussion (3, Adequate; 1.5, Fair; 0,
Inadequate).--To obtain a score of "adequate," side effects
should be reported and discussed, and adequate statistical analyses
should be carried out. Comparison of percentages with a statistical
test of significance and the observed probability should be done if the
sample size warrants it. The name of the test (and test statistic where
applicable) and the P value must be recorded. A score of
"fair" reflects an incomplete report or discussion. When no attempt
is made to deal with side effects beyond simply listing them, or not
listing them at all, a score of "inadequate" should be given.
Statistical Analyses
Differences in continuous variables between articles published in a
supplement issue and those published in the parent journal were
assessed by t tests. Separate variance estimates were used
where appropriate, and in such cases Satterthwaite's approximation to
the numbers of df was used.[8] We used
chi2 tests to assess differences in categorical variables
between articles published as supplements and those published in the
parent journal. To assess the consistency of scoring between the three
reviewers, we performed a principal component analysis that took into
account all three quality scores simultaneously. Analyses were
performed with SPSS 4.0 for the Macintosh[9] and SAS System for Windows 3.10.[10]
RESULTS
Characteristics of Articles
Of the 242 articles in our sample, 133 (55.0%) were published in the
American Journal of Cardiology, 68 (28.1%) in the
American Journal of Medicine, and 41 (16.9%) in the
American Heart Journal. Overall, 67 (27.7%) of the articles
were published in journal supplements, with the American Journal
of Medicine publishing the highest percentage of articles in
supplements (Figure 1). Most of the
therapies evaluated in these articles were cardiovascular (177 [73.1%]) and
antibiotic (24 [9.9%]). Others included gastrointestinal, oncologic,
and respiratory. The number of patients randomized in the trials ranged
from six to 8245, with a median of 50.
Support was acknowledged from a pharmaceutical manufacturer in 142
(58.7%), from a government or foundation source in 16 (6.6%), and
from both a pharmaceutical manufacturer and a government or foundation
source in 25 (10.3%); no source of funding was acknowledged in 59
(24.4%) of the articles. The sources of financial support acknowledged
differed between supplement articles and those published in the parent
journal (chi2=50.1, df=3, P<.0001)
(Figure 2). No supplement article received funding solely
from a government or foundation source. All of the supplement articles
in our sample were published in a supplement that was sponsored by a
pharmaceutical manufacturer. In 63 (94.0%) of the 67 supplement
articles in our sample, only support from a pharmaceutical manufacturer
was acknowledged. In the remaining four supplement articles, support
was also acknowledged from a government or foundation source. In 39
(58.2%) of the 67 articles published in journal supplements, the
acknowledgment of pharmaceutical manufacturer support was referred to
only in the introduction to the supplement issue, with no mention of
such support in the article itself.
A systematic examination of the references listed in each
article in the study sample indicated that at least one article
published in a journal supplement was referenced in 165 (68.2%) of the
articles reviewed. Articles published in supplements were more
frequently referenced in supplement articles in the sample than in
nonsupplement articles. Supplement articles comprised 17.2%+/-16.1%
(mean+/-SD) of the references listed in articles published in journal
supplements compared with 8.1%+/-9.8% of those referenced in
nonsupplement articles (t[85]=4.4, P<.001).
No statistically significant difference was found between the number of
subjects randomized in trials published in either supplement issues or
the parent journal. On average, articles published in journal
supplements randomized 280.0+/-1001.5 patients (mean+/-SD) in their
trials compared with 148.0+/-664.2 in the parent journal
(t[89]=1.0, P=.32). Articles published in journal
supplements included in their final analysis a smaller proportion of
patients initially randomized. Supplement articles included
85.0%+/-15.8% of patients initially randomized, compared with
91.3%+/-11.0% of patients included in the final analysis by articles
published in the parent journal (t[75]=2.8, P=.007).
Overall Quality Score
Article quality scores ranged from 4.2% to 87.5% of a possible 100%,
with a mean score of 37.2%+/-13.1%. On average, articles published in
the American Journal of Cardiology received a score of
38.1%+/-13.9%, those published in the American Journal of
Medicine received a score of 35.3%+/-13.0%, and articles
published in the American Heart Journal received a score of
37.4%+/-10.7%. Quality scores were lower in articles published in
journal supplements than in those published in the parent journal
(t[240]=2.61, P=.01). The mean quality score for
articles published in journal supplements was 33.6%+/-12.8% compared
with a score of 38.5%+/-13.1% for articles published in the parent
journal.
A principal-component analysis suggested that the quality scores
assigned by the three reviewers were consistent. The first principal
component explained 76% of the total variation between the quality
scores obtained by the reviewers, and these scores contributed evenly
to this first principal component.
Summary of Individual Items Used to Measure Quality
Articles published in journal supplements and those published in the
parent journal were compared on each of the 14 items used to calculate
the overall quality score. The average scores obtained for each of the
items used to assess the quality of the articles are summarized in the Table. In 12 of the 14 items that were used to evaluate quality, supplement articles received a lower score than
articles published in the parent journal. For two of the quality score
items, one that assessed whether the randomization process was blind
and another that evaluated whether the description of the drug side
effect profile was adequate, articles published in supplements received
higher scores, although these differences were not statistically
significant.
Six of the 14 quality scoring items that we used were found to
differentiate clearly the quality of articles published in supplements
compared with the parent journal. Articles published in journal
supplements were less likely to provide information on the testing of
compliance (t[240]=4.5, P<.001), provided fewer
details on the results of prerandomization (t[96]=3.0,
P=.003), provided less information about the use of major end
points (t[94]=2.7, P=.008), were less likely
to include a post-beta estimate when applicable (t[78]=2.3,
P=.022), were judged to be poorer in overall presentation of
the statistical analysis (t[240]=2.7, P=.007), and
dealt with the withdrawal of subjects from the trials in a less
satisfactory manner (t[240]=3.75, P<.001).
COMMENT
Quality Issues
Our data suggest that scientific articles on randomized control trials
of drug therapies published in journal supplements are generally of
inferior quality compared with articles published in the parent
journal. To our knowledge, this is among the first
studies[11] to evaluate formally the quality of
supplement articles and to make such a comparison.
Quality was evaluated by several measures. Using a standardized quality
scoring system, we objectively scored the article quality. The quality
scoring system method has been described in its original form[6] and adapted forms[7] [12]
[13]
[14]
and has been applied in its original or adapted form to assess the quality of
articles included in meta-analyses of clinical
topics[15]
[16]
[17]
[18]
[19]
[20]
[21]
[22]
[23]
[24]
and for other research applications.[12]
[25]
[26]
[27]
Overall, the quality scores that were obtained for both the parent and the
supplement journals in our sample were low compared with those in
previous reports. Reitman et al[27] found that quality scores
improved over time. The mean (+/-SD) quality score for articles
published in the 1980s was 43%+/-18% in a study of more than 400
articles describing the results of randomized control trials.
In examining the individual items used to determine the overall quality
score, our results showed generally lower scores for articles published
in supplements than for articles published in the parent journal. In
the assessment of the study protocol, articles published in journal
supplements provided less information to ensure that adequate
steps had been taken to assess compliance with the study medication.
Supplement articles provided less satisfactory details about the
baseline characteristics of the groups of patients being studied before
being randomized to their treatment. In the data analysis, articles
published in journal supplements were less likely to provide complete
information on major end points, and, when a negative trial was
reported, a post-beta estimate or discussion was less often provided.
When the reviewers were asked about their global rating of the
statistical analysis, supplement articles received, on average, lower
scores. Finally, data on patients who withdrew from the study were less
likely to be handled in a satisfactory manner.
Another measure used to assess article quality was an evaluation of the
discrepancy between the number of patients randomized and subsequently
analyzed in each trial. It is generally accepted, according to the
"intention to treat" principle, that all patients initially
entering a study should be included in the final analyses. Deviations
from this policy lead to concerns about the quality of the study
results.[6] [28] We found that, on average, there is a greater
discrepancy between the number of patients randomized and the number of
patients analyzed in articles published in journal supplements than in
parent journal articles. Because withdrawals could influence the final
composition of the various treatment groups and undermine the benefit
of randomization,[6] this measure was believed to be
important.
Consistent with a previous report,[5] our results suggest
that articles published in journal supplements frequently do not
acknowledge association with a pharmaceutical manufacturer in the
article itself. Previous work[21] suggested that there may be
a relationship between manufacturer sponsorship and findings of drug
efficacy and toxicity studies. Information regarding sponsorship is
therefore potentially relevant in the comprehensive assessment of the
literature regarding specific therapeutic agents. Because individual
articles are frequently distributed as reprints, photocopied, or read
outside of the context of the entire journal issue, it is important
that the sources of financial support be acknowledged in each article.
Information provided only on the cover sheet or in the introduction to
the supplement issue may not always be available to those reading
selected reprints.
Why Supplements Are of Lesser Quality
One reason that articles published as supplements are of less quality
than the parent journal may be a difference in the peer review process.
A report of published symposiums dealing with therapy for angina found
that few articles published in this form underwent the traditional peer
review process.[5] While supplement articles may in fact
undergo peer review, this review process is not consistently stated in
the three journals we reviewed.[1] Some journals have already
taken steps to review their policy on journal supplements with the goal
to improve their quality.[29] This is an important step given
our findings that supplement articles are frequently referenced in
articles, as well as a report that they are frequently referenced in
drug advertisements.[30]
Because of the fairly restrictive criteria used in the selection of
articles included in the study sample, the quality score results that
we obtained are not necessarily representative of the overall quality
of the individual journals. We included only randomized control trials
that compared two drug therapies in our sample and excluded those that
did not strictly fit with this definition. For example, if a trial
appeared to be a randomized control trial but did not specifically use
the word "randomized" in the text, then it was excluded.
Epidemiologic studies were also excluded from evaluation.
What Does This Mean in Clinical Practice?
Clinicians reading the medical literature should be aware that journal
supplements are not always peer reviewed and should use this
information to guide their interpretation of the results presented in
these articles and to read supplement articles in a more critical
manner. Supplement articles must be clearly distinguished from those
published in regular journal issues. We found that no consistent method
was used to identify supplement articles either in the MEDLINE search
or in the actual article. This lack of uniformity would make it
difficult for clinicians to differentiate supplement articles from
nonsupplement articles. Since supplements carry the name of an
established journal, general readers may incorrectly assume that
articles published in supplements undergo the same editorial and peer
review process as do articles published in the parent journal.
In our sample, all of the supplement articles were published in
pharmaceutical manufacturer-sponsored journal supplements. The funding
source may influence whether or not a particular study is published and
thus lead to a form of publication bias.[31]
Manufacturer-sponsored publications may tend to favor the
manufacturer-sponsored drug.[21] [32] Including non-peer-reviewed and manufacturer-sponsored supplement articles in
meta-analyses may bias the results. The findings of carefully
constructed meta-analyses are increasingly being relied on to guide
medical practice. Given the lesser quality of supplement articles, the
routine inclusion of supplement articles in meta-analyses needs to be
reconsidered.
Under some circumstances, journal supplements can play an important
educational role for the medical community. A special issue that
focuses on a single topic may provide a handy and useful reference for
clinicians. One example is the "Third ACCP Consensus Conference on
Antithrombotic Therapy" published in October 1992 as a supplement
issue to Chest. However, this type of supplement issue did not
contain original articles describing randomized control trials of drug
therapies and therefore is quite different from our study sample.
CONCLUSION
We demonstrated, by using a series of measures of quality, that
articles published as journal supplements are of lesser quality than
articles published in the parent journal. Given our findings and the
general concern about journal supplements, the role of these
publications in the medical literature should be questioned and
reevaluated.
We recommend two major strategies to deal with journal supplements in
the medical literature. The first is that they should be clearly and
consistently identified as being different from traditional
peer-reviewed articles in databases, such as MEDLINE, as well as in
individual articles. We also strongly suggest that all journal
supplements be subjected to the same rigorous peer-review process
employed by the parent journal.
From the Baycrest Centre for Geriatric Care (Drs Rochon and Cheung
and Mr Hayes) and Mount Sinai Hospital, Division of Geriatric Medicine,
Department of Medicine (Drs Rochon and Cheung), and Department of
Preventive Medicine and Biostatistics (Dr Rochon), University of
Toronto (Ontario); Program for the Analysis of Clinical Strategies,
Gerontology Division, Department of Medicine, Brigham and Women's
Hospital, Harvard Medical School, Boston, Mass, and the Brockton/West
Roxbury (Mass) Veterans Affairs Medical Center (Dr Gurwitz); and
MetaWorks Inc, West Lebanon, NH (Dr Chalmers).
Presented in part at the Second International Congress on Peer Review
in Biomedical Publication, Chicago, Ill, September 9, 1993.
This study was supported in part by grant R01 HS-05936 and
by Clinical Investigator Award K08 AG00510 from the National Institute
on Aging (Dr Gurwitz). Mr Hayes was supported by the Max and Rosalyn
Gordon Summer Scholarship Fund through Baycrest Centre for Geriatric
Care.
We are indebted to Geoffrey Litner and Jason Litner for data
management, Bruce Kupelnick for his assistance with various aspects of
the project, Malcolm A. Binns, statistician, for reviewing the
manuscript, and Michael Gordon, MD, FRCPC, for his editorial comments
and ongoing support.
Reprint requests to Baycrest Centre for Geriatric Care, University of
Toronto, 3560 Bathurst St, North York, Ontario, Canada M6A 2E1 (Dr
Rochon).
References
1. Bero LA, Galbraith A, Rennie D. The publication of
sponsored symposiums in medical journals. N Engl J Med.
1992;327:1135-1140.
2. Soffer A. Hazards in publication of proceedings.
Arch Intern Med. 1982;142:2074-2076.
3. Kessler DA. Drug promotion and scientific exchange: the
role of the clinical investigator. N Engl J Med.
1991;325:201-203.
4. Finucane T. Drug company-sponsored symposia: pros and
cons. Am J Med. 1987;83:811-812.
5. Massie BM, Rothenberg D. Publication of sponsored
symposiums in medical journals. N Engl J Med.
1993;328:1196-1197.
6. Chalmers TC, Smith HJ, Blackburn B, et al. A method for
assessing the quality of a randomized control trial. Controlled
Clin Trials. 1981;2:31-49.
7. Antczak AA, Tang J, Chalmers TC. Quality
assessment of randomized control trials in dental research, 1: methods.
J Periodont Res. 1986;21:305-314.
8. Snedecor GW, Cochran WG. Statistical Methods.
8th ed. Ames: Iowa State University Press; 1989;1.
9. SPSS Inc. SPSS Inc Base System User's Guide.
Chicago, Ill: SPSS Inc; 1990:520.
10. SAS Institute Inc. SAS/STAT User's Guide, Release
6.03 Edition. Cary, NC: SAS Institute Inc; 1988;1:1029.
11. G\ozsche PC. Methodology and overt and hidden bias in
reports of 196 double-blind trials of nonsteroidal antiinflammatory
drugs in rheumatoid arthritis. Controlled Clin Trials.
1989;10:31-56.
12. Chalmers I, Adams M, Dickersin K, et al. A cohort study
of summary reports of controlled trials. JAMA.
1990;263:1401-1405.
13. Detsky AS, Naylor CD, O'Rourke K, McGeer AJ, L'Abbe
KA. Incorporating variations in the quality of individual randomized
trials into meta-analysis. J Clin Epidemiol. 1992;45:255-265.
14. L'Abbe KA, Detsky AS, O'Rourke K. Meta-analysis in
clinical research. Ann Intern Med. 1987;107:224-233.
15. Lam W, Sacks HS, Sze PC, Chalmers TC. Meta-analysis of
randomized controlled trials of nicotine chewing gum. Lancet.
1987;2:27-30.
16. Goldstein P, Berrier J, Rosen S, Sacks HS, Chalmers TC.
A meta-analysis of randomized control trials of progestational agents
in pregnancy. Br J Obstet Gynecol. 1989;96:265-274.
17. Sze PC, Reitman D, Pincus M, Sacks HS, Chalmers TC.
Antiplatelet agents in the secondary prevention of stroke:
meta-analysis of the randomized control trials. Stroke.
1988;19:436-442.
18. Sacks HS, Ancona-Berk A, Berrier J, Nagalingam R,
Chalmers TC. Dipyridamole in the treatment of angina pectoris: a
meta-analysis. Clin Pharmacol Ther. 1988;43:610-615.
19. Liberati A, Himel HN, Chalmers TC. A quality assessment
of randomized control trials of primary treatment of breast cancer.
J Clin Oncol. 1986;4:942-951.
20. Baum ML, Anish DS, Chalmers TC, Sacks HS, Smith HJ,
Fagerstrom RM. A survey of clinical trials of antibiotic prophylaxis in
colon surgery: evidence against further use of no-treatment controls.
N Engl J Med. 1981;305:795-799.
21. Rochon PA, Gurwitz JH, Simms RW, et al. A study of
manufacturer supported trials of nonsteroidal anti-inflammatory drugs
in the treatment of arthritis. Arch Intern Med.
1994;154:157-163.
22. Naylor CD, O'Rourke K, Detsky AS, Baker JP. Parenteral
nutrition with branched-chain amino acids in hepatic encephalopathy: a
meta-analysis. Gastroenterology. 1989;97:1033-1042.
23. Wong DKH, Cheung AM, O'Rourke K, Naylor CD, Detsky AS,
Heathcote J. Effect of alpha-interferon treatment in patients with
hepatitis B e antigen-positive chronic hepatitis B: a meta-analysis.
Ann Intern Med. 1993;119:312-323.
24. Detsky AS, Baker JP, O'Rourke K, Goel V. Perioperative
parenteral nutrition: a meta-analysis. Ann Intern Med.
1987;107:195-203.
25. Antczak AA, Tang J, Chalmers TC. Quality assessment of
randomized control trials in dental research, II: results: periodontal
research. J Periodont Res. 1986;21:315-321.
26. Emerson JD, Burdick E, Hoaglin DC, Mosteller F,
Chalmers TC. An empirical study of the possible relation of treatment
differences to quality scores in controlled randomized clinical trials.
Controlled Clin Trials. 1990;11:339-352.
27. Reitman D, Sacks HS, Chalmers TC. Technical quality
assessment of randomized control trials (RCTs). Controlled Clin
Trials. 1987;8:282.
28. Gozsche PC. Bias in double-blind trials. Dan Med
Bull. 1990;37:329-336.
29. Cavalli F. Our policy regarding supplements of
sponsored symposia. Ann Oncol. 1993;4:99.
30. Finucane TE. Publication of sponsored symposiums in
medical journals. N Engl J Med. 1993;328:1197.
31. Dickersin K. The existence of publication bias and risk
factors for its occurrence. JAMA. 1990;263:1385-1389.
32. Davidson RA. Source of funding and outcome of clinical
trials. J Gen Intern Med. 1986;1:155-158.
Table of Contents
