Full Publication of Results Initially Presented in Abstracts
A Meta-analysis
(JAMA. 1994;272:158-162)
Roberta W. Scherer, PhD; Kay Dickersin, PhD; Patricia
Langenberg, PhD
Objectives.--To estimate the rate of full publication
of the results of randomized clinical trials initially presented as
abstracts at national ophthalmology meetings in 1988 and 1989; and to
combine data from this study with data from similar studies to
determine the rate at which abstracts are subsequently published in
full and the association between selected study characteristics and
full publication.
Data Sources.-- Ophthalmology abstracts were
identified by review of 1988 and 1989 meeting abstracts for the
Association for Research in Vision and Ophthalmology and the American
Academy of Ophthalmology. Similar studies were identified either from
reports contained in our files or through a MEDLINE search, which
combined the textword "abstract" with "or" statements to the
Medical Subject Headings ABSTRACTING & INDEXING, CLINICAL TRIALS,
PEER REVIEW, PERIODICALS, MEDICAL SOCIETIES, PUBLISHING,
MEDLINE, INFORMATION SERVICES, and REGISTRIES.
Study Selection.-- Ophthalmology abstracts were
selected from the meeting proceedings if they reported results from a
randomized controlled trial. For the summary study, similar studies
were eligible for inclusion if they described follow-up and subsequent
full publication for a cohort of abstracts describing the results of
any type of research study. All studies had to have followed up
abstracts for at least 24 months to be included.
Data Extraction.--Authors of ophthalmology abstracts were
contacted by letter to ascertain whether there was subsequent full
publication. Other information, including characteristics of the study
design possibly related to publication, was taken from the abstract.
For the summary study, rates of full publication were taken directly
from reported results, as were associations between study factors (ie,
"significant" results and sample size) and full publication.
Data Synthesis.--Sixty-six percent (61/93) of ophthalmology
abstracts were published in full. Combined results from 11 studies
showed that 51% (1198/2391) of all abstracts were subsequently
published in full. Full publication was weakly associated with
"significant" results and sample size above the median.
Conclusions.--Approximately one half of all studies
initially presented in abstract form are subsequently published as
full-length reports. Most are published in full within 2 years of
appearance as abstracts. Full publication may be associated with
"significant" results and sample size.
(JAMA. 1994;272:158-162)
Results of many types of clinical research are
presented at professional meetings and summarized in abstract format.
Often these abstracts are not available to the general scientific
community. Many are "published" in meeting proceedings accessible
only to those who attended the meeting or special journal issues that
are not indexed by MEDLINE.
Abstracts may be considered "publication" by some investigators who
do not consider it necessary to write up findings for full publication.
Considerable evidence from many clinical fields shows that a
substantial proportion of studies described in abstracts does not later
appear in the scientific literature as full-length reports.
[1]
[2]
[3]
[4]
[5]
[6]
[7]
[8]
[9]
[10]
Little else is known about the factors leading to full publication, however, in part
because abstracts contain so little descriptive information.
Failure to publish fully the results of studies, and randomized
clinical trials (RCTs) in particular, presents a problem for those
conducting meta-analyses or systematic reviews, especially if failure
to publish is associated with the results of the study (publication
bias). Any attempt to identify all studies in a field will be thwarted
by the existence of unpublished studies, and the conclusions drawn from
the review may therefore be imprecise or biased.
MATERIALS AND METHODS
Determination of Publication Rate of Abstracts in Ophthalmology
All abstracts summarizing reports made in 1988 and 1989 at either the
Association for Research in Vision and Ophthalmology or the American
Academy of Ophthalmology annual meetings were reviewed. These abstracts
are published in a special volume of Investigations in Visual
Science (for the Association for Research in Vision and
Ophthalmology) and in a supplement to Ophthalmology (for the
American Academy of Ophthalmology). Abstracts were selected for
inclusion in our study if the results were from a controlled trial
involving human beings and if the abstract stated or implied that the
treatment assignment was or could have been randomized.
Information regarding study features was extracted by one of us
(R.W.S.) from the abstract by means of a pretested form before
information was gathered from an author of the abstract. Items
extracted included trial design, sample size, number of clinical
centers, subspecialty, unit of randomization, whether the author(s)
explicitly stated or only implied that treatment assignment was
randomized, and whether results were presented as "statistically
significant." Results were classified as statistically significant if
a P value for any result was .05 or less, or if results were
stated to be "significant."
A letter and questionnaire were sent in September 1991, and reminder
letters and telephone calls were made in early 1992 to one of the
authors listed on the abstract. The questionnaire asked (1) whether
treatment was assigned by means of randomization, quasi-randomization
(eg, use of alternation, hospital record number, or birth date), or
neither; (2) if randomization was used, what method was used
to assign treatment; (3) whether the study was "complete"; (4)
whether the results had been published in full; (5) if the results had
not been published, whether any manuscripts had been prepared and
submitted; and (6) if no manuscript had been prepared, why not.
We used the author's reply to the questionnaire to calculate the rate
of full publication of abstracts. In December 1992, a MEDLINE search by
author was conducted to determine whether any new full-length reports
for RCTs had been published for previously unpublished abstracts.
Publications for each author of an individual abstract were examined,
starting with those found in the same year the abstract was presented
and going forward to 1992. If a full-length publication was found that
corresponded in content to the subject matter and a majority of the
authors, it was then assumed that the results had been published in
full.
Data from the abstraction form and the questionnaire were entered into
Paradox, version 3.5; PC-SAS, version 6.04, was used to determine the
frequencies of the various study characteristics and publication. The
association between study characteristics and publication was estimated
by means of relative risk.[11]
Meta-analysis: Weighted Average Rate of Publication
We identified reports of studies that examined the subsequent rate of
full publication of abstracts in three ways: our own files, MEDLINE
search, and word of mouth. The MEDLINE search involved combining the
textword "abstract" with Medical Subject Headings by means of a
series of "or" statements. The Medical Subject Headings included
ABSTRACTING & INDEXING, CLINICAL TRIALS, PEER REVIEW,
PERIODICALS, MEDICAL SOCIETIES, PUBLISHING, MEDLINE,
INFORMATION SERVICES, and REGISTRIES.
Reports were included in our summary study if published abstracts were
identified and followed up for at least 24 months to assess full
publication. A published abstract was defined as either an abstract or
a "summary report" (brief communication without description of
methods) in a journal, meeting proceedings, or other printed form. We
did not otherwise assess the quality of the studies examined.
We obtained a weighted average publication rate of all included
reports, weighting by the square root of the total number of abstracts
in each report. We also combined data from reports that examined the
association between various study characteristics and subsequent
full-length publication, including presence of "significant"
results and sample size. Sample size was categorized as being above or
below the median sample size of the cohort of abstracts followed up in
a particular study. The association between study characteristics and
publication was estimated by relative risk.[11] Combined
relative risks with 95% confidence intervals were calculated by means
of the Mantel-Haenszel method and SAS.
RESULTS
Determination of Publication Rate for Abstracts in Ophthalmology
Of 6014 abstracts presented in 1988 and 1989 at Association for
Research in Vision and Ophthalmology or American Academy of
Ophthalmology meetings, 149 fulfilled the inclusion criteria for this
study. We received 133 responses to the questionnaire mailed to the
authors of the 149 abstracts, for an overall response rate of 89%.
Most of the responding authors confirmed that the study described in
the abstract was an RCT (70% [93/133]). Authors completing the
questionnaire classified the remaining 40 studies as not randomized
(23% [31/133]) or "quasi-randomized" (5% [6/133]), or did not
respond to the question about study design (2% [3/133]). Of the 93
confirmed RCTs, most (75% [70/93]) had been described as such in the
abstract, but 25% (23/93) had not. Of interest, nine of 40 studies
reported as not RCTs on the questionnaire were described explicitly as
RCTs in the abstract.
Responses of authors to the questionnaire indicated that 61% (57/93)
of abstracts were subsequently published in full. The December 1992
MEDLINE search disclosed four additional full-length publications.
Thus, within at least 3 years of presentation as an abstract at a
national ophthalmology meeting, 66% (61/93) of RCTs were published as
full-length reports.
Of the 32 abstracts not published in full, five were considered to
represent incomplete studies by the author. Eleven authors had
submitted a manuscript, of which six had been rejected. Sixteen authors
had not prepared a manuscript, and a majority of these (56% [9/16])
cited "lack of time" as the predominant reason. Three authors (19%
[3/16]) stated that no manuscript had been prepared because of a
problem in study design.
A few abstracts (3% [2/61]) had been published in full before the
presentation. Most abstracts (74% [45/61]) were published in the
first 2 calendar years after the national Association for Research in
Vision and Ophthalmology or American Academy of Ophthalmology meeting
(Figure). The rest (23% [14/61]) were
published within the 4-year maximum follow-up period.
Table 1 lists the study characteristics derived from
review of the ophthalmology abstracts and the proportion published.
Positive associations were observed between publication and multicenter
status, sample size above the median, use of a method of treatment
assignment with low selection bias, and "statistically significant"
results, but none of these associations was statistically significant
at the P=.05 level.
Meta-analysis: Weighted Average of Rate of Publication
Eleven reports (10 reports[1-10] in addition to our own
results for ophthalmology) were found that estimated the percentage of
abstracts subsequently published in full. All fulfilled our inclusion
criteria, with the exception of a portion of one study,[3]
which followed up abstracts for less than 2 years; this portion was
excluded from our analysis. Six reports[1-4,7] had been
present in our files; two[6,8] were found by MEDLINE search
and two were identified at the Second International Congress on Peer
Review in Biomedical Publication, one as a presentation at that
meeting[10] and one by word of mouth.[9] Seven
medical specialties are represented among the 11 reports; the three
individual reports from [3,9,10] and
ophthalmology[6,8] each include nonoverlapping sets of
abstracts ( Table 2). Nine of the 11 reports examined
subsequent publication of results initially presented at a national
meeting of a medical specialty society; either all the abstracts
presented at the meeting(s) or a random sample of the abstracts were
followed up and analyzed. One report examined the publication rate of
abstracts that had been submitted by members of a single department to
various meetings.[10] The remaining report examined the rate
of publication of "summary reports" included in a register of
perinatal RCTs[5] Our report and one other [5] were confined to RCTs; other
reports did not select for type of study.
The average rate of full publication, weighted by the square root of
the number of abstracts, was 51%, with a 95% confidence interval of
45% to 57% and a range of 32% to 66% (Table 2). Eight reports
included data on cumulative rates of publication at various time
points. In each of the eight studies, full-length reports appeared
within 2 years of publication of the abstract (Figure).
Full publication may be associated with "statistically
significant" results (summary relative risk, 1.17; 95% confidence
interval, 0.99 to 1.39) (Table 3) and
with sample size above the median (summary relative risk, 1.48; 95% confidence
interval, 1.14 to 1.94) (Table 4).
There was evidence of nonhomogeneity in the relative risks for statistically
significant results (P=.01); the relative risk for
perinatology was in the opposite direction from those for oncology and
vision research.
COMMENT
Sixty-six percent (61/93) of ophthalmology RCT abstracts followed up
for at least 3 years were subsequently published in full. Forty percent
(11/27) of ophthalmology authors with a completed study who had not
published in full had prepared and submitted a manuscript.
Dirk[10] found that the majority (78%) of manuscripts
submitted and rejected by at least one journal were resubmitted,
sometimes to as many as six other journals, and eventually published.
Thus, assuming a similar rate of full publication for these 11
ophthalmology abstracts, the final rate of full publication of
abstracts could be as high as 75% (70/93).
Using pooled data from 11 studies across many fields of medicine, we
found that only 51% of abstracts were subsequently published in full.
Thus, for half of the studies presented in abstract form, this is the
only exposure of results to a public forum. Easterbrook and her
colleagues[12] followed up 285 initiated studies approved in
1984 by the Oxford ethics committee. The results of 207 (73%) were
published in an abstract, book chapter, or full-length article, but
only 61% (107/176) of the studies that were presented as abstracts
were also published in full, similar to the findings presented here.
Data from this and other studies have shown that a substantial
proportion of initiated studies never appear in print at
all. [12] [13]
[14]
Cumulative rates of publication for eight studies, representing six
medical specialties, were remarkably consistent in terms of timing of
full publication. Most results appearing as abstracts were published in
full within 3 years; only a few appeared later. It is therefore
unlikely that the variability seen in the rates of full publication
resulted from differing lengths of follow-up. Editors may wish to
disallow inclusion of abstracts as references if it has been more than
3 years since publication of the abstract, since most abstracts that
are going to be published will have been published within that time
frame.[9]
The issue of publication of results initially presented as
abstracts is especially important if there is an association between
full publication and study findings. A meta-analysis of studies that
investigated publication of initiated studies found a positive
association between statistically significant results and publication
(odds ratio, 2.88; 95% confidence interval, 2.13 to
3.90).[14] We found in our combined analysis that full
publication is only weakly associated with statistically significant
results and sample size above the median. Our ophthalmology study
defined "statistically significant" results as inclusion of a
statement in the abstract to that effect or a P value
indicating superiority of one group to another group. Chalmers et
al[5] used a broader definition, including any positive
results, not necessarily limited to those with statistical
significance, and did not find an association. De Bellefeuille et
al[7] used a definition similar to ours for RCTs, but they
used a definition similar to that used by Chalmers et al for phase II
trials. The difference in definition of "statistically significant"
between the study by Chalmers et al and the other two studies may
contribute to the difference found among these studies in the
relationship between full publication and study results. Because of
heterogeneity and the small number of studies, these results cannot be
considered conclusive but should be viewed as a starting point for more
research in this area.
Our study also indicates that studies with larger sample sizes are more
likely to be published in full. While the association between sample
size and publication of any sort has long been suspected, it has only
been shown in two studies[12] [15]
and was not found in a meta-analysis focusing on clinical
trials.[16] The obvious
relationship between sample size and significant results may confound
any association between full publication and sample size, so these
results should be interpreted cautiously.
The rate of full publication was the highest for ophthalmology. One
reason for our results might be that we focused on RCTs. One might
expect that RCTs would have a higher rate of publication than studies
that used other designs, simply because of the time and effort required
on the part of the investigators and volunteers. On the other hand,
Chalmers et al[5] followed up perinatal RCTs and found a
publication rate of only 36%. This apparent lack of relationship
between study design and publication was also found by Dickersin et
al.[13] Juzych et al[6,8] found publication rates
similar to our own in their two studies that followed up ophthalmology
abstracts describing studies of various designs. Thus, publication rate
differences may be related more to field than to study design.
Of considerable concern to us is the confusion on the part of some
authors about whether a study is an RCT. Nine of 79 authors who
explicitly stated in the abstract that treatment assignment was
randomized indicated the opposite in their questionnaire response. We
are unable to discern whether authors reported the method of treatment
assignment incorrectly in the abstract or whether the author completing
the questionnaire did not remember the details of the study.
Conversely, study design was not always explicitly described in some
abstracts that were subsequently reported as RCTs by the authors on the
questionnaires (23 of 54). Others have shown that this information is
not always clear even in full-length
articles.[17]
[18]
[19]
[20]
Structured abstracts address this
problem and should be employed not just when the abstract is part of a
full-length article, but also when the abstract is used to summarize
results presented at a meeting.
Initiated studies of acceptable quality should be reported in full.
Randomized controlled trials might be considered by their design alone
to meet minimal quality standards, and therefore all of them should be
reported in full. In fact, only three authors of verified RCTs in this
study reported that a design problem led to lack of full publication.
Failure to publish appears to lie mainly with investigators who do not
prepare manuscripts primarily because of "lack of
time."[10,12-14] [21] Journal editors may
not wish to publish articles that conclude no difference in treatment effect
because of their perception of readership
interest.[22] Online journals,
such as the Online Journal of Current Clinical Trials, may
alleviate this problem because there are no space or page limitations
and editors can focus on design and execution rather than results.
One solution to the problem of failure to publish is to require that
all initiated studies, especially RCTs, be registered prospectively at
initiation. With universal registration, persons interested in a
particular disease or treatment will have access to information about
completed, ongoing, published, and unpublished trials. Any registered
ongoing trial would remain in the registry after completion and thus
remain accessible, even if never published. Registries currently exist
for a number of medical conditions, including acquired immunodeficiency
syndrome, stroke, and cancer.[23] [24] Efforts are now under
way to set up new registries, to network existing registries, and to
educate the scientific, biomedical, and patient communities about the
existence and purpose of registries. For example, the recently
established international Cochrane Collaboration is committed to
"preparing, maintaining and disseminating systematic reviews of the
effects of health care,"[25] including construction of a registry of all RCTs in medicine.
One obvious mechanism for registering ongoing trials is through
institutional review boards.[26] Their charge to protect the
safety of participants in research makes it appropriate that they play
a critical role in seeing that information on all studies involving
human volunteers is disseminated. Dissemination of trial information,
ideally through registration, preserves the trust that investigators
make with those who participate in research. Not to disseminate the
results of well-designed and well-conducted trials is unethical.
From the Department of Epidemiology and Preventive Medicine, University of
Maryland School of Medicine, Baltimore, Md.
Presented in part at the Second International Congress on Peer Review in Biomedical Publication, Chicago, Ill, September 10, 1993.
This study was supported in part by Public Health Service grant
EY07766 from the National Eye Institute, National Institutes of Health,
Bethesda, Md (Dr Dickersin).
We wish to thank Luca Rossetti, MD, for helping with the analysis and
for many fruitful discussions.
Reprint requests to Department of Epidemiology and Preventive Medicine,
University of Maryland School of Medicine, 660 W Redwood St, 228 Howard
Hall, Baltimore, MD 21201 (Dr Scherer).
References
1. Dudley HAF. Surgical research: master or servant.
Am J Surg. 1978;135:458-460.
2. Goldman L, Loscalzo A. Fate of cardiology research
originally published in abstract form. N Engl J Med.
1980;303:255-259.
3. Meranze J, Ellison N, Greenhow DE. Publications resulting
from anesthesia meeting abstracts. Anesth Analg.
1982;61:445-448.
4. McCormick MC, Holmes JH. Publication of research
presented at the pediatric meetings: change in selection.
AJDC. 1985;139:122-126.
5. Chalmers I, Adams M, Dickersin K, et al. A cohort study
of summary reports of controlled trials. JAMA.
1990;263:1401-1405.
6. Juzych MS, Shin DH, Coffey JB, Parrow KA, Tsai CS, Briggs
KS. Pattern of publication of ophthalmic abstracts in peer-reviewed
journals. Ophthalmology. 1991;98:553-556.
7. De Bellefeuille C, Morrison CA, Tannock IF. The fate of
abstracts submitted to a cancer meeting: factors which influence
presentation and subsequent publication. Ann Oncol.
1992;3:187-191.
8. Juzych MS, Shin DH, Coffey J, Juzych L, Shin D. Whatever
happened to abstracts from different sections of the Association for
Research in Vision and Ophthalmology? Invest Ophthalmol Vis
Sci. 1993;34:1879-1882.
9. Yentis SM, Campbell FA, Lerman J. Publication of
abstracts presented at anaesthesia meetings. Can J Anaesth.
1993;40:632-634.
10. Dirk L. Incidence of acceptance, rejection,
resubmission, and nonpublication of full reports of research presented
at academic meetings. Presented at the Second International Congress on
Peer Review; September 10, 1993; Chicago, Ill.
11. Kahn HA, Sempos CT. Statistical Methods in
Epidemiology. New York, NY: Oxford University Press; 1989.
12. Easterbrook PJ, Berlin JA, Gopalan R, Matthews DR.
Publication bias in clinical research. Lancet.
1991;337:867-872.
13. Dickersin K, Min Y-I, Meinert CL. Factors influencing
publication of research results: follow-up of applications submitted to
two institutional review boards. JAMA. 1992;267:374-378.
14. Dickersin K, Min Y-I. NIH clinical trials and
publication bias. Online J Curr Clin Trials. 1993;Doc 50.
Serial online.
15. Begg C, Berlin J. Publication bias and dissemination of
clinical research. J Natl Cancer Inst. 1989;81:107-115.
16. Dickersin K, Min Y-I. Publication bias: the problem that
won't go away. Ann N Y Acad Sci. 1994;703:135-146.
17. Schuyler P, Dickersin K, Scherer R,
Wright N. Identification of randomized clinical trials by means of MEDLINE.
Presented at the Second International Congress on Peer Review in
Biomedical Publication; September 10, 1993; Chicago, Ill.
18. Lefebvre C, Kelleher TA, Tihanov G.
Identification of
randomized controlled trials using MEDLINE: the situation in 1993.
Presented at An Evidence-Based Health Care System: The Case for
Clinical Trials Registries, National Institutes of Health, Office of
Medical Applications of Research; December 6, 1993; Bethesda, Md.
19. Altman DG, Dore CJ. Randomisation
and baseline comparisons in clinical trials. Lancet. 1990;335:149-153.
20. Schulz KF, Chalmers I, Grimes DA,
Altman DG. Assessing the quality of randomization from reports of controlled
trials published in obstetrics and gynecology journals. JAMA.
1994;272:125-128.
21. Dickersin K, Chan S, Chalmers TC, Sacks HS, Smith H.
Publication bias and clinical trials. Controlled Clin Trials.
1987;8:343-353.
22. Instructions to authors. Arch Dis Child.
1985;60:90-91.
23. Dickersin K. Research registers. In: Cooper H, Hedges
L, eds. The Handbook of Research Synthesis. New York, NY:
Russell Sage; 1994.
24. International Collaborative Group on Clinical Trial
Registries. Position paper and consensus recommendation on clinical
trial registries. Clin Trials Meta-anal. 1993;28:255-266.
25. Chalmers I, Dickersin K, Chalmers TC. Getting to grips
with Archie Cochrane's agenda. BMJ. 1992;305:786-788.
26. Easterbrook PJ. Reducing publication bias.
BMJ. 1987;295:1347.
Table of Contents
