POSITIVE OUTCOME BIAS AND
OTHER LIMITATIONS IN THE SELECTION
OF RESEARCH ABSTRACTS
FOR A SCIENTIFIC MEETING
Michael L Callaham,1 Robert L Wears, Ellen Weber, Christopher Barton, and Gary Young
1Annals of Emergency Medicine; University of California, San Francisco, Box 0208, San Francisco, CA 94143-0208, USA
Objective: to determine features influencing the outcome of research abstracts submitted to a national meeting.
Design: Abstracts of all research submitted for consideration to a national research meeting in emergency medicine (n=493) were classified and rated by a minimum of 2 blinded reviewers, study-effect size calculated for each, and MEDLINE search with 5-year follow-up (and author questionnaire) was conducted to determine publication as a full manuscript in a peer-reviewed journal. The characteristics of the meeting and the submitted research were similar to those of 31 other US medical specialty societies.
Results: Of the total 493 submitted abstracts, 179 (36%) were accepted for presentation, and 214 (43%) were published in 41 journals. Abstract scientific quality did not strongly predict either outcome or journal prestige. The best predictors (by logistic regression) of meeting acceptance were subjective "excitement" factor assigned by the blinded reviewers (OR=2.8) and positive results (OR=1.9); and for publication, meeting acceptance (OR=2.6) and large sample size (OR=2.0). Despite a mandatory structured format, much key information on methodology was absent; 48% of abstracts did not report on blinding, and 23% did not report on randomization. Funnel plots of effect sizes showed the classic distribution of positive-outcome ("publication") bias, favoring studies with positive effects regardless of methodologic rigor. Represented studies had 104% more positive effect sizes than those rejected, and those published had 113% more than those not published (P=.03). Meeting acceptance predicted publication with a sensitivity of only 51%, specificity of 71% (positive predictive value of 57% and negative predictive value of 66%).
Conclusions: Despite a mandatory structured format, primary abstracts provided insufficient information about design and results to allow meaningful peer review. Positive-outcome bias was evident when studies are first submitted for consideration to the meeting, and was amplified both in the selection of abstracts for presentation and in selection for publication as a full manuscript in a journal, neither of which was strongly related to study design or quality or publishing journal prestige. Better methods for selection of abstracts presented at meetings are needed.
Return to Session Information
