EFFECT OF ATTENDANCE AT A TRAINING SESSION
ON PEER-REVIEWER QUALITY AND PERFORMANCE
Michael L Callaham,1 Robert L Wears, and Joseph F Waeckerle
1Annals of Emergency Medicine; University of California, San Francisco, Box 0208, San Francisco, CA 94143-0208, USA
Objective: To determine whether peer reviewer attendance at a brief journal-sponsored training workshop changes review quality.
Design: Subjects included all 340 peer reviewers for Annals of Emergency Medicine, 45 of whom decided to attend a voluntary workshop on how to perform peer review, open to all reviewers.
Methods: Reviews were routinely rated by editors on a subjective ordinal l to 5 global quality scale, details of which are reported in another abstract. Comparisons were made between reviewers who chose to attend a 4-hour workshop on peer review sponsored by the journal in 1995, and 2 groups of reviewers who did not attend: controls matched for average quality rating and experience, and unmatched controls. Reviewer performance calculations and quality ratings in the 20 months before and 14 months after the workshop were compared.
Results: Of the 45 reviewers who attended the workshop, 40 had sufficient rated reviews. The 40 matched controls were optimally matched to these attendees. The remaining 260 nonattending reviewers (unmatched controls) had completed fewer reviews (5.4 vs 7.4, P=.009) and accepted more papers (23% vs 16%, P=.12) than attendees. Attending reviewers had a larger ratings improvement compared to matched controls or unmatched controls, and improved congruence with editorial decisions compared to unmatched controls, but differences were not significant.
Conclusions: Reviewers attending a voluntary workshop were more experienced than non-attendees, and performance improved for all reviewers during the study. Although attendees had more improvement in quality ratings than either control group, and greater improvement in congruence than unmatched controls, none of these differences was significant. It is unknown whether the rating system can detect differences produced by this sort of training, and improvement may have been better detected had the attendees been inexperienced or of lower quality. The study population was limited to reviewers with necessary data, and the power to detect small changes was low.
Return to Session Information
