CONSISTENCY OF REVIEWER RATINGS
AND IMPACT ON EDITOR MANUSCRIPT
DECISIONS
Michael L Callaham and Joseph Waeckerle
Annals of Emergency Medicine, University of California, San Francisco, Box 0208, San Francisco, CA 94143-0208, USA;
University of Missouri at Kansas City, School of Medicine, Kansas City, KS, USA
Objective: Whether quality ratings of peer reviewers by editors are consistent or can be replaced by readily available journal statistics.
Design: Thirty-three editors rated the quality of each manuscript's review on a subjective ordinal l to 5 scale. These ratings were examined
for interreviewer agreement, sources of variance, and relationship to other reviewer performance calculations.
Results: The study included 2,220 reviews of 770 manuscripts by 526 reviewers. Forty-nine percent of reviewer recommendations matched editorial decisions.
Reviews received an average rating score of 3.8 (SD, 1.0). The weighted kappa for the acceptance decisions of reviewers vs editors was 0.30. The within-reviewer
intraclass correlation was 0.41 (P=.0001), indicating that reviewer attributes explained 41% of the variance not explained by manuscript and editor random effects.
By comparison, intraclass correlations for editor and manuscript were only 0.16 and 0.12, respectively. In a subset of 1,395 reviews by 151 reviewers with 6 or more
rated reviews, the average rating score for all reviews by each reviewer was not a strong predictor of rank correlation with final editorial decision (R=0.37; 95% CI, 0.22-0.49)
or the reviewer's average acceptance rates (R=0.41).
Conclusions: Editor ratings of individual reviewers were only modestly consistent. They were not strongly related to easily calculated journal statistics such as reviewer
acceptance rates or concordance with editorial decisions (which is often determined by factors other than review quality), and thus cannot be replaced by them. Reviewer ratings
have nonetheless seemed useful in eliminating poor reviewers and in recognizing exemplary ones.
Return to Session Information
