Interrater Agreement

Cochrane reviews showed significantly higher intrapair agreement than non-Cochrane reviews, likely because stricter Cochrane reporting standards reduce the ambiguity appraisers must navigate. Interrater agreement across all 14 appraisers averaged 0.59 (95% CI 0.48–0.70), spanning slight to substantial, while intrapair agreement averaged 0.75 (95% CI 0.68–0.82), spanning fair to almost perfect. Agreement scores did not change significantly across the testing period, so improvements in completion time reflected gains in efficiency rather than rating accuracy. Collapsing answer options by combining Yes with Probably Yes and No with Probably No improved agreement across all three levels of analysis. Earlier AMSTAR-PF questions covering review planning, literature searching, and study inclusion showed higher agreement than later questions addressing synthesis and interpretation, reflecting greater subjectivity in those domains.