Table 3.

Inter-rater agreement for study selection and data abstraction in review of trials of clinical decision support systems (CDSS)

Decision evaluated	Raw agreement (%)	Agreement beyond chance (Cohen's κ) (%)
Study selection
Study is potentially relevant based on examination of abstract and use of screening algorithm	99.8	96.4
Study is potentially relevant based on examination of full text and use of screening algorithm	94.9	89.5
Study meets all inclusion and exclusion criteria based on examination of full text	84.6	66.3
Data abstraction
Use of CDSS resulted in statistically and clinically significant improvement in clinical practice	97.2	93.6
CDSS intervention incorporated the potential success factor of interest (mean agreement level for the 15 features)	97.8	90.5