Table 3. Multilevel meta-analyses of the metric inter-rater-reliabilities (Fisher-Z √rtt or r).
| Model | Model 0Intercept | Model 1Number of Manuscripts | Model 2Method | Model 3Discipline | Model 4Object of Appraisal | Model 5Cohort | Model 6Blinding | Model 7Rating System | Model 8Number of Manuscripts, Rating System |
| Fixed effects | Coeff / SE | Coeff / SE | Coeff / SE | Coeff / SE | Coeff / SE | Coeff / SE | Coeff / SE | Coeff / SE | Coeff / SE |
| Intercept | .67 / .04* | .77 / .04* | .69 / .07* | .66 / .05* | .74 / .07* | .60 / .08* | .71 / .05* | 1.03 / .13* | 1.06 / .11* |
| Number of Manuscripts/100 | −.03 / .007* | −.03 / .006* | |||||||
| Method (RC = r) | |||||||||
| ICC | −.02 / .08 | ||||||||
| Discipline (RC = Social Sciences) | |||||||||
| Economics/Law | .08 / .13 | ||||||||
| Natural Sciences | −.02 / .09 | ||||||||
| Medical Sciences | −.006 / .09 | ||||||||
| Object of Appraisal (RC = Abstract) | |||||||||
| Paper | −.06 / 0.08 | ||||||||
| Cohort (RC = unknown) | |||||||||
| 1950–1979 | .10 / .09 | ||||||||
| 1980–1989 | .07 / .11 | ||||||||
| 1990–1999 | .15 / .15 | ||||||||
| 2000–2008 | −.007 / .12 | ||||||||
| Blinding (RC = unknown) | |||||||||
| Double | .15 / .11 | ||||||||
| Single | −.05 / .08 | ||||||||
| Rating System (RC = unknown) | |||||||||
| Categorical | −.40 / .14* | −.32 / .11* | |||||||
| Metric | −.33 / .16* | −.33 / .13* | |||||||
| Random effects (σ2) | |||||||||
| Study Level 3 | .03 /.01* | .016 / .009* | .03 / .012* | .03 / .01* | .027 / .01* | .03 / .01* | .03 / .01* | .017 / .01 | .0036 / .02 |
| Coefficient Level 2 | .01 /.009 | .007 / .007 | .009 / .009 | .009 / .009 | .009 / .009 | .007 / .008 | .005 / .007 | .01 /.01 | .01 / .02 |
| −2LL | −8.4 | −23.7 | −8.5 | −8.8 | −12.7 | −10.7 | −10.7 | −15.3 | −30.0 |
| BIC | 2.0 | −9.9 | 1.9 | 8.5 | −2.4 | 10.1 | 3.2 | −1.4 | −12.6 |
Note: For each categorical variable, one category was chosen as a reference category (RC, e.g., RC = Social Sciences for the categorical variable discipline). For categorical variables, effect for each predictor variable (a dummy variable representing one of the categories) is a regression coefficient (Coeff) that should be interpreted in relation to its standard error (SE) and the effect of the reference category. Variance components for level 1 are derived from the data, but variance components at level 2 and level 3 indicate the amount of variance that can be explained by differences between studies (level 3) and differences between single reliability coefficients nested within studies (level 2). The loglikelihood test provided by SAS/proc mixed (−2LL) can be used to compare different models, as can also the Bayes Information Criteria (BIC). The smaller the BIC, the better the model is.
*p<.05.