Appendix Table 3:

Estimates of inter-rater agreement via two-level random effects models

Model	Model description	ICC	ICC 95% CI
Model 3	Binary response to the question “Is this patient’s goal achievable?”
3a.	Random effect for patient, no fixed effects	0.67	0.54 – 0.78
3b.	Random effect for patient, fixed effects for patient characteristicsb	0.64	0.51 – 0.76
3c.	Random effect for patient, fixed effects for rater’s training background	0.67	0.54 – 0.78
3d.	Random effect for patient, fixed effect for the combinationa of patient goal & code status	0.21	0.08 – 0.45
Model 4	Did the intervention meet the criteria for goal-concordant care?
4a.	Random effect for patient, no fixed effects	0.55	0.42 – 0.68
4b.	Random effect for patient, fixed effects for patient characteristicsb	0.53	0.40 – 0.66
4c.	Random effect for patient, fixed effects for rater’s training background	0.58	0.45 – 0.70
4d.	Random effect for patient, fixed effect for the combinationa of patient goal & code status	0.20	0.08 – 0.41

Abbreviations: CI, Confidence Interval; ICC, Intraclass correlation

^aThe 10 combinations of patient goals and treatment limitations were derived from a study of ICU proxies conducted at Johns Hopkins Hospital in 2016.

^bPatient characteristics include age, gender, ICU, sofa score, and days in the ICU at the time of the intervention