Inter-rater agreement of intensivists evaluating the goal-concordance of preference-sensitive ICU interventions

Abstract

Context:

Goal-concordant care has been identified as an important outcome of advance care planning and shared decision-making initiatives. However validated methods for measuring goal-concordance are needed.

Objectives:

To estimate the inter-rater reliability of senior critical care fellows rating the goal-concordance of preference-sensitive interventions performed in intensive care units (ICUs) while considering patient-specific circumstances as described in a previously-proposed methodology.

Methods:

We identified ICU patients receiving preference-sensitive interventions in 3 adult ICUs at Johns Hopkins Hospital. A simulated cohort was created by randomly assigning each patient 1 of 10 sets of goals and preferences about limiting life support. Critical care fellows then independently reviewed patient charts and answered two questions: 1) Is this patient’s goal achievable? and 2) Will performing this intervention help achieve the patient’s goal? When the answer to both questions was yes, the intervention was rated as goal-concordant. Inter-rater agreement was summarized by estimating intraclass correlation (ICC) using mixed-effects models.

Results:

Six raters reviewed the charts of 201 patients. Interventions were rated as goal-concordant 22% – 92% of the time depending on the patient’s goal-limitation combination. Percent agreement between pairs of raters ranged from 59% – 86%. The ICC for ratings of goal concordance was 0.50 (95% CI 0.31 – 0.69) and was robust to patient age, gender, ICU, severity of illness, and length of stay.

Conclusion:

Inter-rater agreement between intensivists using a standardized methodology to evaluate the goal-concordance of preference-sensitive ICU interventions was moderate. Further testing is needed before this methodology can be recommended as a clinical research outcome.

Introduction

Researchers evaluating interventions to facilitate advance care planning, patient-family engagement, and structured communication¹ share a common struggle: we lack a good outcome measure. Process measures, such as conducting family meetings and documenting patient preferences, are important but are not an end unto themselves. Meetings are conducted and preferences are documented to help patients receive care that matches their preferences and help achieve their goals. Measures of patient and family satisfaction are problematic. For example, families of patients who die may report greater satisfaction than those who survive,² and physicians providing an overly optimistic prognosis may receive higher satisfaction scores than their more accurate colleagues.³ Scales, such as The Quality of Dying and Death (QODD) and CAESAR,⁴ are sophisticated measures that capture the experiences of families. However, these scales may not measure a single unidimensional construct, do not apply to patients who survive their ICU stay, and are not currently recommended as primary outcome measures.^5,6

Faced with these challenges, stakeholders now cite “goal-concordant care” as an important outcome of both advance care planning,⁷ and communication interventions for seriously ill patients and families.^8,9 A policy statement endorsed by the American Thoracic Society and American College of Critical Care Medicine calls on intensivists to engage patients and proxies in a shared decision-making process to achieve goal-concordant care when “making major treatment decisions that may be affected by personal values, goals, and preferences.”^12,13 However, there are currently few objective measures of goal-concordance, making it challenging to assess whether this outcome was achieved.^7,10

We previously proposed a methodology for measuring the incidence of goal-concordant care in the ICU setting.¹¹ This proposal is part of a research agenda to develop a measure for evaluating the impact of patient-engagement and communication interventions on ICU care (see Appendix Figure 1). However for the proposed methodology to be valid, trained clinicians should demonstrate strong agreement when rating concordance between patients’ goals and preferences, and the care received. Therefore, we designed a simulation study using clinical data from real ICU patients to estimate the inter-rater reliability of senior critical care fellows rating the goal-concordance of preference-sensitive interventions in the ICU setting. Preference-sensitive interventions were defined using the consensus of an expert panel of ICU stakeholders that was convened to identify non-emergent ICU interventions requiring consideration of a patient’s goals and treatment limitations in routine critical care clinical practice.¹⁴ The panel reached consensus on 8 procedures referred to hereafter in this study as preference-sensitive interventions.¹⁴

Methods

Identifying ICU patients with orders for preference-sensitive interventions

In collaboration with the Johns Hopkins Institute for Clinical and Translational Research (ICTR), we developed a screening algorithm to identify orders in the electronic medical record (EMR) for 6 of the 8 previously-identified preference-sensitive interventions at Johns Hopkins Hospital (JHH).¹⁴ The 6 interventions identified by the screening algorithm were tracheotomy, peripherally inserted central catheter, nasogastric tube, in-hospital dialysis, percutaneous endoscopic gastrostomy, and long-term dialysis catheter. This algorithm was applied retrospectively to identify adult patients with orders for these interventions in the medical, surgical, and surgical-oncology ICUs at JHH during 2015. To estimate the inter-rater reliability using intraclass correlation (ICC) with precision of +/− 0.20, we randomly selected 40 patients with orders for each of the 6 preference-sensitive interventions creating a sample of 240 unique patients for analysis. (Figure 1)

Figure 1:

Study Flow Diagram

Simulating patient goals and treatment preferences

Patient goals and preferences for life-sustaining treatments are not reliably recorded in EMRs.^15,16 Therefore, we were not able to use the actual goals or treatment limitations of study patients for analyses. Instead, we used data on patient goals and preferences for the use of life-sustaining treatments collected as part of a study conducted in the JHH medical ICU.¹⁷ In this previous study, ICU proxies were asked about the patient’s goals using a 7-item, multiple-choice question with previously validated response options.^18–20 Proxies were also asked a 5-item multiple choice question about limitations in the use of life support preferred by their loved ones avoiding medical terminology. Together, these questions created 35 potential combinations of patient goal and treatment limitation. For the current study, we used the 9 most common combinations, plus the goal/limitation combination “to be comfortable” and “Focus on keeping me as comfortable as possible, even if that means I die sooner” creating 10 goal/limitation combinations. The comfort-focused combination was included because it described a unique subset of patients whose goals and preferences are qualitatively different from the majority of ICU patients. These 10 goal-limitation combinations were randomly assigned to the patients identified by the screening algorithm. The resulting cohort consisted of 240 patients with real physiologic data, real orders for preference-sensitive treatments, and a randomly assigned goal and preference about treatment limitations. From this point forward, this cohort is described as being comprised of “simulated patients” to acknowledge the simulated nature of the preferences.

Training intensivists to rate preference-sensitive interventions as goal-concordant or goal-discordant

In January 2017, 6 critical care fellows in their 2^nd-4^th year of training were hired and trained to rate preference-sensitive interventions as either concordant or discordant with a patient’s goals using a previously-published conceptual framework for goal-concordant care in the ICU.¹¹ Among the 6 raters, 3 fellows were in the Pulmonary and Critical Care Medicine (PCCM) program at JHH and had completed the majority of their clinical training in the medical ICU, and 3 fellows were in the Anesthesia and Critical Care Medicine (ACCM) program and had completed the majority of their training in JHH surgical ICUs. All fellows participated in a 90-minute in-person training, received a 25-page operations manual, and had to successfully rate 6 – 10 test patients before reviewing the charts of patients in the study cohort. The training and operations manual included the conceptual framework, guidance on reviewing patient charts, instructions for interpreting statements about patient goals and treatment limitations, and examples of preference-sensitive interventions that did and did not constitute goal-concordant care for each of the 10 assigned goal-limitation combinations.

After completing training, each rater worked independently to review the history and physical, consult notes, lab and imaging data, and progress notes up until the day of the order for a preference-sensitive intervention for 120 patients in the study cohort. After reviewing each patient’s chart and their assigned goal and treatment limitations, raters answered two questions:

1. Is this patient’s goal achievable?

2. Will performing this intervention, or continuing to use this intervention help achieve the patient’s goal?

Response options for both questions were “Yes or maybe” vs “No.” Interventions were defined as being goal-concordant if the rater selected “Yes or maybe” in response to both of these questions. Responses were entered into a data collection form in a REDCap database.²¹ Raters also abstracted data on each patient’s age, gender, and days of hospitalization and ICU care prior to the order for the preference-sensitive intervention. Patient’s SOFA score^22–24 on the day of the intervention order were calculated separately by an investigator (R. Nikooie). The study was approved by the Institutional Review Board of Johns Hopkins University (IRB00053330).

Data analysis

Patient characteristics were summarized using frequencies for categorical variables, and medians with interquartile ranges (IQR) for continuous variables. The proportion of patient goals rated as potentially achievable and the percent of interventions rated as goal-concordant were summarized for each of the 10 assigned goal-limitation combinations, by each rater and by rater fellowship program. Percent agreement was calculated for all pairs of raters. Finally, three-level, mixed-effects models with no independent covariates were fit for the binary response to the question “Is this patient’s goal achievable?” and for the definition of goal-concordance. In these models the goal-limitation combinations represent the third nesting level, individual patients represent the second nesting level, the responses of three raters are nested within each patient and the models included random intercepts for both the goal-limitation combination and patient. The intraclass correlation²⁵ from this model estimates the proportion of variance in responses explained by the random effects for patient and the patient’s assigned goal-limitation combination. To assess the robustness of this estimates, fixed effects for the patient’s age, gender, ICU and SOFA score at the time of the order, days in the ICU prior to the order, and for the rater’s fellowship program were included in the mixed models. As a sensitivity analysis, we also fit a two-level, mixed-effects model with the goal-limitation combination included as a fixed effect, i.e. a single random intercept for patient was included. Descriptive statistics were generated using R (Version 3.3.2; R Development Core Team, Vienna, Austria), and all models were fit using Stata (StataCorp. 2013. Stata Statistical Software: Release 13. College Station, TX: StataCorp LP.)

Results

We identified 203 eligible patients with 2 patients having missing data, leaving 201 patients for analysis (Figure 1). The median age, SOFA score, and ICU length of stay on the day of the order for a preference-sensitive intervention were 60 years (IQR 48, 68), 6 (IQR 3, 9), and 3 days (IQR 0, 10) respectively (Table 1). Each patient was reviewed and rated by 3 intensivists, creating 603 ratings. Intensivist ratings were strongly influenced by which goal and limitation was assigned. For example, the proportion of “Yes or maybe” responses to the question “Is this patient’s goal achievable?” ranged from 22% to 100% depending on the patient’s assigned combination of goal and treatment limitations (Table 2).

Table 1:

Characteristics of study patients

Characteristic	Patients, n = 201
Age, median (IQR)	60 (48, 68)
Female, n (%)	89 (44%)
ICU, n (%)
Medical	93 (46%)
Surgical Oncology	77 (38%)
Surgical	31 (15%)
SOFA score on the day of the order, median (IQR)	6 (3, 9)
Days in hospital prior to intervention, median (IQR)	4 (1, 14)
Days in ICU prior to intervention, median (IQR)	3 (0, 10)
Intervention
Tracheotomy	40 (20%)
Peripherally inserted central catheter	38 (19%)
Nasogastric tube	37 (18%)
In-hospital dialysis	36 (18%)
Percutaneous endoscopic gastrostomy	34 (17%)
Long-term dialysis catheter	16 (8%)

Abbreviations: ICU, intensive care unit; IQR, interquartile range; SOFA, sequential organ failure assessment

Table 2:

Summary of simulation ratings by assigned patient goal and treatment limitations

			Patient goal rated as potentially achievable given treatment limitations			Interventions rated as goal-concordanta
Goal-limitation combinationb	Patient Goalc	Treatment Limitationsd	All intensivists	PCCMe	ACCMe	All intensivists	PCCMe	ACCMe
1: n=60	To be cured	None	38%	31%	45%	38%	31%	45%
2: n=54	To be cured	Some	22%	4%	38%	22%	4%	38%
3: n=66	To be cured	Unsure	35%	32%	38%	35%	32%	38%
4: n=66	To live longer	None	94%	97%	91%	88%	85%	91%
5: n=57	To live longer	Unsure	98%	96%	100%	93%	84%	100%
6: n=63	To improve health	None	81%	72%	88%	79%	69%	88%
7: n=60	To improve health	Some	95%	91%	100%	92%	85%	100%
8: n=69	To improve health	Unsure	87%	92%	82%	86%	89%	82%
9: n=54	To maintain health	None	76%	78%	74%	76%	78%	74%
10: n=54	To be comfortable	Strong	100%	100%	100%	35%	8%	61%

^aInterventions are defined as being goal-concordant if the rater selected “Yes or maybe” in response to the question “Is this patient’s goal achievable?” and the question “Will performing this intervention, or continuing to use this intervention help achieve the patient’s goal?”

^bThe 10 combinations of patient goals and treatment limitations were derived from a study of ICU proxies conducted at Johns Hopkins Hospital in 2016. Examples and instructions on how to interpret each combination were provided in the training and operations manual provided to each rater.

^cRaters were instructed to interpret patient goal statements as follows: To be cured - To return to a state of health that a person of the same age without any significant illness or injury is expected to experience. To live longer - To live long as possible in any health state, even if they require 24–7 care in an acute care facility, LTAC, or chronic vent facility. To improve health - To be well enough to live outside the hospital setting. Living in a nursing home or discharge to a rehabilitation facility is an acceptable outcome for these patients. To maintain health - To be well enough to return to the baseline health status and level of independence that they personally had before hospital admission. To be comfortable - To prioritize comfort over longevity and hope to be as free of pain and discomfort as possible.

^dTreatment limitations were interpreted as follows: None - Use life support machines to keep me alive no matter what. If my heart stops, do CPR. Some - Try to help me get better, but don’t use life support machines and if my heart stops don’t do CPR. Strong - Focus on keeping me as comfortable as possible, even if that means I die sooner. Unsure - I (the patient’s proxy) don’t know what the patient would say

^eACCM refers to the subset of ratings by 3 fellows in the Anesthesia and Critical Care Medicine program who trained primarily in surgical ICUs. PCCM refers to ratings by 3 fellows in the Pulmonary and Critical Care Medicine program who trained primarily in the medical ICU.

The difference between PCCM and ACCM raters was greatest when the patient’s goal was to prioritize comfort even if it meant dying sooner (goal-limitation combination #10) and when a patient’s goal was to return to a state of health that a person of the same age without any significant illness or injury is expected to experience (i.e. To be cured) and the patient’s treatment limitation statement was “Try to help me get better, but don’t use life support machines and if my heart stops don’t do CPR” (goal-limitation combination #2). Under both of these conditions, there was unanimous agreement about goal-concordance for only 44% of patients. When comfort was prioritized, PCCM vs AACM fellows rated interventions as being goal-concordant 8% and 61% of the time respectively (Table 2). Under goal-limitation combination #2 the PCCM fellows rated patient goals as potentially achievable 4% of the time, while intensivists in the ACCM fellowship program rated patient goals as potentially achievable 38% of the time. These two goal-limitation combinations remained the most divisive even when comparing the responses of raters in the same fellowship program (Appendix Table 1).

Percent agreement about whether a patient’s goal was potentially achievable ranged from 73% to 91% across the 15 rater pairings (Appendix Table 2). Percentage agreement on the goal-concordance of interventions ranged from 59% to 86%. Inter-rater reliability estimates are presented in Table 3, using the intraclass correlation coefficient (ICC). The ICC represents the fraction of the total variance that is due to variability in ratings between groups (patients) as opposed to within groups (variation in responses among raters assigned to the same patient). Perfect agreement among raters generates an ICC value of 1, while an ICC of 0 can occur if the raters randomly generate their response (i.e. a flip of a coin).” The intraclass correlation (ICC) for responses to the question “Is this patient’s goal achievable?” was 0.64 (95% confidence interval (CI) 0.41 – 0.82). Adding fixed effects for patient characteristics (0.62, 95% CI 0.37–0.81) or the rater’s fellowship program (0.65, 95% CI 0.43 – 0.83) to the model did not change the ICC substantially. The ICC for ratings of goal concordance was 0.50 (95% CI 0.31 – 0.69), and again remained relatively unchanged when fixed effects for patient characteristics (0.47, 95% CI 0.28 – 0.68) or rater’s fellowship program (0.53, 95% CI 0.34 – 0.72) were included in the model. The sensitivity analyses using a two-level random effects model yielded similar ICCs (Appendix Table 3).

Appendix Table 2:

Percent agreement between raters

Agreement in response to the question: “Is this patient’s goal potentially achievable?”
	Rater
	a – PGY5	b – PGY8	c – PGY6	d – PGY9	e – PGY5	f – PGY5
a – PGY5	n = 98
b – PGY8	79%	n = 101
c – PGY6	87%	82%	n = 98
d – PGY9	78%	73%	80%	n = 99
e – PGY5	77%	69%	91%	80%	n = 103
f – PGY5	83%	69%	78%	76%	77%	n = 104
Does the intervention meet the definition of goal-concordant carea for this patient?
	a– PGY5	b – PGY8	c – PGY6	d – PGY9	e – PGY5	f – PGY5
a – PGY5	n = 98
b – PGY8	74%	n = 101
c – PGY6	80%	74%	n = 98
d – PGY9	73%	59%	80%	n = 99
e – PGY5	69%	60%	86%	80%	n = 103
f – PGY5	75%	60%	76%	71%	70%	n = 104

^aInterventions are defined as being goal-concordant if the rater selected “Yes or maybe” in response to the question “Is this patient’s goal achievable?” and the question “Will performing this intervention, or continuing to use this intervention help achieve the patient’s goal?”

Table 3:

Estimated inter-rater agreement from a three-level random effects model

Model	Model description	ICC	95% CI
Model 1	Response to the question “Is this patient’s goal achievable?”
1a.	Random effects for patient and goal-limitation combinationa, no fixed effects	0.64	0.41 – 0.82
1b.	Random effects for patient and goal-limitation combinationa, fixed effects for patient characteristicsb	0.62	0.37 – 0.81
1c.	Random effects for patient and goal-limitation combinationa, fixed effects for rater’s fellowship program	0.65	0.43 – 0.83
Model 2	Did the intervention meet the criteria for goal-concordant care?
2a.	Random effects for patient and goal-limitation combinationa, no fixed effects	0.50	0.31 – 0.69
2b.	Random effects for patient and goal-limitation combinationa, fixed effects for patient characteristicsb	0.47	0.28 – 0.68
2c.	Random effects for patient and goal-limitation combinationa, fixed effects for rater’s fellowship program	0.53	0.34 – 0.72

Abbreviations: CI, Confidence Interval; ICC, Intraclass correlation

^aThe 10 combinations of patient goals and treatment limitations were derived from a study of ICU proxies conducted at Johns Hopkins Hospital in 2016.

^bPatient characteristics include age, gender, ICU, sofa score, and days in the ICU at the time of the intervention

Discussion

This study is the first to evaluate the inter-rater reliability of a measure of goal-concordance for clinical research in the ICU setting. Our findings suggest that after a single training session, the inter-rater reliability as measured by intraclass correlation lay between 0.31 and 0.69, and was robust to patient age, severity of illness, and length of stay. While further development is needed before this methodology can be recommended, these findings are encouraging.

There is currently great enthusiasm for measuring consistency between patient preferences and the care received.²⁶ However ICU patient goals and preferences are not consistently elicited, are often poorly documented,^{15,16,27–31} and may change over time³² creating major logistical challenges to measurement. As a result, current recommendations are to measure 1) the timing of serious illness communication, 2) patient or surrogate experience, and 3) whether bereaved caregivers report that their loved ones received care consistent with their preferences.¹⁰ We support these recommendations, and also advocate for a direct measure of goal-concordance.

The methodology evaluated in this study in unique in using decisions, not patients, as the unit of analysis. For example, if 48 preference-sensitive interventions are ordered in an ICU and 36 are rated as goal-concordant, the incidence rate of goal-concordant treatment is 75 per 100 preference-sensitive decisions (95% CI 60 – 86). This approach allows complex or long-stay patients to contribute more data than short-stay patients, and produces more precise estimates (smaller confidence intervals) of ICUs that perform more preference-sensitive treatments. Requiring raters to consider the patient-specific circumstances and context of the intervention also avoids penalizing clinicians for responding appropriately in unusual situations. For example, a percutaneous endoscopic gastrostomy (PEG) tube is normally inappropriate for a patient prioritizing comfort at the end of life. However, a PEG tube could be rated as goal-concordant if it was placed to vent the stomach of a patient with a malignant bowel obstruction who wished to spend his final days at home.³³

Inter-rater agreement in this study was heavily influenced by patient goal. This means that the inter-rater reliability of this measure will vary depending on the mix of goals expressed by patients in a study cohort. Therefore, a clinical researcher considering this measure as an outcome would be well-served to collect preliminary data on the goals expressed by patients or proxies in their study population.

Discordance in the ratings of fellows who train in the medical vs surgical ICU highlight the differences in how goal-concordance is interpreted across sub-specialties. This suggests the value of efforts to develop consensus across specialties in the implementation of goal-concordant care and to develop interdisciplinary educational programs to facilitate shared understanding and implementation of goal-concordant care.

Our study was limited by using simulated clinical scenarios created by randomly assigned goals and preferences about limiting treatment to actual ICU patients from a single hospital. This could have resulted in a disproportionate number of situations where patients identified unachievable goals. We also required raters to work completely independently. In the setting of a clinical study, blinded raters could confer about cases where there was initial disagreement. The process of continually discussing such cases may lead to better shared understanding of what interventions constitute goal-concordant care. Finally, all raters in this simulation were fellows at a single institution who may have demonstrated higher or lower agreement than more senior intensivists or intensivists from other sites.

In conclusion, this simulation study found moderate inter-rater agreement between intensivists using a standardized methodology to evaluate the goal-concordance of preference-sensitive interventions in ICUs. Agreement was not substantially impacted by patient age, length of stay, or severity of illness, but did vary by the patient’s goals and preferences. Further testing and development is needed before this methodology can be recommended as an outcome in clinical research. This will require following ICU patients and their proxies longitudinally, documenting their changing goals and limitations over the course of a hospital stay, and allowing physicians (including attending physicians) to determine whether preference-sensitive interventions constituted goal-concordant care.

APPENDIX

Figure 1:

Methodology for Developing a Measure of Goal-concordant Care in the ICU Setting for Reaserch Purposes

Appendix Table 1:

Chart review results by assigned patient goal, treatment limitations, and rater

			Patient goal rated as potentially achievable given treatment limitations						Interventions rated as goal-concordanta
			Rater (Raters a-c are PCCM fellows; raters d-f are ACCM fellows)
Goal-limit combinationb	Patient Goalc	Treatment Limitationsd	a n=98	b n=101	c n=98	d n=99	e n=103	f n =104	a n=98	b n=101	c n=98	d n=99	e n=103	f n=104
1: n=60	To be cured	None	30%	38%	17%	88%	56%	14%	30%	38%	17%	88%	56%	14%
2: n=54	To be cured	Some	10%	0%	0%	60%	12%	36%	10%	0%	0%	60%	12%	36%
3: n=66	To be cured	Unsure	46%	40%	0%	50%	22%	40%	46%	40%	0%	50%	22%	40%
4: n=66	To live longer	None	91%	100%	100%	80%	100%	83%	91%	67%	100%	80%	100%	83%
5: n=57	To live longer	Unsure	100%	90%	100%	100%	100%	100%	80%	80%	90%	100%	100%	100%
6: n=63	To improve health	None	80%	75%	64%	82%	82%	100%	80%	62%	64%	82%	82%	100%
7: n=60	To improve health	Some	82%	100%	93%	100%	100%	100%	82%	88%	86%	100%	100%	100%
8: n=69	To improve health	Unsure	100%	79%	100%	100%	71%	80%	100%	79%	90%	100%	71%	80%
9: n=54	To maintain health	None	83%	71%	80%	75%	88%	64%	83%	71%	80%	75%	88%	64%
10: n=54	To be comfortable	Strong	100%	100%	100%	100%	100%	100%	20%	0%	0%	57%	55%	70%

Abbreviations: ACCM refers to the subset of ratings by 3 fellows in the Anesthesia and Critical Care Medicine program who trained primarily in surgical ICUs. PCCM refers to ratings by 3 fellows in the Pulmonary and Critical Care Medicine program who trained primarily in the medical ICU

^aInterventions are defined as being goal-concordant if the rater selected “Yes or maybe” in response to the question “Is this patient’s goal achievable?” and the question “Will performing this intervention, or continuing to use this intervention help achieve the patient’s goal?”

^bThe 10 combinations of patient goals and treatment limitations were derived from a study of ICU proxies conducted at Johns Hopkins Hospital in 2016. Examples and further instructions on how to interpret each combination were provided in the training and operations manual provided to each rater.

^cRaters were instructed to interpret patient goal statements as follows: To be cured - To return to a state of health that a person of the same age without any significant illness or injury is expected to experience. To live longer - To live as long as possible in any health state, even if they require 24–7 care in an acute care facility, LTAC, or chronic vent facility. To improve health - To be well enough to live outside the hospital setting. Living in a nursing home or discharge to a rehabilitation facility is an acceptable outcome for these patients. To maintain health - To be well enough to return to the baseline health status and level of independence that they personally had before hospital admission. To be comfortable - To prioritize comfort over longevity and hope to be as free of pain and discomfort as possible.

^dTreatment limitations were interpreted as follows: None - Use life support machines to keep me alive no matter what. If my heart stops, do CPR. Some - Try to help me get better, but don’t use life support machines and if my heart stops don’t do CPR. Strong - Focus on keeping me as comfortable as possible, even if that means I die sooner. Unsure - I (the patient’s proxy) don’t know what the patient would say.

Appendix Table 3:

Estimates of inter-rater agreement via two-level random effects models

Model	Model description	ICC	ICC 95% CI
Model 3	Binary response to the question “Is this patient’s goal achievable?”
3a.	Random effect for patient, no fixed effects	0.67	0.54 – 0.78
3b.	Random effect for patient, fixed effects for patient characteristicsb	0.64	0.51 – 0.76
3c.	Random effect for patient, fixed effects for rater’s training background	0.67	0.54 – 0.78
3d.	Random effect for patient, fixed effect for the combinationa of patient goal & code status	0.21	0.08 – 0.45
Model 4	Did the intervention meet the criteria for goal-concordant care?
4a.	Random effect for patient, no fixed effects	0.55	0.42 – 0.68
4b.	Random effect for patient, fixed effects for patient characteristicsb	0.53	0.40 – 0.66
4c.	Random effect for patient, fixed effects for rater’s training background	0.58	0.45 – 0.70
4d.	Random effect for patient, fixed effect for the combinationa of patient goal & code status	0.20	0.08 – 0.41

Abbreviations: CI, Confidence Interval; ICC, Intraclass correlation

^aThe 10 combinations of patient goals and treatment limitations were derived from a study of ICU proxies conducted at Johns Hopkins Hospital in 2016.

^bPatient characteristics include age, gender, ICU, sofa score, and days in the ICU at the time of the intervention