Reliability of an Interactive Computer Program for Advance Care Planning

Abstract

Despite widespread efforts to promote advance directives (ADs), completion rates remain low. Making Your Wishes Known: Planning Your Medical Future (MYWK) is an interactive computer program that guides individuals through the process of advance care planning, explaining health conditions and interventions that commonly involve life or death decisions, helps them articulate their values/goals, and translates users' preferences into a detailed AD document. The purpose of this study was to demonstrate that (in the absence of major life changes) the AD generated by MYWK reliably reflects an individual's values/preferences. English speakers ≥30 years old completed MYWK twice, 4 to 6 weeks apart. Reliability indices were assessed for three AD components: General Wishes; Specific Wishes for treatment; and Quality-of-Life values (QoL). Twenty-four participants completed the study. Both the Specific Wishes and QoL scales had high internal consistency in both time periods (Knuder Richardson formula 20 [KR-20]=0.83–0.95, and 0.86–0.89). Test-retest reliability was perfect for General Wishes (κ=1), high for QoL (Pearson's correlation coefficient=0.83), but lower for Specific Wishes (Pearson's correlation coefficient=0.57). MYWK generates an AD where General Wishes and QoL (but not Specific Wishes) statements remain consistent over time.

Introduction

Advance care planning is the process by which people plan for future medical treatment in the event that they cannot speak for themselves. It is typically accomplished by completing advance directives (ADs), which outline specific health care instructions and/or designate a proxy decision maker. Because up to 75% of adults lack decision-making capacity when life-or-death medical decisions must be made¹ and neither family members nor doctors accurately predict what patients want,^2,3 the absence of advance planning can lead to moral distress for decision makers,⁴ medical care inconsistent with an individual's wishes,⁵ and unintended financial burdens to patients, families, and society.⁶

Despite widespread agreement that individuals should plan for their medical futures,⁷ AD completion rates remain consistently low,⁸ and significant barriers exist to their implementation, including unease about the value of ADs.^4,9–17

To address some of these concerns, we (BHL and MJG) developed Making Your Wishes Known: Planning Your Medical Future (MYWK), which has been described in detail elsewhere.^18–26 In brief, MYWK is an interactive computer-based decision aid that guides users through the process of advance care planning by providing tailored education, values clarification exercises, and a decision-making algorithm that generates a personalized AD documenting an individual's values/goals/preferences. The present study examines the reliability of this decision aid by exploring whether (absent major life changes) the AD generated by MYWK on two separate occasions remains stable over time.

Methods

Study procedure

Study participants were recruited from 121 individuals who responded to a research advertisement posted in a local senior center. During the initial phone call, eligible individuals were invited to attend an in-person session at which informed consent was elicited and screening was conducted to assure that participants could read at the eighth-grade level (≥26 on the Wide Range Achievement Test [WRAT-3]),²⁷ were cognitively able to use the program (≥25 on the Mini–Mental State Examination),²⁸ and did not have moderate/severe depression (≤19 on the Beck Depression Inventory-II).²⁹ Study participants then completed a demographic questionnaire and the MYWK computer program. During the second study visit 4 to 6 weeks later, participants again completed the MYWK program plus a questionnaire about interim life events that might influence responses to health care decisions, and they subsequently received a $25 gift certificate.

Intervention details

The computer-based decision aid MYWK includes six sections (described elsewhere in fuller detail¹⁸). “Getting Started” provides an overview of the program. “Choosing a Spokesperson” reviews surrogate decision making and prompts users to designate primary and alternate spokesperson(s). “Exploring Your Values” helps users clarify their values and goals regarding medical care, death and dying, and disability. “Your Medical Wishes” explains health conditions (stroke, dementia, coma, and terminal illness) that can prevent a patient from communicating preferences for medical treatments, describes interventions that commonly involve life-or-death decisions (cardiopulmonary resuscitation [CPR], mechanical ventilation, dialysis, and tube feeding), and prompts users to make a series of decisions involving specific conditions and treatments. Individuals' responses become data for the program's decision-making algorithm, which creates an AD document that users review in the section “Putting It All Together”—confirming and editing: 1) their choice for surrogate decision maker(s); 2) the general wishes statement chosen to best reflect a general stance toward life-sustaining medical interventions; and 3) their wishes regarding treatments under various conditions. “The Next Step” reinforces the importance of communicating one's wishes and explains how to print the AD document.

Intervention procedure

Participants completed the MYWK program twice, 4 to 6 weeks apart. Each in-person session lasted 1 to 3 hours, including completion of demographic questions at visit 1, and interim life events questions at visit 2.

Statistical methods

To assess reliability, three components of the AD document were examined: 1) General Wishes statements; 2) Specific Wishes for treatment under various scenarios; and 3) Quality-of-Life (QoL) values. The stability of responses between the two time points was examined using kappa coefficients. Stability of the (combined) Specific Wishes scale and QoL scale (sum score of binary items) was assessed by Pearson's correlation coefficients between scores at the two time points. Internal consistency of the scales was evaluated by using Cronbach alpha coefficients (equivalently, the Knuder Richardson formula 20 [KR-20] for binary responses).

Unidimensionality of the Specific Wishes and QoL scales were assumed beforehand. Despite the small sample size (n=24), we examined the tenability of this assumption by conducting a confirmatory factor analysis. The MPlus latent variable modeling program (Muthén & Muthén, Los Angeles, CA) was used to fit the one factor model for binary items and the default weighted least squares mean variance (WLSMV) estimator was used to derive the parameters of the factor models. Items with particularly low factor loadings were considered for removal from the scales.

Results

To reach the recruitment goal, 67 individuals were contacted, of whom 18 could not be reached and 20 declined participation (reasons included being too busy, already having a living will, or discomfort working with computers). Of the 29 people who agreed to participate, 3 did not show up for their scheduled study visit and 2 screen-failed (for depression). The remaining 24 participants completed both study visits (79% female; mean age 68 years, range: 43–89), of whom 37% reported being college graduates, 63% being comfortable using a computer, 58% having previously created an AD, 42% having previously assigned a health care spokesperson, and 95% being in good (or better) health (1 excellent, 13 very good, 9 good, 1 fair, 0 poor).

At the second study visit, 20/24 (83%) reported no change in their medical wishes for treatment; 16 (67%) had shared their AD with others since study visit 1; and 22 (92%) had changed their mind about their spokesperson. Only 2/24 participants reported a major life event between visits 1 and 2, although 16/24 selected nonidentical options on this item's 5-point scale.

General wishes

Participants indicated their general wishes by selecting one of six statements (Table 1). Comparison between time 1 and time 2 yielded perfect agreement (κ=1.00).

Table 1.

General Wishes Statements

Item #	Wish statements
1.	I cherish my life regardless of its quality. I would want any and all medical treatments that might prolong my life, even if the result is a quality of life that others regard as very poor. This means that I want all treatments:
	• Even if treatment would prolong my life by only hours or days
	• Even if their chance of success is very low
	• Regardless of the cost of treatment
	• Regardless of the burden of treatment on me or others
2.	I cherish my life regardless of its quality. I would want all medical treatments that are likely to prolong my life, unless my family and loved ones would consider the burden to them to be unbearable. This means that I want all treatments:
	• Even if the result is a quality of life that others regard as very poor
	• Even if treatment would prolong my life by only hours or days
3.	I cherish my life, so long as my quality of life is acceptable. I want only those medical treatments that are likely to be successful in preserving what I consider a good quality of life. This means that if my quality of life is likely to be poor, I would rather live a shorter period of time than undergo medical treatments that prolong my life. For me, an unacceptably poor quality of life means:
	• (list of conditions/experiences drawn from user's responses to the program)
4.	I cherish my life, so long as my quality of life is acceptable and efforts to prolong it do not impose on my family and loved ones a burden they consider to be unbearable. I want only those medical treatments that would not impose such a burden and are likely to preserve what I consider a good quality of life—even if this means I would live a shorter period of time. For me, an unacceptably poor quality of life means:
	• (list of conditions/experiences drawn from user's responses to the program)
5.	I do not want any medical treatments that would prolong my life, unless the purpose of the treatments is to help other people, such as:
	• To make organ donation possible
	• To use my body for research or education
	• To allow family or friends to say goodbye to me
6.	I do not want any medical treatments that would prolong my life, even if the treatments would:
	• Return me to my current state of health
	• Decrease my discomfort
	• Have a high probability of success
	• Impose minimal burden on me or on others
	• Benefit others (such as organ donation, research or education)

Quality of life

Table 2 shows the agreement level and factor loadings for test-retest questions regarding conditions that might constitute a poor QoL (yes/no). This ranged from very high (κ=1.00) for “Had to get around in wheelchair” to quite low (κ=0.10) for “Could not have meaningful relationships.” Factor model convergence was achieved despite the small sample size at both time points; however, comparatively low factor loadings for items 1, 4, and 12 warranted their removal from the summed score QoL final scale.

Table 2.

What Counts as an Unacceptably Poor Quality of Life: T1 versus. T2 Comparison

Quality-of-life items	Agreement, expressed in Kappa valuea	Standardized factor loadings Time 1b	Standardized factor loadings Time 2b
1. Had to get around in wheelchair3	1.00	0.610	0.593
		(0.157)	(0.189)
2. Confined to bed all the time	0.89	0.971	0.882
	(0.69–1.00)	(0.031)	(0.085)
3. Severe pain most of the time	0.81	0.962	0.939
	(0.57–1.00)	(0.049)	(0.059)
4. Had to live permanently in nursing homec	0.70	0.631	0.648
	(0.31–1.00)	(0.169)	(0.189)
5. Cost of care a severe financial burden for family	0.65	0.829	0.732
	(0.34–0.95)	(0.156)	(0.152)
6. Discomfort most of the time	0.50	0.561	0.836
	(0.01–0.99)	(0.144)	(0.125)
7. Unable to control bladder/bowels	0.48	0.831	0.969
	(0.11–0.86)	(0.114)	(0.041)
8. Could not think clearly/confused most of time	0.42	0.876	0.934
	(0.02–0.83)	(0.011)	(0.047)
9. Care caused a severe burden for family	0.39	0.920	0.741
	(0.05–0.74)	(0.121)	(0.221)
10. Could no longer make own decisions	0.32	0.919	0.878
	(–0.08–0.72)	(0.073)	(0.092)
11. Could not communicate/be understood by others	0.23	0.963	0.875
	(–0.15–0.61)	(0.044)	(0.100)
12. Could not have meaningful relationships3	0.10	0.253	0.612
	(–0.25–0.43)	(0.271)	(0.189)

^a,b95% confidence intervals and standard errors in parenthesis.

^cDue to low factor loading, this item was not included in calculating the summed Quality of Life final scale.

QoL sum scores (mean, standard deviation [SD], range) of the final 9 items were 2.83 (2.82) [0–8] at time 1 and 3.63 (3.09) [0–9] at time 2. Internal consistency of the QoL scale as measured by the KR-20 was 0.87 at time 1 and 0.89 at time 2. Refitting the confirmatory factor model after excluding the three items resulted in the main factor explaining 77% of the total item variation at time 1 and 76% at time 2. The Pearson correlation coefficient between test and retest QoL scores was 0.81.

Specific wishes

The internal consistency of the Specific Wish items was consistently high across scenarios and time points, with the KR-20 ranging from 0.83 to 0.95. From the factor analysis models, the proportion of total item variations explained by the Specific Wish factor ranged from 0.70 to 0.92 within scenario. No items were removed from the wish scale. Mean scores varied across scenario from 5.50 to 0.50, with a possible range from 0 (no wish) to 8 (max wish).

Participants' Specific Wishes for treatment in response to five clinical scenarios at time 1 and time 2 varied across clinical scenarios. For example, we found perfect agreement (κ=1.00) regarding CPR in the event of irreversible coma, but low agreement regarding kidney dialysis lasting <1 month in the setting of a stroke that would not improve (κ=0.04). Combining the eight treatment items—kidney dialysis (up to 1 month; >1 month), CPR, mechanical ventilation (<24 hours; up to a month; >1 month), feeding tube (up to 1 month; >1 month)—we found high test-retest reliability only for the clinical scenarios involving coma (Table 3). However, a test for correlations across groups³⁰ was not able to reject the hypothesis of unequal correlations (p=0.093).

Table 3.

Clinical Scenario and Medical Wish Scale Characteristics at Time 1, Time 2

	Clinical Scenario	Moderate/severe stroke that would significantly improve within a year	Moderate/severe stroke that would NOT improve	Coma resolve within a year	Irreversible coma	Dementia
Descriptive Statistics	Mean (SD) [range]	T1:5.50(2.87) [0–8]	T1:1.83(2.75) [0–8]	T1:2.38(2.67) [0–8]	T1:0.50(1.32) [0–5]	T1:1.88(2.66) [0–8]
		T2:5.83(2.35) [1–8]	T2:2.83(2.67) [0–8]	T2:4.96(3.36) [0–8]	T2:0.79(2.00) [0–8]	T2:2.92(3.19) [0–8]
Test-retest reliability between Time 1 and Time 2	Pearson's correlation coefficient combined 8 treatment items	0.25	0.45	0.73	0.85	0.58
		95% CI (–0.17–0.59)	95% CI (0.05–0.72)	95% CI (0.46–0.87)	95% CI (0.68–0.93)	95% CI (0.24–0.80)
Agreement of individual wish items between T1 and T2	Kappa	<0.70	<0.70	<0.75	>0.70	<0.70
Internal consistency at individual time points	Cronbach alpha	T1: 0.92	T1: 0.93	T1: 0.95	T1: 0.83	T1: 0.92
		T2: 0.86	T2: 0.90	T2: 0.95	T2: 0.94	T2: 0.94
Unidimensionality	% item variation explained by wish factor (R2)	T1:0.83	T1:0.83	T1:0.92	T2: N/Aa	T1:0.70
		T2: N/Aa	T2: 0.77	T2:0.84	T2:0.74	T2:0.90

^aFactor model unable to be estimated because at least one item had only one response.

CI, confidence interval; SD, standard deviation.

Discussion

This pilot study shows that the computer-based decision aid MYWK was reliable in representing users' General Wishes and QoL preferences for future medical treatment when administered twice, 4 to 6 weeks apart. By contrast, we found lower consistency over time for participants' Specific Wishes for treatment in response to various clinical scenarios.

To better understand the sources of variation in the present findings, future studies will examine the reliability of MYWK by having individuals complete the program three times to help account for the impact that the MYWK program, itself might have on individuals' preferences. This is important because the decision aid not only provides education about end-of-life issues and prompts users to reflect on their values and preferences, but also encourages them to discuss their views with loved ones. As a result, we anticipate that the potential for change in a person's views and preferences is significantly greater between the first and second use of the decision aid than between the second and third use. Thus, reevaluating reliability across these three visits will help us better explain the variations seen in the present pilot study.

Limitations

Limitations to this study include a small sample size, single geographic location, a predominance of female and older participants, and the potential impact of MYWK itself on changes in individuals' values/preferences between time 1 and time 2. Another important limitation is that the test-retest method to establish reliability assumes that the object of measurement (the individual's values/preferences for end-of-life care) is stable over time. However, even in the absence of a major life event, the true scores for the variables we measured may be unstable, in which case the overall reliability of MYWK is diminished.³¹

Conclusion

MYWK generates an AD whose General Wishes and QoL (but not Specific Wishes) statements remain consistent over time. Additional studies are needed to assess the educational impact of MYWK on the stability of individuals' wishes for specific medical treatments.

Author Disclosure Statement

Two of the authors (BHL and MJG) have intellectual property and copyright interests for the decision aid MYWK used for this study.To encourage individuals to reflectively and systematically engage in advance care planning regarding end-of-life medical decisions, it is anticipated that MYWK will be made available free of charge for use by the general public, as well as for education purposes. However, users who wish to archive, revise, and electronically transmit AD documents will be charged a modest fee.

This study was funded by a grant from the National Institutes of Health (NIH), National Institute of Nursing Research (1R21NR008539), and Penn State University (Social Science Research Institute, Woodward Endowment for Medical Science Education, and Tobacco Settlement Fund Award).