Reliability and Validity of the Traditional Chinese Version of the Advance Care Planning Engagement Survey: A Pilot Evaluation in Taiwanese Outpatients

Abstract

Objective: The Advance Care Planning Engagement Survey (ACP-ES) has proven effective in evaluating individuals’ engagement in advance care planning (ACP). However, a Traditional Chinese version of ACP-ES (ACPES-TC) has not yet been developed. Therefore, this study aimed to translate and preliminarily validate the ACPES-TC in the Taiwanese context. Material and Methods: A forward and backward translation process was conducted. The translated questionnaire was confirmed by clinical and academic experts. The ACPES-TC was then evaluated for its reliability and validity with participants in the community and from an outpatient clinic in a medical center in Northern Taiwan. The participants comprised healthy people aged 20 to 30 years and patients ≥55 years old, recruited from September 17 to October 28, 2019. Results: Seventy people were recruited, including 20 people aged 20 to 30 years in the community and 50 patients ≥ 55 years old from clinics. The ACPES-TC scores are significantly higher among those of older age, having financial independence, and under long-term medication (p < .05). The patients’ preference for health-related decision-making is significantly correlated with the ACPES-TC score; the point-biserial correlation coefficient is 0.46 (p < .001). The discriminant and criterion-related validities are verified. The ACPES-TC demonstrated a good internal consistency (Cronbach's α .97), acceptable one-week test-retest reliability (overall intraclass correlation coefficient 0.86), and low practice effect between the test and retest (Cohen's d .43). Conclusion: The overall reliability and validity of the ACPES-TC are fair, which could be used to evaluate the patients’ engagement in ACP in Taiwan. However, further studies with a full-scale psychometric evaluation are needed.

Introduction

Advance care planning (ACP) is a process where individuals of sound mind plan the medical treatments they will undergo should they lose their ability to make decisions in the future. The crux of ACP is establishing patients’ values and treatment preferences through clinician-patient interactions, and then incorporating these values into treatment objectives and preferences if the patients become disabled. ¹ In addition, patients would be able to communicate their end-of-life (EOL) care desires, as well as plan about and prepare for future care, through the ACP. However, by conducting ACP conversations with the individual and their family members, healthcare providers may learn about the individuals’ care expectations and life objectives. ²

Patient-centered care has long been emphasized in terms of both concept and delivery. The Patient Self-Determination Act was enacted in the United States in 1990, 30 years before Taiwan. Taiwan's government has also been actively promoting the Hospice and Palliative Care Act to facilitate terminal patients’ autonomy on EOL decisions for 30 years. However, many clients with other diseases (eg, patients suffering from early dementia or vegetative states) were unable to express their care preferences since they were not yet in the terminal stages of their diseases. Therefore, The Patients Right to Autonomy Act (PRAA) was legislated in 2016, and it was made official in early 2019 in Taiwan to include the disease groups mentioned above for respecting their care decisions. This political movement made Taiwan the first country in Asia to legalize the ACP discussions process and provide legally binding written documents/records advance decisions (AD). Specifically, individuals with mental capacity can request signing an AD after careful and professional ACP consultation to formulate their treatment goals and preferences in the event that they get a serious disease, or their health condition deteriorates. ³

Evidence showed that undertaking ACP can benefit the patients and their family caregivers. This includes (1) facilitating the AD completion rate, (2) increasing patient satisfaction with services, and (3) reducing anxiety, stress, and depression in their family members. ⁴ However, after a year of campaign following the legislation's enactment (September 15, 2020), the AD completion rate in Taiwan remains low, with only 16,603 completed ADs, ⁵ or around 0.086% of the current adult population (over 20 years old; n = 19, 403, 128 as of August 31, 2020). ⁶

Culture shapes ACP differently. It is crucial to provide responsive healthcare to patients and their family members based on different cultural values on EOL care. Evidence suggests that cultural adaptation is the key to increase the ACP uptake and the AD completion rate. ⁷ For example, a study in Taiwan exploring older adults and their family members’ perspectives on AD revealed that older adults tended to (1) receive less medical information; (2) not used to consider ACP-related issues; (3) be less aware of disease severity; (4) have no access to ACP, and (5) have little chance to discuss with their family members prior to EOL care decision-making. ⁸ In addition, Chinese people tend to avoid discussing death and dying issues since it is considered taboo and might cause bad luck. Furthermore, family-led decision-making is commonly noticed.^9,10

Although ACP has been seen as beneficial and endorsed worldwide to improve palliative care quality, ¹¹ the ACP intervention varies, and there are very few outcome measures to evaluate the quality of ACP intervention. ¹² Evidence showed that a patient's readiness to engage in the ACP conversation is important for ACP quality, ¹³ as the higher the level of readiness, the greater the self-efficacy for ACP engagement. ¹⁴ Sudore et al developed Advance Care Planning Engagement Survey (ACP-ES) as a validated measurement to investigate an individual's readiness and engagement in ACP.^15,16 This survey has been translated and validated into other languages (eg, Japanese), ¹⁷ but not in Traditional Chinese. Therefore, the present study aimed to translate the ACP-ES into the Traditional Chinese version (ACPES-TC) and conduct a preliminary validation study in Taiwan.

Material and Methods

Original ACP-ES

The ACP-ES evaluates future medical care-related decision-making processes in individuals, regardless of age and health condition. The development of the ACP-ES scale in 2013 comprised the measurement dimensions from the Process Measures to Action Measures. ¹⁵ In 2017, the Sudore et al reconstructed the original 82 ACP-ES questions to form a truncated questionnaire. ¹⁶ At present, the shortest version comprises four questions in two dimensions: Quality of Life (QoL) and Medical Decision-Maker. The scale is based on Likert's five-point items and average sum scores from the four questions. A higher score indicates better preparation for ACP. The Cronbach's α value was .84 for the 4-question ACP-ES.^16,17

Study Procedure

The development and validation of the ACPES-TC comprised two steps: (1) a forward and backward translation of the original ACP-ES (4 items version), ¹⁶ and (2) criterion-related validity, discriminant validity, and test-retest reliability. This study was extended research in the HOLISTIC project ¹⁸ and has been reviewed and approved by the Institutional Review Board in Cathay General Hospital (ref: CGH-P108045).

Step 1: Forward and Backward Translation of ACP-ES (4 Items Version)

Two hospice and palliative care physicians conducted forward translation of ACP-ES into Traditional Chinese (the official writing language in Taiwan) independently. The translated questions were then evaluated by an expert panel (two physicians, two nurses, and one pharmacist) in terms of clarity, appropriateness, and face validity. A Traditional Chinese version of ACP-ES was confirmed, followed by a backward translation into English by a professional translation agency. Finally, the expert panel compared the back-translated English version with the original version and provided suggestions and corrections to the final version of ACPES-TC.

Step 2: Scale Validation

The ACPES-TC was delivered to study participants for testing its criterion-related validity, discriminant validity, and test-retest reliability. We explored the association between ACPES-TC scores and three ACP preparation questions suggested by expert panels to evaluate the criterion-related validity (refer to statistical analysis section for detail). Discriminant validity was assessed by comparing the survey results between older adults aged ≥55 years (n = 50) and healthy young adults aged 20 to 30 (n = 20). Test-retest reliability was performed in a 7-day interval.

Sample Size Calculation and Study Participants Recruitment

The sample size calculation was based on the original ACP-ES publication in 2013. ¹⁵ According to their results, a significant effect size was found (d ≒ 1.37, calculated by the G*Power), implying that the sample size of only 20 participants was sufficient. However, ACP is a new and culture-sensitive issue in Asia; we cautiously assumed the effect size = 0.8 (large effect size), α = 0.05, and power = 0.80 to reach the requisite statistical power and consider the sample attrition. The requisite sample size was calculated for 52 people, but we finally decided to enroll 70 participants with the same distribution of age groups in the original ACP-ES publication. ¹⁵

Participants aged ≥55 years were recruited from an outpatient clinic in a medical center in Northern Taiwan, and 20 healthy individuals aged 20 to 30 years were recruited from the community. Healthy individuals cannot have any chronic disease or take any regular medication. Those participants who were unwilling to give the informed consent, could not communicate in either Mandarin or Taiwanese, or had a potential mental impairment that the investigators were concerned about were excluded. The recruitment period was from September 17 to October 28, 2019.

Questionnaire Measurement

The survey for validating ACPES-TC comprises (1) nine questions on their demographic characteristics (eg, age, gender, education, religion, living status, marital status, economic status, medication, and disease history); (2) four questions on ACPES-TC; (3) one question on medical decision-making for themselves; (4) one question on AD completion status, and (5) one question on medical decisions made for others when they are seriously ill or near the end of their life.

Statistical Analysis

Data were imported into Excel and managed and analyzed in SPSS version 25® (SPSS Inc.). Sample size estimation was conducted by the G*Power 3.1.9.7. (Heinrich-Heine-Universität Düsseldorf, Germany). ¹⁹

Scale Reliability

Cronbach's α, intraclass correlation coefficient (ICC), and Cohen's d were examined for reliability. Cronbach's α was used to indicate the internal consistency of the ACPES-TC, whereas the value ≥ 0.7 indicated an acceptable value of reliability. ²⁰ Considering the dependency between variables from repeated measures, the ICC was used to estimate the one-week test-retest reliability of the ACPES-TC. The value of ICC ≥ 0.7 was considered acceptable and the value ≥ 0.90 was deemed to be excellent. ²¹ Cohen's d was used to examine the practice effect.^22,23 The practice effect was classified into small, medium, and large for the coefficient values of .20–.49, .50–.79, and ≥ .8, respectively. ²⁴

Scale Validity

Discriminant validity and criterion-related validity were examined. Discriminant validity was performed using independent t-tests to determine the difference in ACPES-TC scores between participants with different demographic characteristics and medication used. Criterion-related validity is an indicator used to examine if the scale (ACPES-TC) has an empirical association with an established standard or “gold standard.” Three questions were used as gold standards for criterion-related validation: (1) “How have you preferred to make medical-related decisions? (by myself with or without consultation of medical professionals vs by relatives or medical professionals)”; (2) “Have you ever filled out an ACP? (yes vs no),” and (3) “Have you ever made medical decisions for other people who were seriously ill or near the end of their life? (yes vs no).” Given the dichotomous nature of the criterion questions and the continuous nature of the ACPES-TC score, the point-biserial correlation coefficient (r_pb) was used to estimate the correlation between the answers of the criterion questions and the ACPES-TC score. The coefficient ranges from −1 to 1 to indicate the direction and strength of the correlation.^22,23,25 The correlation coefficient is categorized into 0.70–0.89, 0.40–0.69, and 0.10–0.39, which indicate strong, moderate, and weak correlation, respectively, and a correlation coefficient ≥ 0.7 is considered acceptable. ²⁶

Results

We approached 70 participants (50 older adults and 20 younger adults) and received a 100% response rate. More than half of the participants are male (55.7%, n = 39), have a college degree or above (51.4%, n = 36), are religious (64.1%, n = 41), live with others (77.9%, n = 53), are married (68.1%, n = 47) and are financially independent (69.6%, n = 48). Furthermore, more than half of the participants are taking long-term medication (52.2%, n = 36) (Table 1).

Table 1.

Participant Characteristics and the Score of ACPES-TC in Older and Young Adults in Taiwan.

	Total (N = 70)	Age group		p-value for t/χ2
		20-30 years (n = 20)	≥55 years (n = 50)
ACPES-TC, mean ± SD	2.81 ± 1.20	2.34 ± 0.98	3.00 ± 1.23	.04
Age (years), mean ± SD	55.47 ± 22.37	21.75 ± 1.07	68.96 ± 7.38	<.001
Gender				<.001
Male, n(%)	39 (55.7)	19 (95.0)	20 (40.0)
Female, n(%)	31 (44.3)	1 (5.0)	30 (60.0)
Education				<.001
Below high school, n(%)	34 (48.6)	1 (5.0)	33 (66.0)
Above college, n(%)	36 (51.4)	19 (95.0)	17 (34.0)
Religious				.003
No, n(%)	23 (35.9)	12 (63.2)	11 (24.4)
Yes, n(%)	41 (64.1)	7 (36.8)	34 (75.6)
Living status				.01
Alone, n(%)	15 (22.1)	9 (45.0)	6 (12.5)
With others, n(%)	53 (77.9)	11 (55.0)	42 (87.5)
Marriage				<.001
Single, n(%)	22 (31.9)	19 (95.0)	3 (6.1)
Married, n(%)	47 (68.1)	1 (5.0)	46 (93.9)
Financial				<.001
Need support, n(%)	21 (30.4)	17 (85.0)	4 (8.2)
Independent, n(%)	48 (69.6)	3 (15.0)	45 (91.8)
Long-term medication				<.001
No, n(%)	33 (47.8)	20 (100.0)	13 (26.5)
Yes, n(%)	36 (52.2)	0 (0.0)	36 (73.5)

Abbreviations: SD, standard deviation; ACPES-TC, advance care planning engagement survey, traditional Chinese version.

The ACPES-TC scores and characteristics of older and young groups are presented in Table 1. The score of the ≥55 years old group (3.00 ± 1.23) was significantly higher than the 20 to 30 years old group (2.34 ± 0.98). The two groups also differed in terms of other characteristics. The ACPES-TC score was categorized by different characteristics in Supplemental Table 1, with the financially independent group (3.00 ± 1.18) scoring significantly higher than the financially dependent group (2.31 ± 1.13), and the group with long-term medication (3.15 ± 1.24) scoring significantly higher than the group without long-term medication (2.45 ± 1.07).

The Cronbach's α coefficient of the ACPES-TC scale was .97, indicating good internal consistency of the questionnaire items. As for one-week test-retest reliability, the overall ICC of the ACPES-TC scale was 0.86, which shows good overall reliability (Table 2). Among the four questions, the ICC value of the third question was slightly lower (ICC = 0.78) than the average (0.83–0.86), but it was still acceptable. The Cohen's d coefficient of the four questions and the ACPES-TC total score ranged from .31 and .48, indicating small practice effects.

Table 2.

Test-Retest Reliability of ACPES-TC (N = 70).

	Test	Retest	Difference		Practice effect
	mean (SD)	mean (SD)	mean (SD)	ICC	Cohen's d
1. How ready are you to talk to your decision-maker about the kind of medical care you would want if you were very sick or near the end-of-life?	2.86 (1.27)	3.16 (1.09)	0.30 (0.62)	0.86*	.48
2. How ready are you to talk to your doctor about the kind of medical care you would want if you were very sick or near the end-of-life?	2.87 (1.18)	3.11 (1.15)	0.24 (0.67)	0.84*	.36
3. How ready are you to sign official papers putting your wishes in writing about the kind of medical care you would want if you were very sick or near the end-of-life?	2.79 (1.27)	3.04 (1.21)	0.26 (0.83)	0.78*	.31
4. How ready are you to sign official papers naming a person or group of people to make medical decisions for you?	2.70 (1.26)	2.99 (1.25)	0.29 (0.73)	0.83*	.40
Total score	2.81 (1.20)	3.07 (1.13)	0.27 (0.63)	0.86*	.43

*p < .001.

Abbreviations: ACPES-TC, advance care planning engagement survey, traditional Chinese version; ICC, intraclass correlation coefficient; SD, standard deviation.

As to medical decision-making for oneself, ACPES-TC score was not significantly correlated with responses to the question “Have you ever made medical decisions for other people who were seriously ill or near the end of their life?” while a borderline correlation was noted for the question “Have you ever filled out an ACP?” (r_pb = 0.23, p = .07) (Table 3). By contrast, the ACPES-TC score (r_pb = 0.46, p < .001) was correlated with responses to the question, “How have you preferred to make medical-related decisions?” Specifically, greater ACP preparedness was more likely for participants who made health-related decisions on their own rather than those relying on relatives’ or medical professionals’ decisions. Consequently, the previous question on the ACPES-TC scale can reflect participants’ past preferences, indicating that the ACPES-TC has postdictive validity (criterion-related validity).

Table 3.

Criterion-Related Validity (N = 70).

	n	Point-biserial correlation coefficient	p-value
ACPES-TC	70	1
How have you preferred to make medical-related decisions? By myself with or without consultation of medical professionals, versus By relatives or medical professionals	67	0.46	<.001
Have you ever filled out an advance care planning? Yes versus No	64	0.23	.07
Have you ever made medical decisions for other people who were seriously ill or near the end of their life? Yes versus No	64	0.18	.15

Abbreviation: ACPES-TC, advance care planning engagement survey, traditional Chinese version.

Discussion

This is the first study examining the reliability and validity of the translated version of a validated ACP instrument and evaluating the level of readiness in different age groups in Taiwan. The findings supported that the ACPES-TC is appropriate for the Taiwanese population because of its acceptable reliability and validity. Furthermore, the ACPES-TC can be more broadly used to understand the real-world pattern of ACP behaviors for academic or public dissemination purposes in the era of PRAA in Taiwan.

Reliability

The internal consistency of the four-question version ACPES-TC scale was large (Cronbach's α = .97), which is higher than Sudore et al's original study (Cronbach's α = .84). ¹⁶ The test-retest reliability is good (ICC range: 0.78-0.86), which is slightly higher than the original study by Sudore et al (ICC value = 0.70) ¹⁵ It may be due to the high homogeneity of our sample. This indicates that even though the simplified version has a practice effect, it still has a fairly stable consistency with high test-retest reliability and low random error. The practice effect was considered small (Cohen's d = 0.31-0.48 in each question). Various factors affect the practice effect, including the interval between the timings of the test and retest and the complexity of the concept and content of ACP. In our study, the practice effect may have been influenced by the fact that participants were more likely to remember the test content due to the small number and simple description of the questions in the questionnaire, and participants may have been aware of the context of ACP from experiencing the PRAA public campaign in Taiwan. The practice effects may rise as more people recognize the value of ACP and are open to discuss the EOL issues.

Validity

The discriminant validity of ACPES-TC has to be acceptable. We expected the ACPES-TC score to be highly correlated with the question: “Have you ever filled out the ACP?” However, a low correlation was found. This could be explained by Asia's traditional social norm of decision-making which emphasizes family-led decision-making rather than individualistic autonomy.^27,28 Therefore, even the patient who is ready and eager to participate in an ACP consultation, AD may not be completed afterward, as this needs to involve their family members’ perspectives. By contrast, Sudore et al found that those who had living wills, funeral plans, or advance directives had significantly higher ACP-ES scores than those who did not have these. ¹⁶ It echoes the importance of cultural adaptation when implementing ACP across different cultures. The development of ACP in Taiwan is still at its infant stage, and public awareness of the importance of patient autonomy is lacking. ²⁹ This would influence their willingness to participate in this study.

The question: “How have you made health-related decisions?” is significantly correlated with the ACPES-TC score (r_pb = 0.46, p < .001). Thus, ACP preparation was greater among individuals who preferred to make health-related decisions independently or after consulting medical professionals, rather than having relatives or medical professionals make these decisions for them. This finding is in line with the core value of ACP, which is the autonomous individual planning of one's medical treatment while one is still lucid. In addition, our results demonstrated that the higher the ACPES-TC score, the more preference would be for medical-related decisions to be made by themselves in the past instead of relying on the decisions made by others later, which indicates postdictive validity (a part of criterion-related validity).

ACPES-TC in the Taiwanese Context

ACPES-TC scores were higher among participants over 55 years than those of 20 to 30 years. This is reasonable as the original scale was developed for the elderly, who are more likely to plan for their EOL process as they become older. Financially independent participants were also more willing to participate in ACP discussions in the Taiwanese context (out-of-pocket fee is required to participate in ACP consultation and AD completion regulated by PRAA). Participants on long-term medication were better prepared for the ACP consultation because they were more likely to pay attention to issues surrounding their illness and care in the future. It is also easier for them to imagine the illness and the relevant EOL care decision-making during their disease trajectory. The result of this study is similar to that in Sudore's original study, while the older population had a higher score on ACP engagement. ¹⁵ The ACPES-TC scale can differentiate ACP preparedness and engagement among different populations by age, economic status, and long-term medication use.

Study Limitation

First, this study used purposive sampling to enroll patients from outpatient clinics and healthy participants in the community in Northern Taiwan. This could have limited the generalizability of the result to other populations. The results should be extrapolated to the Taiwanese population with caution. Second, this is a pilot evaluation for the reliability and validity of ACPES-TC with a relatively small sample size. We recommend a more comprehensive survey with more participants, using a stratified random sampling in a future study. Construct validity is difficult to conduct because there is no other ACP-related scale available (criterion-related validity). This study could be the benchmark for further relevant studies in Asia.

Conclusion

The ACPES-TC scale has acceptable internal consistency and test-retest reliability in Taiwanese outpatient settings. The ACPES-TC performed well on discriminant validity by detecting differences in ACP preparation between people of different ages, economic statuses, and long-term medication usage in Taiwan. The ACPES-TC also has fair criterion-related validity. The ACPES-TC scale has been approved for use in research, clinical practice, and further education. Given the preliminary nature of this study, a full-scale psychometric evaluation of ACPES-TC is needed in the future before embedding this scale in routine care.