Psychometric Properties of a Single Item Visual Analog Scale Measuring Goals of Care in Patients with Advanced Cancer

Abstract

Purpose:

The purpose of this study was to examine the psychometric properties of a single item visual analog scale (VAS) to measure goals of care in patients with advanced cancer.

Methods:

Data were obtained from 378 patients with diagnoses of advanced lung, gastrointestinal, or pancreatic cancer. Goal of care was measured at baseline and every 3 months until patient death or completion of the 15-month study period. A single-item VAS ranging from 0 (quality of life is all that matters) to 100 (length of life is all that matters) was used to measure patients’ goals of care for all study subjects; a sub-sample of subjects also completed the Quality of Life-Length of Life scale which asked patients to select categories of preferences. Test-retest reliability (intra-class correlation) and construct validity (known groups, convergent, divergent) were evaluated.

Results:

At 9 and 12 months, the test-retest reliability for patients with stable symptoms (n=107) was established with the ICC_(1,3) =0.81, p<.001. Known groups (r=0.99, p<.001), convergent (r=0.78, p<.001) and divergent (r=.06, p=0.24) validity all demonstrated evidence of good construct validity.

Conclusions:

Preliminary psychometric testing for a single-item VAS that measures goals of care in a sample of patients with advanced cancer met standard requirements for reliability and validity. While further testing with a larger sample size is recommended, the tool’s use in the clinical area to assess cancer patients’ goals of care is appropriate. Such a tool could facilitate goals of care discussions in the clinical area.

Introduction

Assessing patients’ goals of care has become an important aspect of shared decision-making and quality care for patients with advanced cancer. Knowledge of patients’ goals of care plays a vital role in facilitating good patient-physician communication and has been linked with better patient outcomes, and improved patient and family satisfaction with care [1–5]. In particular, patients with a diagnosis of advanced cancer are often faced with agonizing and challenging choices throughout the trajectory of their disease [6–9]. Studies have shown that these patients want their clinicians to initiate discussions about goals of care, yet having these conversations is difficult for many clinicians [10]. In response, The American Society of Clinical Oncology [10] recommended that when discussing treatment plans, clinicians should start by first identifying the patient’s goals prior to recommending any treatments. While this recommendation appears straightforward, the lack of time in the clinical setting along with lack of brief tools to measure goals of care serve as barriers [10].

In an attempt to address these barriers, researchers and clinicians have sought to develop tools to measure goals of care. To date, few have been developed and tested and of those that have been tested, they have involved either multiple questions or multiple response selections [11, 12]. Besides the time it takes to complete a multi-question tool, there are other challenges to using more complex tool in the clinical setting. Level of education of the reader, understanding various response options, or understanding Likert response items all affects the ability to complete a more complex questionnaire. As a result, the use of visual analog scales in the clinical setting has been seen as an attractive alternative to more complicated strategies [13–14].

The reliability and validity of single-item visual analog scales (VAS) are well established in the literature and have been used successfully in the clinical arena to track pain progression, asthma, quality of life, mood among other key symptoms and concepts [13, 15–16]. In a VAS, subjects are asked to mark a line, usually using a 100-point scale, that characterizes how they feel about a specific concept [17, 18]. Given the advantages of using VAS’ in a clinical setting, and the need to measure goals of care, the purpose of this study was to examine the psychometric properties of a single VAS to measure goals of care in patients with advanced cancer.

Patients and Methods

Patients

Data employed for psychometric testing came from a longitudinal, descriptive correlational design study that was conducted from January 2015 to October 2018 in the outpatient clinics at Seidman Comprehensive Cancer Center at University Hospitals Cleveland Medical Center in Cleveland, Ohio. Convenience sampling was used to recruit 378 patients who were in active treatment or being seen regularly (at least monthly by their oncologist). Inclusion criteria were: 18 years or older, diagnosis of Stage IV gastrointestinal or lung cancers or any stage pancreatic cancers, and cognitively intact. Written informed consent was obtained from all individual participants included in the study and approval was obtained from the hospital Institutional Review Board prior to data collection.

Measurements

Goal of care was measured with a single item VAS developed by three research-clinicians with expertise in the areas of oncology and goal of care research. A horizontal line ranging from 0 (quality of life is all that matters) to 100 (length of life is all that matters) was used to measure responses to the question, “Regarding your care, what is most important to you right now?” (Figure 1). The 0–100 point scale was used given its common use in other single item VAS tools used in clinical settings (e.g. National Comprehensive Cancer Network [NCCN] distress thermometer) [18]. The conceptual definition of goal of care used for this study was a patient’s desired health expectation from an individual seeking medical care [19] and this definition guided the selection of the question used to assess goals of care [10].

Fig 1.

Single item visual analog scale measuring goal of care

Quality of Life (QOL) and Length of Life (LOL) Preferences (QOL-LOL) was measured with a single item scale designed to determine the relative value that an individual assigns to quality of life and quantity of life [20]. This instrument [11, 21] asks cancer patients, “Which item most accurately reflects your feelings about what the focus of your care should be today?” The patient is to select from among 4 choices about whether QOL or LOL was more important (QOL is all that matters-scored as 1, QOL is more important but LOL matters-scored as 2, LOL is more important but QOL matters-scored as 3, LOL is all that matters-scored as 4). Scores ranged from 1–4 with higher scores indicating a greater preference for length of life. Reliability of the QOL-LOL scale and a single item question assessing desire for QOL versus LOL was reported as was divergent validity [11]. In the present study, reliability of the tool was .77.

Values relevant to end of life goals of care was evaluated using the End-of-Life (EOL) Values scale [22–23]. This is an 8-item validated scale that evaluates values relevant to EOL care, treatments, or goals of care. The scale for each item ranges from 0 (not at all important) to 5 (extremely important) [22]. Psychometric properties of this scale have been established [22–23] with three underlying factors identified (pain management, longevity, and avoidance of burden to others). Cronbach α for the three factors were 0.90 for pain management, 0.87 for burden, and 0.76 for longevity. The predictive validity of each factor for life-prolonging treatment preferences was established in prior studies [22–23]. For the purposes of this paper, the longevity factor assessing the patient’s wish to live as long as possible was used. In the present study, the longevity factor had a Cronbach’s α of 0.89.

The Edmonton Symptom Assessment System (ESAS) was used to evaluate severity of symptoms of patients [24]. The ESAS is a tool used at each data collection point to assess nine common symptoms experienced by cancer patients (pain, tiredness, nausea, depression, anxiety, drowsiness, appetite, well-being, and shortness of breath). The severity at the time of assessment of each symptom is rated from 0 to 10 on a numerical scale with 0 meaning that the symptom is absent and 10 meaning that it is the worst possible severity. Total scores range from 0–90 with higher scores indicating greater severity of symptoms. Reliability has been reported (Test-retest reliability=0.80, Cronbach’s α=0.75) as has concurrent validity (Kappa=0.66) [25]. ESAS change scores of <3 are considered indicative of non-meaningful clinical differences [24]. In the present study, a Cronbach’s α of 0.85 was reported.

Procedures

Data collection was conducted using an iPad. All patients were provided with an iPad loaded with the study tools at study enrollment (baseline), 3, 6, 9, 12 and 15 months afterwards. Patients completed the study tools while they waited for their oncologist visit in the outpatient clinic and data were automatically downloaded to a Health Insurance Portability and Accountability Act (HIPAA compliant) SPSS file. Study tools included: (1) a single-item VAS to assess their current goals of care, (2) an 8-item EOL values scale [22–23], (3) the ESAS to assess their severity of symptoms [24], and (4) for a subset of patients, a single-item QOL-LOL Preferences scale [20]. In order to minimize any effect of testing, the order in which the tools were administered was randomized. Three types of construct validity were evaluated [26]. First, in assessing convergent validity, 23 stable patients were selected from the larger sample to complete the QOL-LOL Preferences scale at the end of the 15-month study period. In assessing known-groups and divergent validity—both types of construct validity--baseline data were used.

In order to assess test-retest reliability, goal of care data at 9 and 12 months post-enrollment from 107 stable patients (who survived the 15 month study period) without evidence of clinically meaningful symptom burden during that time period (ESAS score difference between 9 and 12 months <3) were used. The 9 and 12 month time points were used because this was the time period when a majority of patients were expected to have completed the intense aspect of their treatment, have fewer treatment related symptoms (which could influence goals of care), and have less variability in symptoms over time than at earlier time points [27].

Statistics

All data were checked and analyzed using the Statistical Package for the Social Sciences (SPSS Version 25.0). The VAS, QOL-LOL Preferences scale, ESAS, and EOL values scales all followed a normal distribution as determined by histograms, Q-Q plots, and 95% confidence intervals around the mean skew and kurtosis values. All assumptions of test statistics were evaluated and met prior to analyses.

Construct Validity: Known-Groups

First, we examined construct validity using the known-groups approach [27–29]. Patients were classified into two groups known to differ regarding their baseline EOL values to live as long as possible (the focal construct—goals of care). Based upon their baseline ranking of the importance of living as long as possible on the EOL Values scale we constructed two groups: those with EOL values scores indicating that living as long as possible was “extremely” important (score of 5) and those with EOL values scores indicating that living as long as possible was “not important at all” (score of 0). We then compared the mean baseline VAS scores for those two groups. These mean differences needed to be statistically significant in the expected direction if the instrument was to be considered valid [26,29]. We hypothesized that the mean VAS scores for the two groups would differ significantly with those in the high EOL values group having significantly higher VAS scores than those in the low EOL values group.

Construct Validity: Convergent

In assessing convergent validity, we tested the correlation between the focal measure (goal of care-VAS) and a measure of a construct with which conceptual convergence is expected (QOL-LOL Preference Scale) [27, 30–31]. Pearson’s correlations were calculated based upon baseline data for all measures for a subset of 23 stable patients selected from the larger study sample. These 23 subjects completed their 15 month VAS and completed the additional QOL-LOL tool. Correlations of <0.20 were considered low, 0.20 – 0.50 were considered moderate, and >0.05 were considered large [32]. It was hypothesized that there would be a significant direct correlation between VAS and QOL-LOL Preference Scale scores since higher scores on the VAS and the QOL-LOL Preference Scale indicated a greater preference for length of life over quality of life. Not only would the correlation need to be statistically significant, but in order to demonstrate convergent validity, the correlations would need to be moderate to high correlation (≥0.45) [29, 33].

Construct Validity: Divergent

In assessing divergent validity, we tested the relationships between the focal measure (goal of care-VAS) and a similar (but distinct) measure in order to ensure that the two are not really measuring the same construct [26, 28–29]. Physical symptoms have not been found to relate to goals of care [19]. For this evaluation, we examined the correlations between VAS and the ESAS physical score. It was hypothesized that there would be small and statistically non-significant correlations [29, 34].

Reliability: Test-Retest

Given the fact that the VAS was a single item measure, reliability testing such as Cronbach’s α were not appropriate. However, for single item VAS’, test-retest reliability is employed to examine reliability of the measure [18, 26]. In order to evaluate reliability, the VAS needed to yield reproducible results when used repeatedly on patients who were not experiencing changes in condition, symptoms or care that could change their goals of care [18]. Test-retest reliability of the VAS was evaluated using VAS goal of care data from stable subjects at 9 and 12 months.

Given limitations associated with inter-class correlations (e.g., Pearson’s product-moment correlation) in evaluating reliability of an instrument, an intra-class correlation (ICC) approach will be used to examine the test-retest reliability of the VAS [35]. This approach is considered superior to Pearson’s correlation as a metric for tool reliability when subjects are measured under similar conditions and when the instrument is intended to be administered multiple times, as is the situation with the VAS [36–38]. Since the instrument is intended to be administered multiple times, the average measure (not single measure) ICC will be reported and a mixed model (timepoints fixed, subjects random) will be used [36]. An ICC >0.90 is considered to demonstrate excellent reliability; 0.75 to 0.9, good reliability; 0.5 to 0.75, moderate reliability; and <0.5, poor reliability [37–38]. It was hypothesized that there would be a strong and direct ICC coefficient demonstrating good-excellent reliability (ICC ≥0.75) between VAS scores at 9 and 12 months.

Results

There were 378 patients enrolled in the study who provided data at baseline. Patient mortality (n=206) and drop-out or no longer eligible (n=52) yielded a sample of 179 patients who provided data at 9 months, 138 who provided data at 12 months, and 107 who survived the entire 15 month study period. As seen in Table 1, the patients were, on average, middle aged, female, Caucasian, and married. All had metastatic disease with almost half having GI cancer and only 15% having had a history of cancer prior to this metastatic diagnosis.

Table 1.

Patient Characteristics (N=378)

	Mean (SD)	N	%
Age (years)	63.4 (11.01)
Range: 19–88
Edmonton Symptom Scorea	24.5 (18.2)
Range: 0–82
Gender: Female		194	51.3
Race: Caucasian		280	74.1
Married: Yes		232	61.4
Income
< 20,000		78	23.1
21,000 – 49,999		104	30.8
>50,000		156	46.2
Employment Status
Employed		104	28.5
Not Employed		88	24.1
Retired		173	47.4
Cancer type
GI		177	46.8
Lung		131	34.7
Pancreas		70	18.5
Cancer stage
III		40	10.6
IV		337	89.4
Progression?: No		277	73.5
Prior cancer?: No		320	84.9

^aEdmonton Symptom Assessment System scores range from 0–90 with higher values representing greater severity of symptoms.

Construct Validity: Known-Groups

Using an independent samples t-test, mean VAS scores between those with high EOL values scores to live as long as possible (scored as 5) were compared to those with low EOL values scores to live as long as possible (scored as 0). There was a statistically significant difference in VAS group means between those with EOL values scores of 5 having a mean VAS score of 55.7 (29.7) compared to a mean VAS score of 6.6 (4.1) for those with EOL values scores of 0. The assumption of homogeneity of variance was violated (Levene test, p=.003) and the corrected analysis with degrees of freedom 53.8 was used for interpretation, t(53.8)=19.5, p<.001, r=0.99.

Construct Validity: Convergent

Pearson correlation was used to assess the relationship between the VAS and QOL-LOL Preference Scale score for 23 stable subjects. A statistically significant direct relationship was found, r(23)=0.78, p<.001.

Construct Validity: Divergent

A Pearson correlation was used to assess the relationship between the baseline VAS and physical subscale score of the ESAS. The relationship was small and not statistically significant, r(378)=0.06, p=.24. The nature of the relationship was unchanged 6 months later, during the midst of treatment when physical symptoms typically are increased, r(125)=0.06, p=.53.

Reliability: Test-Retest

Test-retest reliability was conducted on 107 stable patients. An ICC (Two-way mixed effects consistency type model) was conducted to assess the reliability of the VAS. The average measure ICC coefficient was statistically significant, ICC_(3,1) = 0.81 (CI_.95: 0.71, 0.87) p<.001.

Discussion

We have provided psychometric testing results for a single-item VAS that measures goals of care in a sample of patients with advanced cancer. In order for a tool to demonstrate validity, it must first demonstrate reliability [26, 33, 39]. Test-retest reliability was demonstrated by an ICC that exceeded the benchmark for good reliability (>0.75) [37]. This finding supported our hypothesis that there would be a strong direct relationship between VAS scores at 9 and 12 months.

The VAS was also found to be a valid measure of goals of care. As hypothesized, the VAS showed construct (known-groups) validity by demonstrating an ability to discriminate between those with high EOL values to live as long as possible and those with low EOL values to live as long as possible on their baseline VAS score. Not only were the results statistically significant, but they were in the expected direction and clinically significant with a large effect size. In addition, the 95% CI around the ICC value showed that even at its weakest, the ICC fell within the range of “good” reliability. Non-significant differences between VAS scores for these two groups would have demonstrated that the VAS was not valid in its ability to discriminate between groups known to differ on a similar construct [26].

As hypothesized, construct (convergent) validity was also demonstrated by finding a strong direct relationship between the VAS and the QOL-LOL Preference scale—a measure of goal preferences with which conceptual convergence was expected. Finally, as hypothesized, construct (divergent) validity was established by finding non-significant relationships between VAS and ESAS Physical sub-scale scores---a subscale that is not a measure of goals of care. Not only were the baseline correlations small and non-significant, but this relationship remained consistent at another point in time when the ESAS Physical sub-scale scores were expected to have changed from baseline. This demonstrates that the VAS does not measure physical symptom burden.

The present study showed that the single item VAS is a reliable and valid measure of goals of care for patients with advanced cancer. There are, however, some areas of limitation. While test-retest ICC demonstrated good reliability, the time period between measures was relatively long (3 months). As a result, it is possible that factors (worsening of diagnosis etc.) could have had an impact on patients’ goals of care over this period. While a time interval for test-retest as long as 6 month has been reported, shorter periods are considered to have higher reliability scores [28, 29, 39]. Additional research should be conducted using shorter time periods between testing (e.g. 1 week and 1 month) in order to more fully evaluate the measure’s reliability [29, 40].

Second, testing should be conducted to evaluate the responsiveness of the VAS to changes in treatment outcomes (e.g. progression of disease) in order to determine whether the VAS performs similarly in non-stable and deteriorating medical conditions. Such information would provide additional evidence regarding the sensitivity of the measure in specific circumstances where changes in goals of care could be used to initiate discussions regarding goals of care (or treatment preferences).

Third, although the VAS demonstrated a strong relationship with a tool that measured the same concept (QOL-LOL Preference Scale), we did not perform cognitive debriefing of the concept of “goal of care” nor of the anchors for the VAS. Thus, it is possible that patients had different understandings of what the term “goal of care” meant or what the anchor terms meant. This information could be used to clarify the wording of the single item VAS if found that there was misperceptions among patients regarding what the item was assessing and what the anchor terms meant.

Conclusions

In summary, preliminary testing shows that this single item VAS is a reliable and valid measure of the concept “goal of care” in patients with advanced cancer and its use in the clinical setting is appropriate. Given the need for early and ongoing discussions of goals of care—especially for patients with advanced cancer—a simple VAS that is reliable and valid could facilitate such discussions. By evaluating a patient’s goal of care over time, the healthcare team can provide more tailored discussions with patients depending upon the patient’s goals. For example, if a patient for whom all treatment has not yielded successful results has goals of care that are still focused towards “survival”, the discussion regarding comfort care will be presented much differently than if the patient had a goal of care that was focused upon “comfort”. Just as patient distress (using the NCCN distress thermometer) is considered standard practice in most oncology clinics, so too should be the evaluation of a patient’s goals of care. A single item VAS, found to meet reliability and validity standards, could facilitate such a change in practice.