Estimation of utility values from visual analog scale measures of health in patients undergoing cardiac surgery

Abstract

Introduction

In health economic evaluations, mapping can be used to estimate utility values from other health outcomes in order to calculate quality adjusted life-years. Currently, no methods exist to map visual analog scale (VAS) scores to utility values. This study aimed to develop and propose a statistical algorithm for mapping five dimensions of health, measured on VASs, to utility scores in patients suffering from cardiovascular disease.

Methods

Patients undergoing coronary artery bypass grafting at Aalborg University Hospital in Denmark were asked to score their health using the five VAS items (mobility, self-care, ability to perform usual activities, pain, and presence of anxiety or depression) and the EuroQol 5 Dimensions questionnaire. Regression analysis was used to estimate four mapping models from patients’ age, sex, and the self-reported VAS scores. Prediction errors were compared between mapping models and on subsets of the observed utility scores. Agreement between predicted and observed values was assessed using Bland–Altman plots.

Results

Random effects generalized least squares (GLS) regression yielded the best results when quadratic terms of VAS scores were included. Mapping models fitted using the Tobit model and censored least absolute deviation regression did not appear superior to GLS regression. The mapping models were able to explain approximately 63%–65% of the variation in the observed utility scores. The mean absolute error of predictions increased as the observed utility values decreased.

Conclusion

We concluded that it was possible to predict utility scores from VAS scores of the five dimensions of health used in the EuroQol questionnaires. However, the use of the mapping model may be inappropriate in more severe conditions.

Introduction

In health economic evaluations, the recommended measure of health effects is quality adjusted life-years, which enables the comparison of interventions across disease areas.^1,2 However, clinical trials are frequently initiated without including questionnaires measuring preference-based health-related quality of life. Instead, nonpreference-based measures of health are often utilized and this renders it difficult to estimate health state utility values. One solution that is gaining popularity is prediction of utility values from the nonpreference-based measures of health. This is frequently called mapping and the technique requires an algorithm based on the statistical association between the tools.³ Mapping techniques have been applied in more than a quarter of technology appraisals submitted to the National Institute of Clinical Excellence.⁴ Most mapping techniques predict the utility values from disease-specific questionnaires.³ However, sometimes health effects are merely measured using a visual analog scale (VAS) instead of disease-specific questionnaires. The VAS is one of the most frequently used methods for assessing pain intensity,^5,6 and has also been applied in the assessment of depression, anxiety, and mobility.^7,8 Currently no method has been developed for predicting utility scores from such VAS scores.

This study aimed to develop a mapping model to predict a single utility score from five VAS scores rating patients’ mobility, self-care, ability to perform usual activities, pain, and anxiety and depression. Such a model may be applied to map the partial effect of reducing patients’ pain measured on a VAS to utility scores, under the ceteris paribus assumption, ie, holding all other factors fixed.

We chose to estimate our mapping model by administering our questionnaire to patients undergoing coronary artery bypass grafting (CABG), for two reasons: 1) clinical trials investigating surgical interventions frequently use the VAS when assessing outcomes; and 2) patients undergoing CABG vary widely in the severity of their health conditions pre- and postoperatively. This heterogeneity in their responses makes them ideal respondents.

Methods

Patients

Data were prospectively collected between August 25, 2011 and May 25, 2013, from patients recruited from the cardiothoracic surgical ward at Aalborg University Hospital, Denmark. Eligible patients were more than 18 years of age, were able to read Danish, and had coronary artery disease requiring elective CABG. Approximately 250 CABG procedures are performed at Aalborg University Hospital every year. We divided the dataset in two, by random sampling, such that 60% of the patients were included in the estimation sample and the remaining 40% of the patients were included in the validation sample.

Questionnaires

For the purpose of developing the mapping-model, all patients were asked to complete the three level version of the EuroQol 5 Dimensions questionnaire (EQ-5D) and to rate their perceived health today on five VAS items (patients’ self-reported mobility, self-care, ability to perform usual activities, pain, and presence of anxiety or depression). Each VAS item was given an introductory title and a short statement representing “no problems” at 0 mm and a short statement representing “worst imaginable problems” at 100 mm.

Patients were asked to fill out the EQ-5D as many as three times. The first time was prior to their admission, the second time was 5 days postsurgery, and the third time was upon the arrival for their follow-up visit at the outpatient clinic. Only patients seen at the outpatient clinic at Aalborg University Hospital were asked to fill out the third questionnaire. In the analyses, each questionnaire was treated as an independent measurement in order to obtain diversity in the severity of health states.

Analyses

For all mapping models, the dependent variable was the utility score calculated using the Danish time trade-off values.⁹ These values for the EQ-5D range from 1 to −0.624 where 1 indicates perfect health, 0 indicates death, and a value below zero is a health state perceived to be worse than death. We fitted two mapping-models using the 60% estimation sample. The first mapping model was fitted using age, sex, and the five VAS scores as explanatory variables. In the second mapping model, the squared terms of the five VAS scores were included. The squared terms were added in the second mapping model because the relationship between explanatory dimensions and utility scores may not be linear in nature.^10,11 Although dimensions may not be additive,^11,12 interaction terms were not considered as they may restrict the use of the mapping models to situations where all five VAS scores are measured. Excluding the interaction terms allows the mapping-models to be used in situations where only one or two of the VAS scores are measured. Both mapping models were initially fitted using random intercepts generalized least squares (GLS) models. Least squared estimation was chosen because of its straightforward interpretation and frequent use in mapping models.³ The random effects part was chosen to handle the fact that some patients had multiple observations. However, if least squared estimation is used in the presence of large proportions of subjects scoring utility values of 1, the bounded nature of the utility value may result in implausible predictions outside of the existing range of the scale.^3,13 In such situations, researchers have proposed using the Tobit model or censored least absolute deviation (CLAD) regression methods.^3,14 If the proportion of patients at the upper ceiling is small the marginal coefficients from the random effects GLS should suffice.¹⁵ Nevertheless, to accommodate the possibility that the observed ceiling effect might reduce the performance of our random effects GLS models, random effects Tobit regression and CLAD regressions were also fitted. The random effects Tobit regression handles the bounded nature of the utility scores and may therefore be an appropriate alternative to least squares estimation.^11,14,16 However, a random effects Tobit model will produce biased results when faced with heteroscedasticity or nonnormality.^11,16 CLAD regression will produce consistent results even if faced with heteroscedasticity or nonnormality.¹⁷ As such, the CLAD regression may seem the optimal choice. However, the downside to a CLAD mapping model is that it is a median model.¹³ Typically, economic evaluations use health valuations based on mean models. To assess the performance of the mapping models, we calculated the root mean squared error (RMSE) and mean absolute error (MAE) to compare the models.^3,13 Errors were subsequently reported for the following subsets of observed utility values: utility <0; 0≤ utility <0.25; 0.25≤ utility <0.5; 0.5≤ utility <0.75; 0.75≤ utility ≤1.

We predicted the utility values in the validation sample using the second (full) random effects GLS mapping model to assess if the estimates were reliable. For both the estimation sample and the validation sample, Bland–Altman plots were used to assess agreement between observed and predicted values. All statistical analyses were performed in Stata version 12.1 (StataCorp LP, College Station, TX, USA). The CLAD regression was performed using user-written programs for Stata.¹⁸ The questionnaires were entered once in EpiData version 3.1 (freeware product by EpiData Association, Odense, Denmark).

Results

A total of 238 patients were invited to enter the study. Of these, 24 patients declined the invitation, 16 did not meet inclusion criteria, and an additional 8 were excluded due to non-response (Figure 1). Therefore, 382 questionnaires from 190 patients were analyzed in the study. Because only patients seen at the outpatient clinic at Aalborg University Hospital filled out the third questionnaire, only 55 patients answered all three questionnaires. The baseline and intraoperative characteristics of the patients are shown in Table 1. No differences in patient characteristics were observed between the estimation dataset and validation dataset.

Figure 1.

Flowchart describing the inclusion of patients.

Abbreviations: n, number; CABG, coronary artery bypass grafting; EQ-5D, EuroQol 5 Dimensions questionnaire; VAS, visual analog scale.

Table 1.

Patients’ baseline and intraoperative characteristics

Variable	Estimation sample n=114	Validation sample n=76	P-value
Age (years), mean ± SD	68.4±9.2	65.9±9.6	0.07
Female, n (%)	25 (21.9)	13 (17.1)	0.42
Redo cardiac surgery, n (%)	2 (1.8)	0 (0)	0.52
Concomitant surgery, n (%)	20 (17.5)	18 (23.7)	0.30

Abbreviations: n, number; SD, standard deviation.

Figure 2 shows the five VAS items and the phrases used at both ends of the scale. The mean utility and VAS scores are reported in Table 2. The number of patients rating themselves in full health was somewhat limited, with observed utility scores ranging from −0.495 to 1 (Figure 3).

Figure 2.

Questionnaire used to assess health on visual analog scales.

Table 2.

Mean utility and VAS scores for all observations

Variable	Estimation sample (233 observations)	Validation sample (149 observations)	P-value
EQ-5D utility score, mean ± SD	0.72±0.20	0.73±0.16	0.39
VAS mobility (mm), mean ± SD	26.8±22.4	26.6±20.3	0.91
VAS self-care (mm), mean ± SD	15.5±21.9	15.2±21.5	0.89
VAS usual activity (mm), mean ± SD	44.4±32.7	41.8±29.9	0.43
VAS pain (mm), mean ± SD	22.8±20.6	23.8±22.1	0.65
VAS anxiety and depression (mm), mean ± SD	16.3±20.6	18.2±21.2	0.39
Disease stage proxy			0.87
Preoperative, n (%)	97 (41.6)	66 (44.3)
Postoperative day 5, n (%)	78 (33.5)	47 (31.5)
Postoperative day 40, n (%)	58 (24.9)	36 (24.2)

Abbreviations: EQ-5D, EuroQol 5 Dimensions questionnaire; SD, standard deviation; VAS, visual analog scale; n, number.

Figure 3.

Observed EQ-5D utility scores in the complete dataset.

Abbreviaton: EQ-5D, EuroQol 5 Dimensions questionnaire.

The first mapping model included age, sex, and the five VAS scores measuring health (GLS 1 in Table 3). Although the functional form of the explanatory variables was restricted to the additive level, the model explained approximately 63% of the variation in the EQ-5D utility score. In the second random effects GLS mapping model (GLS 2), the quadratic terms of the five VAS scores measuring health were added. Comparing the MAE and the RMSE between the two random effects GLS models, we found that the GLS 2 mapping model yielded the best fit to the estimation sample. Approximately 65% of the variation in EQ-5D utility scores was explained by the GLS 2 mapping model. The variables used in the GLS 2 model were also used to fit mapping models using random effects Tobit and CLAD regressions. The random effects Tobit mapping model was inferior to the GLS 2 mapping model, as it yielded higher MAE and RMSE values. The CLAD mapping model had a slightly lower MAE and a higher RMSE than the GLS 2 mapping-model had. In general, the mapping models performed better at higher observed EQ-5D utility scores (Table 4). In Figure 4 it can be seen that the reduction in performance in more severe health states is due to an over prediction of the EQ-5D utility score.

Table 3.

Results of the estimated mapping-models

Parameter	Random effects
	GLS 1 β (SE)	GLS 2 β (SE)	Tobit β (SE)	CLAD β (SE)
Age	0.000259	0.000255	0.000599	0.00198‡
	(0.00101)	(0.00102)	(0.00106)	(0.000206)
Female	−0.00263	−0.00351	−0.00898	−0.0165‡
	(0.0218)	(0.0220)	(0.0227)	(0.00464)
VAS scores for
Mobility	−0.00177†	0.0000882	−0.000198	−0.00227‡
	(0.000582)	(0.00128)	(0.00140)	(0.000261)
Self-care	−0.000934	−0.00204	−0.00208	−0.00405‡
	(0.000548)	(0.00135)	(0.00146)	(0.000309)
Usual activity	−0.00146‡	−0.00146	−0.00204	0.000328
	(0.000323)	(0.00111)	(0.00121)	(0.000232)
Pain	−0.00158†	−0.00120	−0.00213	−0.000580*
	(0.000543)	(0.00136)	(0.00146)	(0.000274)
Anxiety and depression	−0.00279‡	0.000335	0.000236	−0.00280‡
	(0.000468)	(0.00118)	(0.00128)	(0.000282)
Squared VAS scores for
Mobility		−0.0000255	−0.0000235	0.0000113†
		(0.0000163)	(0.0000177)	(0.0000034)
Self-care		0.0000165	0.0000166	−0.0000429‡
		(0.0000171)	(0.0000185)	(0.0000030)
Usual activity		−0.0000008	0.0000037	−0.0000136‡
		(0.0000105)	(0.0000115)	(0.0000022)
Pain		−0.0000069	0.0000049	0.0000105*
		(0.0000205)	(0.0000222)	(0.0000043)
Anxiety and depression		−0.0000445†	−0.0000442*	0.0000105†
		(0.0000161)	(0.0000175)	(0.0000038)
Constant	0.909‡	0.877‡	0.892‡	0.738‡
	(0.0691)	(0.0721)	(0.0753)	(0.0140)
Overall	0.632	0.652
R-squared
MAE	0.0857	0.0838	0.0871	0.0818
RMSE	0.121	0.119	0.120	0.141

Notes:

^*P<0.05

^†P<0.01

^‡P<0.001.

Abbreviations: GLS, generalized least squares; CLAD, censored least absolute deviation; SE, standard error; VAS, visual analog scale; MAE, mean absolute error; RMSE, root mean squared error.

Table 4.

Mean absolute error of mapping-models at subsets of observed EQ-5D utility scores

Utility subset	GLS 1	GLS 2	Tobit	CLAD
Utility <0	0.601	0.526	0.527	0.796
0≥ utility <0.25	0.216	0.179	0.175	0.376
0.25≥ utility <0.5	0.123	0.122	0.124	0.106
0.5≥ utility <0.75	0.074	0.078	0.082	0.054
0.75≥ utility ≤1	0.074	0.071	0.074	0.072

Abbreviations: EQ-5D, EuroQol 5 Dimensions questionnaire; GLS, generalized least squares; CLAD, censored least absolute deviation.

Figure 4.

Mean observed and predicted EQ-5D utility scores in the validation sample.

Notes: The graph shows the agreement between the observed EQ-5D utility score and the mean of the predicted score. The observed health states are ordered on the x-axis according to their severity as valued by the Danish time trade-off tool.

Abbreviation: EQ-5D, EuroQol 5 Dimensions questionnaire.

Bland–Altman plots of agreement between observed and predicted values of EQ-5D utility scores for the GLS 2 model are shown in Figure 5. Figure 5A shows agreement in the estimation sample and Figure 5B shows agreement in the validation sample. In both samples, the mapping-model overestimated the EQ-5D utility scores for patients with low observed EQ-5D utility scores. The Bland–Altman 95% limits of agreement from the validation sample (−0.212, 0.240) were similar to those from the estimation sample (−0.233, 0.232). A slight bias of 0.014 was observed in the validation sample.

Figure 5.

Bland–Altman plots of agreement between observed and predicted EQ-5D utility scores.

Notes: (A) Agreement in the estimation sample. (B) Agreement in the validation sample. The x-axis depicts the mean of the observed value and the predicted value, and the y-axis shows the difference (observed minus predicted). The lines show the mean difference, ie, the estimated bias, and the 95% limits of agreement (±1.96 SD of the mean difference).

Abbreviations: EQ-5D, EuroQol 5 Dimensions questionnaire; SD, standard deviation.

Discussion

The GLS 2 mapping-model showed promising ability to predict mean utility scores. Our findings indicate that VAS scores for mobility, self-care, usual activity, pain, and mood could be used for obtaining estimates of utility among patients with cardiovascular disease. As much as 65% of the variability in utility scores could be explained, which is quite high compared with existing mapping models.³ Goldsmith et al predicted EQ-5D utility scores in a group of patients with coronary artery disease using demographic and clinical outcome variables and explained 48%–49% of the variability in utility scores and found RMSE and MAE to be 0.170 and 0.122, respectively.¹⁹ Longworth et al developed a model to map EQ-5D utility scores from clinical indicators for patients with stable angina and were able to explain 37% of the variation and showed an RMSE of 0.4764 (RMSE = √Mean Squared Error =√0.227).²⁰ Our mapping-model may therefore be viewed as a reasonable method for mapping VAS scores to EQ-5D utility scores.

To illustrate the use of the GLS 2 mapping-model, imagine a comparison of a new, less-invasive surgical method with the conventional open surgical method in a health economic evaluation. The new surgical method reduced pain measured on a VAS scale by 10 mm, from 40 mm to 30 mm. For simplicity, assume that all other health outcomes are unchanged. The GLS 2 mapping-model is then applied to map the partial effect of reducing patients’ pain, under the ceteris paribus assumption, ie, holding all other factors fixed:

$$\text{Utility gain}=({\beta }_{\text{pain}}*{\text{VAS}}_{\text{new}}+{\beta }_{\text{pain squared}}*{\text{VAS}}_{\text{new}}{}^2)- ({\beta }_{\text{pain}}*{\text{VAS}}_{\text{conventional}}+ {\beta }_{\text{pain squared}}*{\text{VAS}}_{\text{conventional}}{}^2)$$

$$\text{Utility gain}=(-0.00120*30-0.0000069*{30}^2)-(-0.00120*40-0.0000069*{40}^2)$$

$$\text{Utility gain}=(-0.0422)-(-0.0590)=0.0168$$

In this example, it was estimated that a 10 mm reduction in pain measured on a VAS, from an average of 40 mm to an average of 30 mm, would increase patients’ utility with 0.0168. Because the GLS 2 mapping-model contains squared terms, the utility gain from a 10 mm reduction on a VAS of pain depends on the severity of the pain, ie, the utility gain from a reduction in pain from 40 mm to 30 mm is not the same as from 60 mm to 50 mm. If the GLS 1 mapping model had been used, a 10 mm reduction on a VAS of pain would yield the same utility-gain regardless of the initial level of pain. The uncertainty in the estimated utility gain could be modeled using the standard errors of the coefficients from the mapping model in a probabilistic sensitivity analysis.

Strengths and limitations

Because we treated all measurements as independent observations, the observed utility scores covered a wide range, especially because patients temporarily felt worse 5 days after CABG. Naturally, this approach increases the sample size somewhat artificially, and the standard errors presented in Table 3 should therefore only be used to estimate approximate confidence intervals. However, by using each measurement as independent we ensured that the mapping model is valid for all stages of the illness. The wide range of VAS scores and utility scores also enabled us to assess the GLS 2 mapping model’s predictive ability at a subset of observed utility scores. This analysis showed that the reliability of our predictions declined as observed values decreased. This is a frequent limitation of mapping-models;^3,11,19 however, it implies that the mapping models presented in this study may have a limited ability to predict utility scores for more severe conditions. Users of the mapping model should therefore be cautious when applying it in populations with large numbers of patients in poorer health. In such situations, alternative methods should be considered. The poor performance of the mapping model in patients with a more severe health condition is likely caused by a combination of the following: 1) the low number of patients with a low EQ-5D utility score in our dataset; 2) our exclusion of interaction terms and other covariates to ensure the mapping model could be used if researchers only had one of the five VAS measures; or 3) a larger intersubject variation in the use of the VAS scales for more severe conditions. The latter is supported by the fact that severe pain measured on a 11-point numeric rating scale could be from seven and upwards for some patients, while severe pain measured on a 100 mm VAS could be from 35 mm and upwards for others.⁶ In another study, the over-prediction of EQ-5D utility scores was shown to be worsened by the N3-term, which is added if severe problems were reported in at least one dimension.¹¹ However, the Danish time trade-off values for EQ-5D do not contain such a jump and therefore the N3 term cannot be contributing to the over prediction among the severely ill in our analyses.⁹

The mapping models presented in this study were fitted using patients undergoing cardiac surgery. Cost-effectiveness analyses within this disease area have previously mapped health outcomes measured on VAS scales to utility scores.^21,22 However, such attempts have also been made in other disease areas.²³ Therefore, future work might include validating the mapping model for different patient groups and assessing the performance of an independent sample.

Conclusion

We conclude that it is possible to predict utility scores from VAS scores of the five dimensions of health used in the EuroQol questionnaires. However, the predictive power decreased as observed utility scores declined. The use of the mapping model may therefore be inappropriate in more severe conditions.