Calculating Utilities From the Spine Oncology Study Group Outcomes Questionnaire

Supplemental Digital Content is available in the text

Quality-adjusted-life-year weights (utilities) for metastatic disease were measured for the Spine Oncology Study Group Outcomes Questionnaire. Analysis revealed that participants did not regard all questionnaire items as equally important. We provide a simple technique for converting the SOSGOQ2.0 to utilities which will facilitate economic analysis and patient counseling.

Study Design.

General population utility valuation study.

Objective.

The aim of this study was to develop a technique for calculating utilities from the Spine Oncology Study Group Outcomes Questionnaire v2.0 (SOSGOQ2.0).

Summary of Background Data.

The ability to calculate quality-adjusted life-years (QALYs) for metastatic spine disease would enhance treatment decision-making and facilitate economic analysis. QALYs are calculated using utilities.

Methods.

Using a hybrid concept-retention and factorial analysis shortening approach, we first shortened the SOSGOQ2.0 to eight items (SOSGOQ-8D). This was done to lessen the cognitive burden of the utility valuation exercise. A general population sample of 2730 adults was then asked to evaluate 12 choice sets based on SOSGOQ-8D health states in a Discrete Choice Experiment. A utility scoring rubric was then developed using a mixed multinomial-logit regression model.

Results.

We were able to reduce the SOSGOQ2.0 to an SOSGOQ-8D with a mean error of 0.003 and mean absolute error of 3.078 compared to the full questionnaire. The regression model demonstrated good predictive performance and was used to develop a utility scoring rubric. Regression results revealed that participants did not regard all SOSGOQ-8D items as equally important.

Conclusion.

We provide a simple technique for converting the SOSGOQ2.0 to utilities. The ability to evaluate QALYs in metastatic spine disease will facilitate economic analysis and patient counseling. We also quantify the importance of individual SOSGOQ-8D items. Clinicians should heed these findings and offer treatments that maximize function in the most important items.

Level of Evidence: 3

Treatment decisions for spine metastases, the most common site of skeletal metastases, are challenging because multiple patient and treatment-related factors including performance status, survival and risk of adverse events, must be considered.¹ Furthermore, with limited life expectancy and scarce resources, the value of care in these patients is hotly debated.^2,3 Due to this controversy the economic value of metastatic spine care is desperately needed.^4,5 The ability to calculate quality-adjusted life-years (QALYs) for metastatic spine disease would facilitate economic analysis and thus enhance treatment decision-making and resource utilization.

QALYs are required in economic analysis because economic decisions are based on the incremental cost-effectiveness ratio which is simply the cost per QALY gained.⁶ QALY analysis could also help patients and clinicians jointly assess the trade-offs between survival, health-related quality-of-life (HRQoL) benefits, recovery, and potential complications to reach an optimal treatment decision.^6,7 QALYs are calculated using utilities which are a number, typically between 0 and 1, that quantifies the preference for (i.e., desirability of) a health state.⁶ The utility of perfect health is set at 1 and the utility of dead is set at 0. Studies on metastatic spine disease have calculated utilities by converting to generic health measure responses (e.g., the EQ-5D, SF-6D, and HUI-3) to utilities using multiattribute utility functions (MAUFs).

Literature examining the economic value of treatments for spinal metastases has been inconclusive. Uncertainty regarding the economic value of treating metastatic spine disease may be a consequence of psychometric limitations of generic health measures, which demonstrate suboptimal validity and responsiveness in conditions relevant to spinal metastases: cancer, spinal cord injury, and musculoskeletal disorders.^6,8–13 In an effort to improve HRQoL assessments in patients with metastatic spine disease, the AO Spine Knowledge Forum Tumor (former Spine Oncology Study Group, SOSG) developed, revised, and validated a spine oncology-specific outcome questionnaire (SOSGOQ2.0).¹⁴ However, since a MAUF has not been developed for the SOSGOQ2.0, this outcome measure cannot be used to calculate QALYs.

We report the development of a MAUF for the SOSGOQ2.0. We first performed shortening of the SOSGOQ2.0 to eight items (SOSGOQ-8D) which was needed to minimize the cognitive burden of the utility valuation exercise.⁶ We then conduct a utility valuation exercise to develop and validate a MAUF for the SOSGOQ-8D using a Discrete Choice Experiment (DCE) with a general population sample.

MATERIALS AND METHODS

Descriptive System

A hybrid concept-retention and factorial analysis approach was used for shortening the SOSGOQ2.0 to the SOSGOQ-8D.^15,16 To maintain content validity and clinical relevance, we planned to retain all four neurologic function single questions (Neurologic:Legs, Neurologic:Arms, Neurologic:Bladder, and Neurologic:Bowel). Therefore, it was necessary to select a single item from each of the remaining four domains (Physical Function, Pain, Mental Health, Social Function) in the SOSGOQ2.0 to achieve an eight item questionnaire. The full analysis is reported in Supplemental Digital Content 1, Text.

The optimal SOSGOQ-8D questions for the non-neurologic domains were: items 3 (self-care), 11 (average pain), 17 (anxiety), 19 (personal relationships). These items were included in the final SOSGOQ-8D which is shown in Table 1.

TABLE 1.

The Eight-item Spine Oncology Study Group Outcomes Questionnaire

Domain (Abbreviation)	Level	Stem	Descriptor
Physical Function (PF)	1	Your spine limits your ability to care for yourself…	Not at all
	2		A little bit
	3		Somewhat
	4		Quite a bit
	5		Very much
Neurologic: Legs (L)	1	You have weakness in your legs…	None
	2		Mild occasionally
	3		Mild constantly
	4		Moderate constantly
	5		Severe constantly
Neurologic: Arms (A)	1	You have weakness in your arms…	None
	2		Mild occasionally
	3		Mild constantly
	4		Moderate constantly
	5		Severe constantly
Neurologic: Bowel (Bow)	1	You have difficulty controlling your bowel function beyond episodes of diarrhea/constipation…	Never
	2		Rarely
	3		Sometimes
	4		Often
	5		Very often
Neurologic: Bladder (Blad)	1	You have difficulty controlling your bladder function…	Never
	2		Rarely
	3		Sometimes
	4		Often
	5		Need a catheter
Pain (P)	1	On average, your back/neck pain is…	None
	2		Very mild
	3		Mild
	4		Moderate
	5		Severe
Mental Health (MH)	1	You feel anxious about your health related to your spine…	Never
	2		Rarely
	3		Sometimes
	4		Often
	5		Very often
Social Function (SF)	1	Your spine condition affects your personal relationships…	Never
	2		Rarely
	3		Sometimes
	4		Often
	5		Very often

Subjects

Participants were recruited from an online market research panel (Toluna Influencers).¹⁷ Quota sampling was used to ensure that the study sample was representative of the general United States population in terms of region, gender, and age based on 2017 United States Census Bureau Population Estimates Program data.¹⁸ The utility valuation protocol was submitted for IRB review and deemed exempt

Health States

The SOSGOQ-8D was converted to a set of health states consisting of eight attributes corresponding to each of the SOSGOQ-8D items and (the duration of survival in the given health state.¹⁹ SOSGOQ-8D rating level phrasing was used in the health state descriptions. Duration of survival was set at: 1 year, 2 years, 5 years, and 10 years.²⁰ Health states were phrased in the second person and structured as declarative sentences.²¹

Discrete Choice Experiment Valuation Task

Utility valuation was conducted using a DCE questionnaire. DCE methodology is simpler than traditional utility valuation and is therefore better suited for online studies.^22,23 In the DCEs for this study, participants were presented with pairs of health states (choice sets) and asked to select the more desirable health state. Choice sets were be presented in a table with differing attributes highlighted (Figure 1).²⁰

Figure 1.

Choice set presentation in online Discrete Choice Experiment. Differing attributes highlighted in green.

Choice Set Selection

To select a manageable subset of choice sets, we first removed clinically unrealistic health states to form a pool of clinically relevant health states. Next, to lessen the cognitive burden of the DCEs, choice sets were developed in which no more than five attributes differed between health states.²⁰ A D-efficient collection of 100 nondominated choice sets was organized into blocks of 10 blocks using the modified Federov algorithm with Ngene software (Supplemental Digital Content 6, Table).^22,24 The design was developed using parameter values from a general population utility valuation study for a cancer-specific quality of life questionnaire using DCE methodology.²⁰ To assess whether participants understood the DCE task, one dominated choice set (one health state is clearly preferable) was added to each block to test for logic. To assess whether participants engaged in the DCE task, one choice set was repeated in each block, but with health state order reversed to test for internal consistency. Therefore, there were a total of 12 choice sets in each block. There were three levels of randomization in the survey. First, participants were randomized to one of the 10 blocks. Second, the order choice set in each block was randomized. Finally health state order was randomized in each choice set.

Survey Procedures

The market research company sent panel members an e-mail invitation to participate in our study. Interested panel members were redirected to a secure website hosting the utility valuation exercise.^17,21 Participants first read brief background information on metastatic spine disease. Next, participants were provided with an explanation of DCEs and shown a worked example. Participants then completed a practice DCE and provided feedback before completing the study DCEs. The website closed by asking patients to provide five-point Likert rating for the statement “[t]his survey was difficult.”

Statistical Analysis

Participants who spent an average of at least eight seconds per choice set; selected the clearly dominant alternative in the dominated choice; and provided consistent responses for the repeat choice set were deemed to have engaged in and understood the DCE tasks. Only these participants were included in analyses.^17,19,25

A multiattribute utility function was estimated from DCE responses using a mixed multinomial-logit regression model (MIXL) using the “mixl” library in the statistical programming language R.^19,26–29 The regression model incorporated the main survival duration effect, and two-way interactions between survival duration and each SOSGOQ-8D item.¹⁹ Each parameter was treated as a random effect to account for participant heterogeneity in the repeated DCE tasks. The random effects were modeled with 1000 draws from a normal distribution. In the base regression model, all SOSGOQ-8D items were coded as nominal categorical (dummy) predictors to avoid assumptions of linear or extra-linear effects. The base regression model was simplified by removing nonsignificant predictors, and combining adjacent predictors to maintain a monotonic decreasing relationship. Model performance during the simplification procedure was monitoring using McFadden's ρ².³⁰ Values between 0.2 and 0.4 indicate very good model fit and are analogous to an R² value between 0.7 and 0.9 for linear regression.

In an effort to strengthen the generalizability of the regression analysis, we implemented validation by allocating participants to a training set and validation set in a 3:1 ratio.³¹ Regression models were fit using only the training set. The performance of the simplified regression model was assessed by prediction accuracy for choice set selections by participants in the validation set using 1000 draws from the MIXL model. Prediction accuracy was quantified using the area under the curve (AUC) interpreted using the following thresholds: excellent, 0.9 to 1; good, 0.8 to 0.9; fair, 0.7 to 0.8; poor, 0.6 to 0.7; and failed, 0.5 to 0.6.³²

Regression coefficients quantify the impact of dysfunction in a particular SOSGOQ-8D item on utility. Since level 1 for all SOSGOQ-8D items is nondysfunctional, this level imparts no change in utility. Under this scheme, utilities can be calculated by substituting the sum of the product of predictors and coefficients for each SOSGOQ-8D item in the formula

$$Utility=1-PF-L-A-Bow-Blad-P-MH-SF.$$ (1)

A worked example is provided in the Results section.

Since, the MIXL model treats each coefficient as a normal (“bell-curve”) random variable, regression results consisted of a mean and standard deviation for each coefficient. In this way MIXL techniques model heterogeneity (differences between individuals) of the utility impact of dysfunction in the SOSGOQ-8D items. Thus to predict how a single individual values the utility of each SOSGOQ-8D item, a random draw is made from the normal distributions estimated by the MIXL model. The mean coefficient values are the expected values for a single individual. In accordance with best practices in health economics, a SOSGOQ-8D utility scoring rubric was developed using mean values.⁶ The importance of individual SOSGOQ-8D items was quantified by calculating the difference in utilities between the best and worst levels of the attribute.³³

Sample Size Calculation

Sample size was guided by S-efficiency which is a measure of the minimum sample size to estimate statistically significant regression parameters at the 95% level.³⁴ The minimum sample for the DCE design shown in Supplemental Digital Content 6, Table is 2039 participants. Since we partitioned data in a 3:1 ratio, the sample size for the training set was 2039, and for the validation set was 691.

RESULTS

Of the participants who began the DCE exercise, 95.3% completed the task. All geographic, sex, and age quotas based on the 2017 United States Census Bureau Population Estimates Program were met.¹⁸ There were no statistically or qualitatively significant differences between the training set and validation set in terms of sex, age, or census region (Table 2). Most of the participants in the training and validation sets (69% and 70%, respectively) did not agree with the statement that “this survey was difficult.”

TABLE 2.

Respondent Demographic Characteristics

	Training Set, N = 2039	Validation Set, N = 691
Sex, no. (%)
Female	1034 (51)	366 (53)
Male	1005 (49)	325 (47)
Age, y, no. (%)
18–29	434 (21)	153 (22)
30–39	358 (18)	112 (17)
40–49	325 (16)	114 (16)
50–59	356 (17)	115 (17)
60–69	299 (15)	98 (14)
≥70	267 (13)	99 (14)
Census region, no. (%)
Northeast	364 (18)	120 (17)
Midwest	412 (20)	155 (22)
South	771 (38)	263 (38)
West	492 (24)	153 (23)

Five pairs of adjacent coefficients were collapsed to simplify the base regression model: L2 and L3, A1 and A2, MH1 and MH2, MH2 and MH3, and SF3 and SF4. Model simplification did not have an adverse effect on performance with the training set as McFadden's ρ² remained unchanged at 0.26 which is indicative of an excellent fit. The simplified regression model had excellent external validity as it predicted DCE choices in the validation set well with an AUC of 0.82 (95% confidence interval [CI]: 0.81–0.83).

The final MIXL regression results are shown in Supplemental Digital Content 7, Table. Statistically significant standard deviation for the majority of coefficients indicated the presence of heterogeneity between participants; therefore, use of a MIXL model was appropriate. Regression results revealed that participants did not regard all SOSGOQ-8D items equally important. The rank order of importance scores for the mean coefficient values (decreasing) was: Neurologic:Bowel/ Neurologic:Bladder, Pain, Physical Function/Neurologic:Legs, Social Function, Neurologic:Arms, and Mental Health.

To calculate utilities with Eq. (1), utility values for each item are obtained from the scoring rubric shown in Table 3. SOSGOQ-8D responses must first be converted to numerical levels using Table 1. To illustrate the use of Eq. (1) and the scoring rubric, we will calculate the utility for Patient A who provided the SOSGOQ-8D responses: PF5, L3, A2, Bow1, Blad1, P4, MH4, SF5. For the PF domain, Patient A provided a rating of 5, the corresponding number in Table 2 is 0.35. For the L domain, the patient provided a rating of 3, the corresponding number is Table 2 is 0.05. This process is repeated for each domain and the values substituted into the Equation 1

TABLE 3.

SOSGOQ-8D Utility Scoring Rubric

Level	SOSGOQ-8D Item
	PF	L	A	Bow	Blad	P	MH	SF
1	0	0	0	0	0	0	0	0
2	0.09	0.05	0	0.06	0.05	0.05	0	0.06
3	0.15	0.05	0.09	0.16	0.12	0.13	0.04	0.06
4	0.28	0.13	0.09	0.32	0.24	0.15	0.04	0.18
5	0.35	0.31	0.24	0.39	0.38	0.36	0.10	0.26

To use this table, SOSGOQ-8D responses must converted to numerical levels using Table 1. The appropriate values from this table are then substituted in Eq. (1) to calculate utilities. A indicates Arms; Blad, bladder; Bow, bowel; L, Legs; MH, Mental Health; P, Pain; PF, Physical Function; SF, Social Function.

$$Utility=1-0.35-0.05-0-0-0.15-0.04-0.26=\mathrm{0.15.}$$

DISCUSSION

In this article, we expanded the use of the SOSGOQ2.0 to QALY calculation. In Part 1 we formally shortened the SOSGOQ2.0 to an eight-item SOSGOQ-8D questionnaire.¹⁶ This step was necessary to lessen the cognitive burden of the utility valuation exercise which was reported in Part2.

We quantified a MAUF for the SOSGOQ-8D for the US general population using DCE methods. The MAUF demonstrated good out-of-sample prediction accuracy with an AUC of 0.82 (95% CI: 0.81–0.83). We provided a worked example for a hypothetical patient to illustrate how to calculate utilities using Eq. (1) and Table 3. This utility value quantifies the desirability of Patient A's health state relative to perfect health (PF1, L1, A1, Bow1, Blad1, P1, MH1, SF1) from the general population perspective. A utility of 0.15 means that the general population would regard 365 days of life in Patient A's health as equivalent to 365 days × 0.15 = 55 days of life in perfect health. In other words, if given the option between living 1 year in Patient A's health state, or only living 55 days in perfect health, on average, members of the general population would choose to live a shorter duration with better health. The SOSGOQ-8D MAUF also provides useful insights for clinicians.

The MAUF can also be used to quantify the importance of each SOSGOQ-8D item. Importance scores are listed in Table 3 and quantify the how much individuals discount life in the worst level of each SOGOQ-8D item relative to the best. For example, an importance score 0.33 for both sphincter function domains means that individuals would be willing to trade 365 days × 0.33 = 120 days of life to reverse sphincter dysfunction from its worst state. In contrast, individuals are only willing to trade 365 days × 0.15 = 55 days of life to reverse arm dysfunction from its worst state. Notably, lower extremity neurologic function, which is a common outcome for metastatic spine disease studies, was less important than sphincter function and pain. It was equally important to physical function which was represented by the ability to care for self. Clinicians should heed these findings and offer treatments that maximize function in the most important attributes.

In Part 1 we formally shortened the SOSGOQ2.0 to an eight-item SOSGOQ-8D questionnaire using the “data-splitting” technique for internal validation.^35,36 Based on a priori selection criteria, SOSGOQ2.0 items 3, 11, 17, and 19 were selected because these items performed best on the validation set. Other decision rules for internal validation yield different sets of items. For example, items 4, 13, 17 and 18 were the best performing on the training set and were the second-best on the validation set. An independent test set, in addition to the training and validation sets used, would be required to evaluate different decision rules. Since the SOSGOQ-8D identified in this study have clinical face validity, we feel it is appropriate to use them in future work without further validation with an additional test set.

It is important for readers to appreciate that utilities provided by the general population (ex ante) are not equivalent to utilities obtained from patients who have experienced the health states of interest (ex post).³⁷ Although ex ante utilities are theoretically restricted to system policy decisions, ex ante utilities have become the de facto standard for individual patient decision making. This is because the widely used utilities obtained from generic health surveys such as the EuroQol-5D, Short Form-6D, and Health Utilities Index 3 are actually ex ante valuations.⁶ Therefore generating ex ante utilities for the SOSGOQ-8D conforms with conventions in the literature.

We expanded the use of the SOSGOQ2.0 to QALY calculation. Eq. (1) and Table 3 can be used together to covert SOSGOQ-8D responses to utilities. The regression modeling exercise revealed the relative importance of SOSGOQ-8D items to the general population. These data can be used to help inform population level health care decision-making, such as the allocation of limited resources for specific treatments. The results of this study can also help clinicians counsel patients.

Supplemental Digital Content 2.

Supplemental Digital Content 3.

Supplemental Digital Content 4.

Supplemental Digital Content 5.

Key Points

• Consideration of QALYs can enhance resource allocation and patient counseling.

• We developed a simple technique for converting the SOSGOQ to QALYs weights (utilities)

• Respondents did regard all SOSGOQ domains equally important; therefore, clinicians should heed these findings and offer treatments that maximize function in the most important domains.