Estimating utility values for vesicoureteral reflux in the general public using an online tool

Abstract

Objective

Cost-utility analyses are useful to study conditions without a widely accepted treatment algorithm; in pediatric urology, one such condition is vesicoureteral reflux (VUR). A necessary component of cost-utility analyses is to accurately calculate the “utility”, a numerical surrogate of quality of life, for various health states. Our aims were to determine utility values for representative VUR health states and to verify the feasibility of a novel online platform for utility elicitation in order to reduce the time and expense of such analyses.

Methods

A cross-sectional survey of American adults was conducted using the time-trade-off (TTO) method. Respondents were recruited from an online work interface, Amazon’s Mechanical Turk (MTurk). Four annualized VUR health states were assessed: VUR treated with/without continuous antibiotic prophylaxis (CAP) and with/without associated febrile urinary tract infection (UTI). A 6-week post-operative scenario following open ureteroneocystostomy was also assessed.

Results

We received 278 survey responses (70% response rate). The respondents were largely between the ages of 25 and 44 (59%), female (60%), and Caucasian (76%). Thirty-seven percent had a college degree, and 44% were parents. Compared with a perfect health state of 1.0, we found mean utilities of 0.98 for VUR, regardless of whether CAP was used or whether UTI was present (p=0.9). The immediate post-operative period following ureteroneocystostomy garnered an annualized utility of 0.94.

Conclusions

Our data suggest that MTurk-based utility assessment is feasible, and that subjects view the VUR health state as only slightly inferior to perfect health. This includes VUR health states incorporating CAP and febrile UTI.

Introduction

Vesicoureteral reflux (VUR) is a condition well-known to pediatric urologists, affecting 1% of asymptomatic children and 33–50% of children suffering a febrile UTI [1,2]. Although some cases are associated with recurrent pyelonephritis, renal scarring, hypertension, and/or renal failure [1,2], many other cases will resolve spontaneously and may never be associated with infection or renal injury [3,4]. Given this variable natural history, it is unsurprising that a range of management options exist.

When debate exists as to the “best” treatment for a given condition, cost-utility analysis (CUA) can be helpful to measure the relative value of all options [5]. However, CUA requires that accurate “utility” values are available for the condition in question. A utility is a number (usually between 0 and 1) that quantifies the impact of a health state on the person’s overall well-being, with 1 representing perfect health and 0 representing death. Thus, a very severe condition that negatively impacts all aspect of daily living might have a utility value near 0, whereas a minor condition that does not result in significant impairment might have a utility value very close to 1. Utility values are linear constructs, so the quality of life of someone with a utility of 0.5 is assumed to be exactly half that of someone with a utility of 1.

Typically, utility values are estimated via in-person interviews with the community, during which the interviewer describes the health state and the respondent states how they value the health state compared with perfect health. However, such interviews are costly and time-consuming; hence, moving the utility elicitation process to an online platform may offer several advantages. However, it is unclear whether online surveys will generate responses similar to traditional interviews.

Thus our goals for this study were twofold: (1) to determine utility values for a variety of moderate-grade VUR health states and (2) to assess the feasibility of utility elicitation via an online instrument, MTurk.

Patients and methods

Study design

A cross-sectional survey study was conducted among the general public to elicit utility values for 5 VUR disease states. A time-trade off (TTO) approach was used to determine all utility values.

Study participants

We used the Amazon.com ‘Mechanical Turk’ (MTurk, www.MTurk.com) platform to recruit study participants. MTurk is a crowdsourcing digital marketplace, allowing easy access to a large, stable, and diverse sample of the US population. MTurk users voluntarily register to complete various tasks; participation is typically compensated at $0.05–$1.00 per 5–30 minute task. Estimating that our survey would take 10 minutes to complete, we set our payment at $0.50. Because all payments were made through an intermediary (Amazon.com), participants remained anonymous. Exclusion criteria included residence outside the USA, age <18 years, and lack of English fluency.

Instrument development

Each participant first completed a TTO model related to paraplegia, which served as a practice scenario to familiarize participants with the TTO format and conventions. Participants were instructed to include the value of “preventing the pain, suffering, inconvenience, and lost time for productive activities (like school, work, and household chores) and leisure” for both their hypothetical child and themselves.

We then introduced all participants to VUR via a 4-minute video vignette in which actors portrayed a mother whose child had VUR and a pediatric urologist experienced in treating the condition. This film clip was meant to introduce survey participants to the condition, as they likely had not been previously exposed to VUR. After the video, participants were directed to an “attention question”, which aimed to confirm that participants had viewed the video.

Participants were then presented with the VUR-based clinical scenarios and TTO utility elicitation questions. Each participant was randomly presented with one of four pre-operative health states, and all were presented with a single post-operative health state.

We surveyed respondent age, gender, race/ethnicity, parental status, education achievement and annual income. Given our a priori hypothesis that previous experience with VUR or other chronic health conditions might affect an individual’s preferences for given health states, we also collected data on the health of respondents and their children (if applicable).

Before release, this survey was internally validated using a convenience sample of eight adults, none of whom had a personal or family history of VUR or previous experience with TTO models.

This study was reviewed by our institutional review board and deemed not human subjects research. No formal consent was required, although the principles of the Declaration of Helsinki were followed.

VUR health states

Four pre-operative VUR health states were described: (1) VUR with CAP and a UTI; (2) VUR with CAP but no UTI; (3) VUR with a UTI but no CAP; and (4) VUR in the absence of CAP or UTI. UTIs were described as: “When a child has a urine infection, they usually have a high fever. They may feel pain or burning with urination and pain in their side or belly. This is treated with antibiotics several times each day for 1–2 weeks. Some children need to be in the hospital for a few days and may need stronger antibiotics. Most children recover, but these infections can cause kidney damage.” A surgery state described open ureteroneocystostomy.

The content of the health state descriptions was compiled based on: (1) interviews with patients and families affected by VUR, (2) review of the literature, and (3) expert opinion. The health states were designed to reflect typical disease courses for a child with grade III VUR.

Utility elicitation

Utility elicitation used the TTO method, with a practice health state (paraplegia) used to allow respondents to become familiar with (1) the format of disease state descriptions and (2) the concept of rating a health state. Participants were instructed to perform the TTO as if they were the parent of a child with VUR and as if they were trading years of their child’s life.

Timeframes offered in the TTO exercises varied by disease state presented. For the preoperative VUR TTO exercises, a 10-year timeframe was used with the respondent offered some decreased amount of time in ‘perfect health’ compared with 10 years in the disease state. Health durations changed by 1 year for responses from 1 to 9 year TTO, and could then be narrowed to between 0 and 365 days for respondents willing to trade less than 1 year’s time. A similar method was used for the post-op VUR state; however, given the speedy recovery usually seen in children, this TTO was based on a 6-week timeframe, with a 0–7 day option being presented to participants who were only willing to trade ≤1 week of life to avoid VUR.

Statistical analysis

TTO utility values were calculated using the formula: ‘Utility = time in full health/time in disease state.’ CDC actuarial life tables were used to estimate remaining years of life for a 6-year-old child.

Mean (± standard deviation (SD)) utility values and descriptive statistics were calculated using SAS version 9.3. Kruskal-Wallis or Wilcoxon rank-sum tests were used as appropriate for bivariate comparisons of utility values and respondent characteristics.

Results

Respondent characteristics

Between April and May 2013, 397 MTurk workers voluntarily participated in the survey, 276 of whom completed the task for a 69.5% response rate (Table 1). Most were aged 25–34 years (41%), female (60%), and Caucasian (76%). Married and single respondents were evenly represented (40% vs. 42%); most did not have children (56%).

Table 1.

Respondent demographics

Characteristic	N (%)
Age
18–24	45 (17)
25–34	114 (41)
35–44	49 (18)
45–54	34 (12)
55–64	29 (11)
65–74	4 (1)
75 or older	2 (0.7)
Gender
Female	166 (60)
Race/Ethnicity
African-American	23 (8)
Asian	21 (8)
Caucasian	210 (76)
Hispanic	12 (4)
Native American	4 (1)
Other	2 (0.7)
Two or more races	5 (2)
Marital status
Living with partner	25 (9)
Married	112 (40)
Separated	19 (7)
Single	118 (43)
Widowed	3 (1)
Parental status
Have children	153 (56)
Educational attainment
Some college	77 (28)
2-year college degree	35 (13)
4-year college degree	104 (37)
Master’s degree	18 (7)
Professional degree	6 (2)
Doctoral degree	8 (3)
High school diploma or GED	29 (10)
Less than high school diploma	2 (0.7)
Household median income
20–29,000	45 (16)
30–49,000	79 (28)
50–69,000	50 (18)
70–99,000	34 (12)
≥100,000	22 (8)

Most of the respondents reported excellent personal health, although some reported having depression (12%), hypertension (8%), diabetes (3%), or other chronic diseases (8%).

Of the complete respondents, 56 (20%) chose an incorrect response to the attention question. However, there was no difference in utility values by their responses (0.9836 incorrect vs. 0.9837 correct, p=0.98). We therefore did not exclude these responses.

Utilities by health state

We found a mean utility of 0.73 for our test state, paraplegia, in line with other reported utility values [6]. This did not vary significantly by respondent characteristics such as age or gender.

For our test condition, non-operative VUR, respondents indicated a mean utility value of 0.98 for non-operative VUR, regardless of whether CAP was used or whether UTI was present (Table 2). Using standard CDC actuarial tables to predict remaining years of life for a 6-year-old child, we found that pre-operative utility for children on CAP with UTIs was 0.9830 (±0.035) and without UTIs was 0.9834 (±0.037). For children with UTIs not taking CAP, annual utility was 0.9829 (±0.033). For children with VUR but no history of CAP or UTIs, pre-operative annual utility was 0.9827 (±0.035). For a 6-year-old child, there was no significant difference between the utilities for any given health state (p=0.93).

Table 2.

Utility values for pre-operative VUR health states

	6-year-old child
VUR Mean (SD)	0.9827 (0.035)
VUR with UTI Mean (SD)	0.9829 (0.033)
VUR with CAP Mean (SD)	0.9834 (0.037)
VUR with UTI and CAP Mean (SD)	0.9830 (0.035)

Utility values were higher among respondents who were themselves parents (0.9905 vs. 0.9768, p=0.0004), but did not vary significantly based on respondent age (p=0.14), gender (p=0.10), race/ethnicity (p=0.28), marital status (p=0.06), education level (p=0.19), household income (p=0.30) or presence of comorbid conditions (p=0.47) (Table 2).

The short-term postoperative state had a mean utility value of 0.50 for the 6-week period immediately after ureteroneocystostomy; averaged over a year, this gives an annualized value of 0.94 (assuming the rest of the year was spent in perfect health). These values did not significantly vary with which pre-operative VUR scenario respondents were given (range 0.35–0.61, p=0.22).

Discussion

CUA can provide valuable insights into management strategies for conditions such as moderate VUR that do not yet have a preferred treatment algorithm. Accurately calculating the utility of various health states is critical for successful CUA research. However, utility elicitation via typical face-to-face interviews can be time-consuming and expensive.

We sought to assess various VUR health states using a novel online survey platform, with the hopes of achieving two goals: first, to determine utility values for several moderate VUR health states and second, to assess the feasibility of utility elicitation via an online environment. We found annual utilities of 0.98 for a variety of pre-operative VUR health states (both with and without CAP and UTIs) and an annual utility of 0.94 for the period immediately following surgery. There is little difference among utility values for the health states we queried. Indeed, obtaining accurate utilities for children’s health states poses several well-known methodological challenges, including a general uneasiness with trading any time for mild decrements in health; difficulty contextualizing a child’s illness in the overall well-being of the family/household; competing choices faced by parents (e.g. positive value being placed on spending time with a sick child); and caretaker guilt.

A recent review reveals 10 competing published treatment guidelines for moderate (Grade III) VUR [7]; the authors point toward the need for ongoing and future research to clarify best practices. However, such research is years away from completion and may not answer all of the clinical questions that arise in the management of VUR [8]. In the interim, urologists still need to make treatment decisions for their patients, explaining the growing interest in CUA and modeling for VUR [9–11]. Accurate estimates of VUR utilities are a critical component of these analyses.

Assessments of utilities remain relatively novel in urology. Fann et al. found that patients valued ureteroscopy (0.914) lower than extracorporeal shock wave lithotripsy (0.967), but noted that both were less preferable than a chronic urolithiasis state (0.974) [12]. Mesrobian estimated the utility of childhood ureteropelvic junction obstruction, but his estimated annual utility range of 0.5–0.8 was drawn from the literature, not determined by interview. Other reported conditions in this range are quite morbid, for example renal failure requiring hemodialysis [13] and progressive lymphoma after chemotherapy (0.6) [14]. By comparison, intermittently obstructive UPJO would seem to have a lesser effect on quality of life, suggesting that Mesrobian’s UPJO utility may be underestimated. Our VUR utility results were similar to others for childhood conditions collected via standard interviews (e.g. otitis media 0.9950, bacteremia 0.9921), lending credence to the hypothesis that online TTO determination is a viable mode of utility elicitation [15].

We found a lower preference for the post-operative health state, with an annualized utility of 0.94 (6-week utility of 0.50). This is very similar to previously published estimates [16–18] and is in line with other pediatric conditions, where short-term health states are often found to have lower utility than longer-term health states [19–21]. In a study of childhood pertussis, a ‘persistent cough’ over 3 months was found to have a utility value of 0.67. The same cough, extended over a lifetime, had a utility of 0.83 [19].

This study should be interpreted in light of its limitations. MTurk respondents are not necessarily representative of the US population, as shown in Table 1. However, MTurk has been shown to be more representative than other commonly used convenience samples (e.g. undergraduates) [22,23]. Multiple studies have demonstrated that survey results derived from MTurk populations are similar to those derived from more traditional, community-interview methods [24–27]. Further, our 70% response rate is well within acceptable standards for survey research [28], and our utility values were quite similar to previous studies [16–18]. Nonetheless, our specific results have not yet been validated against a sample using standard face-to-face interview techniques and should not be widely adopted prior to validation.

TTO methods are sensitive to risk aversion, albeit less than standard gamble methods [29]. For example, respondents may be unwilling to trade any length of life for a quality improvement. As such, our use of TTO may have biased our utility values to be lower than expected had we used a standard gamble method.

The relative unimportance of the attention question merits further comment, as we expected this would significantly impact our results. We suspect this to be a multifactorial effect, as respondents are educated to VUR by reading our introductory material, viewing the video vignette, and by reading each health state description. The redundant nature of these descriptions likely explains much of this effect. Importantly, other groups using MTurk have reported similar lack of importance for the attention question on well-validated instruments [30].

As our sample consisted of both parents and childless respondents, it is reasonable to question how the latter group can adequately assess the utility of any pediatric health state. However, it is a common misconception that one must study an at-risk or affected to quantify the utility of a given health-state; this approach actually inflates utility scores and makes them less meaningful. The standard in cost-utility and cost-effectiveness research is to assess various health-states from a societal perspective, not a parental or patient perspective [5,6,31–33]. Thus, inclusion of parents and non-parents alike in this TTO utility calculation is the most appropriate route.

Finally, as we asked respondents to evaluate the TTO based on both the child’s and the parents’ perceived distress, our utility values may not be fully comparable with those in other studies which took an exclusively parental- or child-centric view, instead of evaluating the combined utility [5]. This question is currently under study.

Conclusion

Using a novel online platform, we estimate the utility value of pre-operative VUR states as 0.98, regardless of CAP use or UTI occurrence. The immediate post-operative period demonstrated a reduced annualized utility of 0.94. These values are commensurate with those collected by traditional TTO interview methods, suggesting that online utility elicitation may be feasible.