Abstract
BACKGROUND
The advent of online task-distribution has opened a new avenue to efficiently gather community perspectives needed for utility estimation. Methodological consensus for estimating pediatric utilities is lacking, with disagreement over whom to sample, what perspective (patient vs. parent), and whether instrument-induced anchoring bias is significant. We sought to determine what methodological factors potentially impact utility estimates for vesicoureteral reflux (VUR).
DESIGN
Cross-sectional surveys using a time-trade-off (TTO) instrument were conducted via Amazon's Mechanical Turk online interface; respondents were randomized to answer questions from child, parent, or dyad perspectives on the utility of a VUR health state and one of three “warm-up” scenarios (paralysis, common cold, none) prior to a VUR scenario. Utility estimates and potential predictors were fitted to a generalized linear model to determine what factors most impacted utilities.
RESULTS
A total of 1,627 responses were obtained. Mean respondent age was 34.9 years; 48% were female; 38% were married; and 44% had children. Utility values were uninfluenced by child/personal VUR/UTI history, income, or race. Utilities were affected by perspective, and were higher in the child group (34% lower in parent v. child, p<0.001; 13% lower in dyad v. child, p<0.001). VUR utility was not significantly affected by the presence or type of TTO warm-up scenario (p=0.17).
CONCLUSIONS
TTO perspective affects utilities when estimated via an online interface, however, utilities are unaffected by the presence, type, or absence of warm-up scenarios. These findings could have significant methodological implications for future utility elicitations in other pediatric conditions.
Keywords: Pediatric, Vesicoureteral reflux, utility
INTRODUCTION
Vesicoureteral reflux (VUR) is among the most frequently encountered conditions in pediatric urologic practice, affecting up to 70% of children who present with febrile UTI and approximately 1% of all children in the United States.1 As understanding of VUR continues to evolve, consensus remains elusive on how best to manage this condition.2,3
Determining optimal treatment pathways for conditions with multiple divergent management options such as VUR can be challenging for providers and families alike. Cost-utility analyses (CUA) provide relative value comparisons and can guide clinical decision-making.4, 5 However, CUA requires condition-specific “utility” values – numerical representations of the impact of health-states or diseases on health-related quality of life (HRQoL).4, 6 Utilities are numerically represented on a scale of 0 (HRQoL equal to death) to 1 (perfect health); as such, the HRQoL of a condition with a utility of 0.5 is assumed to be exactly half that of a condition with a utility of 1. Similarly, more severe conditions with greater impact on aspects of daily living are more likely to have utility values nearer 0 than minor conditions with less impairment.
Typically estimated via direct interviews with members of the general public, or indirectly via conversion from HRQoL instruments, utility value estimates are time, cost, and resource-intensive. Pediatric utility estimation is additionally complex given the dependency of children on their caregivers and the attachment required of caregivers caring for a sick child.7 As such, the impact of a particular health-state is not only felt by the child, but also by families.8 Whether parents, affected children, or both should form the basis for utility estimation is unclear, and it is unknown to what degree existing utility methodologies truly capture accurate estimates of health-state utilities in children. Given these challenges, it is unsurprising that utility values – and, by extension, CUAs – are scarce for pediatric conditions.6, 9
We have previously estimated the utility of pediatric VUR health states via online and traditional methods with comparable results, demonstrating the potential for use of such platforms in urologic research.10, 11 Despite growing use of online utility estimation, standardized methodologies for such approaches remain elusive. To identify considerations for approaching utility estimation online, we sought to identify potential sources of bias inherent in online utility instruments. In particular, to determine how disease point-of-view affects pediatric utilities given aforementioned HRQoL spillover effects, and whether attempts to educate survey participants about utility estimation via “practice” or “warm-up” scenarios may inadvertently introduce degrees of anchoring bias, hypothesizing that both would significantly alter estimates of VUR utility.
PATIENTS & METHODS
Study Design
Cross-sectional survey studies were conducted amongst the general public to elicit utility values for VUR. Survey instruments varied either in time-trade perspective (patient, parent, or dyad of patient and parent) with consistent paraplegia warm-up scenario, or varied warm-up scenario (paraplegia, common cold, or blank/no-scenario) with consistent dyad perspective. A time-trade off (TTO) approach was used to determine all utility values.
Study Participants
We used Amazon.com’s ‘Mechanical Turk’ (AMT, www.mturk.com) online survey environment to recruit study participants. This platform serves as a crowdsourcing digital marketplace, allowing ready access to a large, stable, and diverse sample of American adults.12, 13 AMT users voluntarily register to complete various “human intelligence” tasks. Each worker is assigned an ID/tracking number, preventing task repetition.
Participation in AMT tasks is typically compensated at a rate of $0.05–$1.00 for a task requiring 5–30 minutes to complete. Estimating that our survey would take 20–30 minutes to complete, we set our payment at $1.00. Because all payments were made through an intermediary (Amazon.com), participants remained strictly anonymous. Exclusion criteria included residence outside the United States, age <18 years, and lack of English fluency. As utility values are calculated based on the perspectives of the community at-large rather than only those affected by a condition, participants were included even if they were unfamiliar with VUR, did not have children, or were not married.
Instrument Development
Participants first completed a TTO model related to an easily-understood, non-urologic health state for familiarization with TTO format and conventions, as participants were assumed to have no background or prior experience with this type of instrument. Respondents in the perspective-variable group were introduced to TTO via a paraplegia scenario, whereas respondents in the warm-up-variable group were randomized to encounter a paraplegia, common cold, or blank/no-warm-up scenario. Participants were instructed to assume they were the parent of a hypothetical 6 year-old child affected by the health state in question, and to include the value of “preventing the pain, suffering, inconvenience, and lost time for productive activities (like school, work, and household chores) and leisure” when evaluating TTO propositions.
Subsequently, all participants were introduced to VUR via a 4-minute video (Appendix 2). A text vignette describing VUR was also presented. Participants were then presented with the VUR-based clinical scenarios and TTO utility elicitation questions based on a hypothetical 6-year-old VUR patient. The content of the health state descriptions was compiled based on: 1) interviews with patients and families affected by VUR, 2) review of the literature, and 3) expert opinion. Health states were designed to reflect typical disease courses for VUR in a child with moderately severe reflux.
Participants in the perspective-variable cohort were randomized into three groups – child, parent, or dyad. Group allocation determined the context of TTO questions posed to the respondent (Appendix 1). Child-group respondents were asked to consider time-trades from the life of their hypothetical, affected 6 year-old child in exchange for the child spending time in a disease-free state. Parent-group respondents considered trades from their own lives to benefit their child. Similarly, questions posed to dyad-group respondents involved combined trades from both the parent’s and child’s lives.
Participants in the warm-up-variable group were asked to exclusively consider time trades from a dyadic perspective.
Utility elicitation
A 10-year timeframe was used for both practice and experimental scenarios, with the respondent offered a variable amount of time spent in ‘perfect health’ compared to 10 years in the disease state, in exchange for hypothetical trades of fixed amounts of time from their overall lifespan. Health durations changed in 1 year intervals for responses from 1 to 9 year TTO, and could be narrowed to between 0–365 days for respondents willing to trade less than one year’s time. This process allows only utilities between 0 and 1, since negative utility values are not possible with this experimental design.
We collected respondent demographics including age, gender, marital status, parental status, race/ethnicity, education achievement, annual income, and prior VUR or TTO related knowledge. Given our hypothesis that previous experience with VUR or other chronic health conditions might affect an individual’s preferences for given health states, we also collected data on the health of respondents and their families.
Prior to release, each survey instrument was internally validated using a convenience sample of medical students and residents, none of whom had a personal or family history of VUR or previous experience with TTO models.
This study was reviewed by our Institutional Review board and deemed to not be human subject research. No formal consent process was thus required, though the principles of the Declaration of Helsinki were followed.
Statistical Analysis
Mean (+/− Standard Deviation (SD)) utility values and descriptive statistics were calculated. Our sample size calculation confirmed that at least 500 patients would be required to give a representative sample of the US population with a margin of error of ±5 percentage points. Multivariate generalized linear model was fitted to determine what factors most impacted utility estimates. Independent variables were selected according to a priori knowledge or p-value of 0.2 on bivariate analysis.
An alpha of 0.05 and 95% confidence intervals (CI) were used as criteria for statistical significance. All analyses were performed using SAS 9.4 (SAS Institute, Cary, NC).
RESULTS
Characteristics of Study Sample
1,852 AMT workers voluntarily participated in the surveys with 1,627 completing the task (87.9% response rate, Table 1). Mean respondent age was 34.9 years; 48% were female. A plurality (623, 38%) was married; 708 (44%) had children. Most were Caucasian (1,361, 84%) and 41% reported completing a 4-year college degree or higher. Most respondents reported limited-or-no knowledge of VUR (1,444, 89%) or of CUA (1,204, 74%). By contrast, respondents reported a higher rate of experience with UTI (personal: 433, 27%; family/relative: 464, 29%)
Table 1.
Respondent Demographics by Perspective Group
| Perspective-Variable Group (n=873) |
Warm-up-Variable Group (n=753) |
Total (%) (n=1627) |
||
|---|---|---|---|---|
| Age (Mean, SD) | 34.4 (11.1) | 35.8 (11.5) | 34.9 (11.3) | |
| Gender | ||||
| Male | 462 (53%) | 385 (51%) | 846 (52%) | |
| Female | 408 (47%) | 369 (49%) | 777 (48%) | |
| Marital status | ||||
| Single | 408 (47%) | 298 (40%) | 706 (43%) | |
| Married | 299 (34%) | 324 (43%) | 623 (38%) | |
| Separated/Divorced | 62 (7.1%) | 48 (6.4%) | 109 (7%) | |
| Widowed | 5 (0.5%) | 6 (0.8%) | 11 (1%) | |
| Living with Partner | 98 (11%) | 78 (10%) | 176 (11%) | |
| Children | ||||
| Yes | 340 (39%) | 369 (49%) | 708 (44%) | |
| No | 531 (61%) | 385 (51%) | 916 (56%) | |
| VUR knowledge | ||||
| Never heard of it | 680 (78%) | 535 (71%) | 1141 (70%) | |
| May have heard of it, but not sure | 156 (18%) | 174 (23%) | 303 (19%) | |
| Concept but not details | 30 (3.4%) | 77 (10%) | 135 (8%) | |
| Pretty good understanding | 5 (0.5%) | 36 (4.8%) | 38 (2%) | |
| Ample knowledge | 2 (0.1%) | 8 (1.1%) | 9 (0%) | |
| UTI knowledge | ||||
| Never heard of it | 15 (1.7%) | 26 (3.5%) | 38 (2%) | |
| May have heard of it, but not sure | 52 (5.9%) | 56 (7.4%) | 102 (6%) | |
| Concept but not details | 452 (52%) | 314 (42%) | 717 (44%) | |
| Pretty good understanding | 297 (34%) | 350 (47%) | 609 (37%) | |
| Ample knowledge | 57 (6.5%) | 116 (15%) | 160 (10%) | |
| CUA knowledge | ||||
| Never heard of it | 399 (46%0 | 315 (42%) | 657 (40%) | |
| May have heard of it, but not sure | 308 (35%) | 258 (34%) | 547 (34%) | |
| Concept but not details | 125 (14%) | 209 (28%) | 311 (19%) | |
| Pretty good understanding | 34 (4%) | 60 (8.0%) | 90 (6%) | |
| Ample knowledge | 7 (0.8%) | 15 (2.0%) | 27 (1%) | |
| VUR history (self) | 2 (0.2%) | 33 (4.4%) | 35 (1%) | |
| UTI history (self) | 227 (26%) | 206 (27%) | 433 (27%) | |
| VUR history (child) | 19 (2.2%) | 47 (6.2%) | 27 (2%) | |
| UTI history (child) | 19 (2.2%) | 47 (6.2%) | 66 (4%) | |
| VUR history (relative) | 3 (0.3%) | 35 (4.6%) | 39 (2%) | |
| UTI history (relative) | 195 (22%) | 203 (27%) | 398 (24%) | |
| Education | ||||
| Less than high school | 7 (0.8%) | 8 (1.1%) | 15 (1%) | |
| High school diploma or GED | 118 (14%) | 65 (8.6%) | 184 (11%) | |
| Some college | 256 (36%) | 186 (25%0 | 442 (27%) | |
| 2-year college degree | 112 (13%) | 79 (11%) | 191 (12%) | |
| 4-year college degree | 311 (36%) | 314 (42%0 | 625 (38%) | |
| Masters degree | 52 (6.0%) | 87 (12%) | 139 (9%) | |
| Doctoral/professional degree | 17 (1.9%) | 14 (1.9%) | 31 (2%) | |
| Income | ||||
| Less than $20,000 | 164 (19%) | 103 (14%) | 307 (19%) | |
| $20,000–$49,000 | 405 (46%) | 321 (43%) | 726 (45%) | |
| $50,000–$89,000 | 208 (24%) | 419 (56%) | 419 (26%) | |
| $90,000 or greater | 96 (11%) | 78 (10%) | 174 (11%) | |
| Race | ||||
| White | 725 (83%) | 537 (71%) | 1361 (84%) | |
| Black | 55 (6.3%) | 54 (7.2%) | 199 (7%) | |
| Other | 108 (12%) | 163 (22%) | 257 (16%) | |
Of the perspective-variable cohort (873 total), we randomly assigned 307 to the child perspective, 284 to the parent perspective, and 282 to the combined parent-child dyad perspective.
Of the warm-up-variable cohort (753 total), we randomly assigned 258 to the paraplegia scenario, 255 to common cold, and 240 to bypass the warm-up altogether.
Utilities by Health State Perspective
No respondent characteristics except for utility perspective were found to be associated with VUR utility on bivariate analysis. Multivariate analysis (Table 2) demonstrated VUR utility estimated from the parent-perspective group was 34.21% lower than utility estimated from the child-perspective (p<0.001); and was 13.16% lower in the dyad-perspective group versus child-perspective (p<0.001). Utility was found to be higher amongst single respondents (0.07 higher in single vs. married, p=0.007), and among older respondents (0.03 increment per 10-year age difference, p=0.008). Estimated utility values of the paraplegia warm-up scenario completed by respondents in this cohort were found to follow a similar pattern (parent:child utility: −44.44%; dyad:child utility: −17.46%; p<0.001).
Table 2.
Multivariate Analysis of Demographic Variables vs. Perspective-dependent VUR Utility
| Respondent Characteristics | Estimate | p value | |
|---|---|---|---|
| Perspectives | |||
| Child | reference | ||
| Parent | −0.27 | <0.001 | |
| dyad | −0.09 | <0.001 | |
| Age (increment of 10 years) | 0.03 | 0.008 | |
| Marital status | |||
| Single | reference | ||
| Married | −0.07 | 0.007 | |
| Separated/Divorced | −0.08 | 0.08 | |
| Widowed | 0.21 | 0.15 | |
| Living with Partner | −0.01 | 0.79 | |
| UTI history (self) | |||
| Yes | reference | ||
| No | −0.04 | 0.12 | |
| UTI history (child) | |||
| Yes | reference | ||
| No | −0.09 | 0.20 | |
| Income | 0.01 | 0.28 | |
| Race | |||
| White | reference | ||
| Black | 0.05 | 0.21 | |
| Other | −0.05 | 0.12 | |
Utilities by Warm-up Scenario
VUR utilities were similar (p=0.17) in the paralysis (0.868), common cold (0.877), and blank/no-warm-up (0.871) groups. After adjusting for previous experience with VUR (present in either a child or a friend/relation), ethnicity, and race, VUR utility was still not significantly affected (p=0.87) by the presence or type of warm-up scenario (Table 3).
DISCUSSION
Utility theory has found significant applications in healthcare and clinical research.4, 14, 15 Requisite health-state utility estimates needed for CUA require considerable financial and time investments to obtain, particularly via traditional TTO methods. Internet-based estimation provides a promising opportunity to bypass these barriers.
We sought to identify potential methodological considerations for researchers considering the use of these platforms for utility estimation in pediatric urology. Our analysis revealed two important findings: that time-trade perspective significantly affects estimates of health-state utility, and that anchoring bias does not seem to be a factor in use of warm-up/familiarization scenarios in VUR instruments.
A perspective-dependent VUR utility was noted despite identical VUR health states being queried. Utilities were lowest when caregivers made time trades from their own lives, intermediate when time was traded from both the caregiver’s and child’s life, and highest when traded exclusively from the child’s life, suggesting respondents’ perceptions of HRQoL impact are not only influenced by the physical effects of a condition but by external factors as well. Critically, utilities may not only be disease-specific, but perspective specific as well – a difference which may propagate through CUAs in which they are used, thereby reducing the trustworthiness of such analyses for medical decision-making.
Our findings lend credence to an intuitive assumption that parents and caregivers disparately consider trade-offs in exchange for their child’s benefit. Likely multifactorial, this observation may stem from perspective-dependent variation in respondent risk aversion.7, 16–18 As noted by Hellinger and colleagues, individuals generally display reluctance to take significant risks during healthcare decision-making, with TTO methods particularly sensitive to this phenomenon.18 Consequently, such methods may inadvertently underestimate HRQoL impact, thereby overestimating health-state utility. Risk aversion is even more pronounced when considering pediatric conditions, given the roles of emotional attachment and parental/familial altruism inherent in such situations.7, 19–21 Studies in other pediatric surgical subspecialties have demonstrated risk aversion behavior amongst parents considering surgical treatment for their children.20, 22 Despite their fervent desire for positive outcomes, parents and other adults display reluctance to consent children for risky or invasive therapies, preferring safer options if available even if less efficacious. It is reasonable to assume this differential would similarly persist when respondents consider hypothetical pediatric TTO scenarios.
Pediatric utilities may further be influenced by emotional, financial, and other strain placed on caregivers and families caring for a sick dependent child. Described by Prosser and colleagues as so-called ‘spillover’ effects on HRQoL, such consideration may account for a degree of the perspective-dependent utility differential observed.8 This finding would seem applicable to other fields where proxy respondents must consider treatment effects both within their own and their ward’s utility, such as geriatrics.
Surprisingly, anchoring effects – whereby exposure to one set of information cognitively biases an individual’s perception of a subsequent set of information – were not associated with use of warm-up scenarios in our study. As noted by Paine et al, literature regarding anchoring phenomenon in patient preference assessment is conflicting and incomplete.23 We are reassured that online utility estimation appears unaffected by this phenomenon given consistent VUR utilities reported after exposure to information about conditions with both severe and minimal HRQoL impact.10, 11 Such exercises are generally assumed to be useful for providing necessary education on otherwise-obscure concepts associated with TTO and utility theory; particularly in the virtual environment where lack of direct interaction limits researchers’ abilities to directly answer respondent questions related to experimental design and procedures. Interestingly, however, VUR utility was also unaffected when respondents were not exposed to any warm-up scenario at all, raising questions about this assumption.
Multiple authors have previously demonstrated comparable results from AMT sample populations to those gathered via traditional methods for utilities research.10, 13, 24–26 This platform, however, is not without limitations. However, gathering a truly representative sample population in a cost and time-effective manner has long been a particular challenge in utilities research.27 Prior studies have found the AMT population to be more diverse and nationally-representative than those from other widely used convenience samples, such as US undergraduates, though not as representative as other (and costlier) online probability samples.12, 13, 26
Additional limitations must also be considered when interpreting our study results. As mentioned, in the absence of direct interviews we were unable to query respondents for further explanation of their decisions. Thus, we can only speculate about their reasoning and its ultimate impact on utility estimation. Our study design restricted participation to respondents over the age of 18. However, we intentionally did not restrict respondents to only those who are/were parents, nor to those familiar with VUR. Indeed, a slight majority of respondents in our study (56%) described themselves as childless. Though it is reasonable to question whether this had an impact on utility values when asking about a pediatric condition, literature supports the use of diverse, non-affected individuals in gathering utility data to prevent over-inflation of utility scores.6 Furthermore, we observed no significant difference in mean utility scores from parents (p=0.33). Lastly, use of a 10-year time frame for VUR TTO is somewhat inconsistent with the condition’s generally benign, but prolonged, clinical course. However, such a time frame has been previously used by other authors for other indolent conditions.4, 11, 28
CONCLUSION
The utility of VUR in children is consistently highest when a child perspective is used and lowest when a parent perspective is used for elicitation. A combined parent-child dyad consistently provides a mid-range value between these two extremes. Warm-up exercises appear to have little effect on utility estimation for our condition of interest.
Supplementary Material
Acknowledgments
Funding Source: Financial support for this study was supported by grants K08-DK100534 (PI: Routh) and K23-DK088943 (PI: Nelson) from the National Institute of Diabetes and Digestive and Kidney Diseases (NIDDK). The funding agreement ensured the authors’ independence in designing the study, interpreting the data, writing, and publishing the report.
Abbreviations Used
- VUR
Vesicoureteral Reflux
- HRQoL
Health-Related Quality of Life
- AMT
Amazon.com Mechanical Turk (MTurk)
- UTI
Urinary Tract Infection
- TTO
Time Trade Off
Footnotes
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
Conflict of Interest: The authors have no relevant financial relationships to the article to disclose.
Presented at 2016 Society for Pediatric Urology meeting, San Diego, CA
REFERENCES
- 1.Pohl HG, Joyce GF, Wise M, et al. Vesicoureteral Reflux and Ureteroceles. The Journal of Urology. 2007;177:1659. doi: 10.1016/j.juro.2007.01.059. [DOI] [PubMed] [Google Scholar]
- 2.Heidenreich A, Ozgur E, Becker T, et al. Surgical management of vesicoureteral reflux in pediatric patients. World J Urol. 2004;22:96. doi: 10.1007/s00345-004-0408-x. [DOI] [PubMed] [Google Scholar]
- 3.Routh JC, Bogaert GA, Kaefer M, et al. Vesicoureteral reflux: current trends in diagnosis, screening, and treatment. Eur Urol. 2012;61:773. doi: 10.1016/j.eururo.2012.01.002. [DOI] [PubMed] [Google Scholar]
- 4.Prosser LA, Hammitt JK, Keren R. Measuring health preferences for use in cost-utility and cost-benefit analyses of interventions in children: theoretical and methodological considerations. Pharmacoeconomics. 2007;25:713. doi: 10.2165/00019053-200725090-00001. [DOI] [PubMed] [Google Scholar]
- 5.Benoit RM, Peele PB, Cannon GM, Jr, et al. The cost-effectiveness of dextranomer/hyaluronic acid copolymer for the management of vesicoureteral reflux. 2. Reflux correction at the time of diagnosis as a substitute for traditional management. J Urol. 2006;176:2649. doi: 10.1016/j.juro.2006.08.054. [DOI] [PubMed] [Google Scholar]
- 6.Matza LS, Swensen AR, Flood EM, et al. Assessment of health-related quality of life in children: a review of conceptual, methodological, and regulatory issues. Value Health. 2004;7:79. doi: 10.1111/j.1524-4733.2004.71273.x. [DOI] [PubMed] [Google Scholar]
- 7.Matza LS, Boye KS, Feeny DH, et al. Impact of caregiver and parenting status on time trade-off and standard gamble utility scores for health state descriptions. Health Qual Life Outcomes. 2014;12:48. doi: 10.1186/1477-7525-12-48. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Prosser LA, Lamarand K, Gebremariam A, et al. Measuring family HRQoL spillover effects using direct health utility assessment. Med Decis Making. 2015;35:81. doi: 10.1177/0272989X14541328. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Prosser LA. Current challenges and future research in measuring preferences for pediatric health outcomes. J Pediatr. 2009;155:7. doi: 10.1016/j.jpeds.2009.03.007. [DOI] [PubMed] [Google Scholar]
- 10.Lloyd JC, Yen T, Pietrobon R, et al. Estimating utility values for vesicoureteral reflux in the general public using an online tool. J Pediatr Urol. 2014;10:1026. doi: 10.1016/j.jpurol.2014.02.014. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Nelson CP, Routh JC, Logvinenko T, et al. Utility scores for vesicoureteral reflux and anti-reflux surgery. J Pediatr Urol. 2015 doi: 10.1016/j.jpurol.2015.03.008. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Briones EM, Benham G. An examination of the equivalency of self-report measures obtained from crowdsourced versus undergraduate student samples. Behav Res Methods. 2016 doi: 10.3758/s13428-016-0710-8. [DOI] [PubMed] [Google Scholar]
- 13.Amir O, Rand DG, Gal YK. Economic games on the internet: the effect of $1 stakes. PLoS One. 2012;7:e31461. doi: 10.1371/journal.pone.0031461. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Carroll AE, Downs SM. Improving decision analyses: parent preferences (utility values) for pediatric health outcomes. J Pediatr. 2009;155:21. doi: 10.1016/j.jpeds.2009.01.040. [DOI] [PubMed] [Google Scholar]
- 15.Hsieh MH, Swana HS, Baskin LS, et al. Cost-utility analysis of treatment algorithms for moderate grade vesicoureteral reflux using Markov models. J Urol. 2007;177:703. doi: 10.1016/j.juro.2006.10.002. [DOI] [PubMed] [Google Scholar]
- 16.Bleichrodt H. A new explanation for the difference between time trade-off utilities and standard gamble utilities. Health Econ. 2002;11:447. doi: 10.1002/hec.688. [DOI] [PubMed] [Google Scholar]
- 17.Groll D. Four models of family interests. Pediatrics. 2014;134(Suppl 2):S81. doi: 10.1542/peds.2014-1394C. [DOI] [PubMed] [Google Scholar]
- 18.Hellinger FJ. Expected utility theory and risky choices with health outcomes. Med Care. 1989;27:273. doi: 10.1097/00005650-198903000-00005. [DOI] [PubMed] [Google Scholar]
- 19.Lipstein EA, Britto MT. Evolution of Pediatric Chronic Disease Treatment Decisions: A Qualitative, Longitudinal View of Parents' Decision-Making Process. Med Decis Making. 2015 doi: 10.1177/0272989X15581805. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Stasio SD, Yang C, Brietzke SE, et al. Tonsillectomy versus tonsillotomy: a study of parental preference. Int J Pediatr Otorhinolaryngol. 2015;79:359. doi: 10.1016/j.ijporl.2014.12.026. [DOI] [PubMed] [Google Scholar]
- 21.Wittenberg E, Prosser LA. Disutility of illness for caregivers and families: a systematic review of the literature. Pharmacoeconomics. 2013;31:489. doi: 10.1007/s40273-013-0040-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Brody JL, Annett RD, Scherer DG, et al. Comparisons of adolescent and parent willingness to participate in minimal and above-minimal risk pediatric asthma research protocols. J Adolesc Health. 2005;37:229. doi: 10.1016/j.jadohealth.2004.09.026. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Paine AM, Allen LA, Thompson JS, et al. Anchoring in Destination Therapy Left Ventricular Assist Device Decision Making: A Mechanical Turk Survey. J Card Fail. 2016 doi: 10.1016/j.cardfail.2016.04.018. [DOI] [PubMed] [Google Scholar]
- 24.Simons DJ, Chabris CF. Common (mis)beliefs about memory: a replication and comparison of telephone and Mechanical Turk survey methods. PLoS One. 2012;7:e51876. doi: 10.1371/journal.pone.0051876. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Sprouse J. A validation of Amazon Mechanical Turk for the collection of acceptability judgments in linguistic theory. Behav Res Methods. 2011;43:155. doi: 10.3758/s13428-010-0039-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Buhrmester M, Kwang T, Gosling SD. Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data? Perspect Psychol Sci. 2011;6:3. doi: 10.1177/1745691610393980. [DOI] [PubMed] [Google Scholar]
- 27.Gold MR, Siegel JE, Russell LB, et al. Cost-effectiveness in Health and Medicine. New York: Oxford University Press; 1996. [Google Scholar]
- 28.Matza LS, Sapra SJ, Dillon JF, et al. Health state utilities associated with attributes of treatments for hepatitis C. Eur J Health Econ. 2014 doi: 10.1007/s10198-014-0649-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.