Methodological Comparison of Mapping the Expanded Prostate Cancer Index Composite to EuroQoL-5D-3L Using Cross-Sectional and Longitudinal Data: Secondary Analysis of NRG/RTOG 0415

PURPOSE

To compare the predictive ability of mapping algorithms derived using cross-sectional and longitudinal data.

METHODS

This methodological assessment used data from a randomized controlled noninferiority trial of patients with low-risk prostate cancer, conducted by NRG Oncology (ClinicalTrials.gov identifier: NCT00331773), which examined the efficacy of conventional schedule versus hypofractionated radiation therapy (three-dimensional conformal external beam radiation therapy/IMRT). Health-related quality-of-life data were collected using the Expanded Prostate Cancer Index Composite (EPIC), and health utilities were obtained using EuroQOL-5D-3L (EQ-5D) at baseline and 6, 12, 24, and 60 months postintervention. Mapping algorithms were estimated using ordinary least squares regression models through five-fold cross-validation in baseline cross-sectional data and combined longitudinal data from all assessment periods; random effects specifications were also estimated in longitudinal data. Predictive performance was compared using root mean square error. Longitudinal predictive ability of models obtained using baseline data was examined using mean absolute differences in the reported and predicted utilities.

RESULTS

A total of 267 (and 199) patients in the estimation sample had complete EQ-5D and EPIC domain (and subdomain) data at baseline and at all subsequent assessments. Ordinary least squares models using combined data showed better predictive ability (lowest root mean square error) in the validation phase for algorithms with EPIC domain/subdomain data alone, whereas models using baseline data outperformed other specifications in the validation phase when patient covariates were also modeled. The mean absolute differences were lower for models using EPIC subdomain data compared with EPIC domain data and generally decreased as the time of assessment increased.

CONCLUSION

Overall, mapping algorithms obtained using baseline cross-sectional data showed the best predictive performance. Furthermore, these models demonstrated satisfactory longitudinal predictive ability.

INTRODUCTION

Health technology appraisals provide health care payers with the necessary evidence on value for cost of new interventions to help them make regulatory and reimbursement decisions.^1-3 The assessment of health-related quality-of-life (HRQoL) information is critical in these evaluations.⁴ Of particular importance are preference-based measures (PBMs) such as EuroQoL-5D-3L (EQ-5D) that capture patient preferences for different health states in terms of health utilities.⁵ Clinical studies designed to assess the effectiveness of health technologies increasingly include outcome measures capable of producing health utility values to calculate quality-adjusted life years.^6,7 However, to maximize survey response and completion rate and minimize patient burden, historically conducted trials have often only included a disease-specific patient-reported outcome measure (PROM) that is sensitive to clinically relevant changes.⁵ Mapping or cross-walking allows incorporation of such evidence in economic evaluation of health care interventions by establishing a link between the PROM and PBM, such that health utilities corresponding to the health states captured by the PROM can be derived.^2,3

CONTEXT

• Key Objective

• This study compares the predictive ability of mapping algorithms obtained using cross-sectional and longitudinal data from a large randomized clinical trial and examines the longitudinal predictive ability of algorithms obtained using baseline cross-sectional data.

• Knowledge Generated

• The study findings can help future researchers undertaking a mapping study to choose appropriate study designs and statistical approaches on the basis of the available data sources.

• Relevance

• Although mapping studies have historically used both cross-sectional and longitudinal data sources in estimating the mapping algorithms, little is known about the effect of choice of these data on the predictive ability of the resulting algorithms.

Mapping techniques to obtain health utilities have become increasingly popular in economic evaluations of health technologies. However, the methodologies used in these studies vary substantially, introducing variability in the resulting cost-effectiveness estimates.⁸ In an effort to make the mapping process consistent across disease areas and interventions, an International Society for Pharmacoeconomics and Outcomes Research (ISPOR) task force on mapping health utilities was established in 2014.² In addition, Longworth and Rowen reviewed the mapping literature to provide guidance on best practices in conducting a mapping exercise for National Institute for Health and Care Excellence (NICE) health technology assessments.^9,10 The guidance on mapping issued by ISPOR and NICE is lacking on any recommendations on the study design (cross-sectional v longitudinal) to use when generating these prediction models. The choice of data and the way it is modeled may affect the resulting mapping algorithm and subsequently, the cost-effectiveness estimates obtained from its implementation. Therefore, the current study aims to compare the predictive ability of mapping algorithms derived using cross-sectional and longitudinal data from an international, multicenter, randomized controlled trial of patients with low-risk prostate cancer (PC) conducted by NRG Oncology (NRG/RTOG 0415).¹¹ A majority of mapping studies have used data from clinical trials, more often using the baseline (pretreatment) data, in estimating mapping functions. It is important to investigate if the mapping algorithms estimated using baseline, pretreatment data are sensitive to the treatment effect.¹² A secondary objective of this study was to examine the longitudinal predictive ability of mapping algorithms obtained using baseline data, in postintervention data.

METHODS

Data Source

This study used data from a previously published international, multicenter, open-label randomized clinical trial of patients with low-risk PC, conducted by NRG Oncology (ClinicalTrials.gov identifier: NCT00331773).¹¹ This clinical trial used a noninferiority design to determine whether the efficacy of hypofractionated radiation therapy (three-dimensional conformal external beam radiation therapy/intensity-modulated radiation therapy; 70 Gy in 28 fractions over 5.6 weeks) in terms of disease-free survival is not worse than a conventional schedule (73.8 Gy in 41 fractions over 8.2 weeks) in men with low-risk PC. The study population composed of favorable-risk patients with histologically confirmed prostate adenocarcinoma, defined as clinical stage T1-2c (American Joint Committee on Cancer sixth edition), pretreatment prostate-specific antigen (PSA) < 10 ng/mL, and Gleason score < 7, with no radical surgery or cryosurgery for PC nor prior or planned androgen deprivation or bilateral orchiectomy. A total of 1,092 men age > 18 years with prostate adenocarcinoma met the inclusion criteria of the study. Of these, 962 patients consented to HRQoL collection and were considered for inclusion in the analysis. HRQoL was collected at baseline and at 6, 12, 24, and 60 months postintervention.

Sample Selection

The analyzable population from the trial (N = 1,092) was randomly split at 70:30, and the 70% random sample was used to create the estimation cohort, whereas the 30% sample was used to create the validation cohort. For inclusion in the estimation cohort, patients in the 70% sample were required to have consented to provide HRQoL data and have complete information on EQ-5D utilities and Expanded Prostate Cancer Index Composite (EPIC) at baseline and at subsequent assessments at 6-, 12-, and 24-months postintervention. Patients who consented to provide HRQoL data and had complete data on EQ-5D utilities and EPIC at baseline in the remainder 30% sample comprised the validation cohort. Baseline demographic characteristics and clinical covariates were also extracted.

Outcome Measures

EuroQol-5D-3L.

EuroQoL-5D-3L, more commonly called EQ-5D, is a generic PBM that measures health as a function of five dimensions (mobility, self-care, usual activities, pain/discomfort, and anxiety/depression) and overall self-reported health.⁹ The respondent can be at one of three severity levels for these dimensions (no problems, some or moderate problems, and extreme problems). A combination of responses for each domain results in 243 unique health states (3⁵ combinations). Two other health states that are not part of this descriptive system are included in the valuation system: unconsciousness and death.¹³ A score, known as a tariff, is attached to each of these health states on the basis of an analysis of preference data obtained from the general population and represents a patient's preference for a given health state. This preference score ranges from 0 to 1, with 0 representing death (worst health) and 1 representing perfect health. Negative utilities for health states worse than death are also assigned.¹⁴ A visual analog scale that measures the respondent's self-rated health on a scale of 0-100 is also used alongside the EQ-5D descriptive system.

EPIC.

The EPIC questionnaire comprehensively evaluates patient function and bother, after PC treatment, and is validated in men with localized PC receiving various treatment options or surgery.¹⁵ It constitutes four summary domains: urinary, bowel, sexual, and hormonal. Each summary domain has a measurable function and bother subscale. In addition, the urinary domain has two distinct incontinence and irritative/obstructive subscales.¹⁶ Responses for each item on EPIC are recorded on a Likert scale, and the scale scores are linearly transformed to a 0-100 scale, with higher scores representing better HRQoL.

Model Development

This methodological assessment compared the predictive ability of mapping algorithms derived using cross-sectional data and longitudinal data. The variables that informed the mapping algorithms included patient demographics (age and race), clinical characteristics (Zubrod performance status and baseline PSA levels), EPIC domains/subdomains, and EQ-5D index scores. EPIC was previously mapped to EQ-5D using baseline data from NRG/RTOG 0415, where several functional forms were tested including linear (ordinary least squares [OLS]) regression, which is the most common approach to derive the mapping function. To account for the anticipated bimodal distribution of EQ-5D for the study population, Tobit and two-part models were estimated, as they account for a significant proportion of patients in full health. The Tobit model assumes that the EQ-5D utility data are censored at 1 and that the true value has a normal distribution whose mean is given by a linear combination of the covariates. Two-part models model the probability of being in full health using a logistic regression and then model the remainder of the distribution using the OLS regression model. In that study, OLS models outperformed all other model types in terms of their predictive ability.¹⁷ Therefore, the current study, which uses data from the same trial (RTOG 0415), used OLS regression models to develop mapping algorithms in baseline cross-sectional data and combined longitudinal data from all assessment periods in the estimation sample (see Appendix Table A1 for tested model specifications). Random effects (RE) specifications that explicitly model the longitudinal nature of the data were also estimated to obtain more precise predictions.¹⁸

Five-fold cross-validation was used for estimation and internal validation.^19,20 In five-fold cross-validation, the data are split into five equal parts and the model is fitted on four parts, with the fifth being held out for validation. The fitted model of the four selected parts is used to compute the predicted residual sum of squares on the fifth omitted part, and this process is repeated for each of the five parts. The sum of the five predicted residual sums of squares is obtained for each fitted model and is the estimate of the prediction error. Indices such as the absolute mean of the residuals or errors and square root of the mean of the residual sum of squares, also known as root mean square error (RMSE), are used to determine model performance. RMSE, a measure of individual prediction error, attaches relatively higher weights to large errors, making it an ideal metric when large errors are undesirable. In this study, candidate models were selected on the basis of root mean square error (RMSE); lower RMSE corresponds to higher predictive ability.² In the absence of an external data set to perform external validation, validation was performed by scoring the baseline data from the 30% sample using the coefficients from the candidate mapping algorithms identified using the estimation sample.

In addition, reduced models were estimated in baseline data using stepwise regression to identify parsimonious models with high predictive ability. The longitudinal predictive ability of the candidate algorithms identified using baseline data (full models and reduced models) was tested in postintervention data. Predicted and observed utilities were compared using paired sample t-tests, and mean absolute differences (MDs) were reported; lower MD indicates better predictive performance. The application of the candidate mapping algorithms in postintervention data (obtained using EPIC domain and subdomain data, respectively) generated a prediction error per patient for each assessment postintervention. To identify factors associated with prediction errors when mapping EPIC to EQ-5D utilities, the absolute prediction error was modeled using fixed effects, as a function of baseline demographic and clinical covariates, EPIC domain/subdomain scores, and observed and predicted baseline EQ-5D utilities.

RESULTS

The study cohort included patients consenting to HRQoL collection who had complete data on EPIC domains/subdomains and EQ-5D dimensions at baseline and all subsequent assessment periods. For models with EPIC domains, 267 patients comprised the estimation cohort and 232 patients comprised the validation cohort. For models with EPIC subdomains, 199 patients comprised the estimation cohort and 213 patients comprised the validation cohort. Table 1 shows the baseline demographic and clinical characteristics of the estimation cohorts for models using the EPIC domain and subdomain data. Overall, the distribution of EQ-5D scores was skewed left, with more than 50% of the patients in each cohort in full health. The plot of the distribution of EQ-5D utilities (Fig 1) was bimodal with peaks at full health and 0.8 (health state with mild severity). The mean EPIC domain and subdomain scores and EQ-5D scores for all study time points are summarized in Table 2.

TABLE 1.

Baseline Characteristics of Patients With Complete EPIC Domain and Subdomain Data

FIG 1.

Distribution of EQ-5D scores in patients with complete data on EPIC domains and subdomains at baseline. EPIC, Expanded Prostate Cancer Index Composite; EQ-5D, EuroQOL-5D-3L.

TABLE 2.

EPIC Domain and Subdomain Scores and EQ-5D Scores at All Study Time Points

Mapping Using Cross-Sectional Versus Longitudinal Data

OLS models were estimated in the estimation cohort for all the model specifications listed in Appendix Table A1, using cross-sectional data from baseline assessment and the combined (longitudinal) data from all assessment periods. To explicitly model the longitudinal nature of the data, RE models were also estimated. Candidate OLS models using combined data outperformed the candidate RE models and the OLS models using baseline cross-sectional data when only EPIC domains or subdomains were modeled (model specifications 1 and 2; Table 3). For all subsequent model specifications (3-6; Table 3), candidate OLS models using baseline cross-sectional data outperformed the candidate OLS models using combined longitudinal data and the RE models. In the estimation sample, the best performing model was an OLS model using combined data with EPIC subdomains, age, race, Zubrod status, and PSA (model 6l). However, when the candidate algorithms were tested in the validation sample, the OLS model using combined data with EPIC subdomains (model 2c) outperformed all other model specifications. When patient covariates were incorporated in the estimation of the mapping algorithms, OLS models using baseline cross-sectional data outperformed the model specifications using longitudinal data in the validation phase; RE models consistently performed poorly across all six model specifications.

TABLE 3.

Model Performance in Five-Fold Cross-Validation

Longitudinal Predictive Performance

Table 4 summarizes the mean observed EQ-5D utilities at each study time point postintervention (6, 12, and 24 months), the mean predicted EQ-5D utilities obtained by testing the candidate algorithms derived from baseline data in postintervention data, and the MDs between the observed and predicted utilities for each tested model specification (full models and reduced models). MDs between reported and predicted utilities were lower for models using EPIC subdomain data compared with EPIC domain data and generally decreased as the time of assessment increased.

TABLE 4.

Observed Versus Predicted EQ-5D Utilities

Factors Influencing Prediction Errors

To identify factors influencing the prediction errors when mapping EPIC to EQ-5D utilities, the absolute prediction error generated from scoring the postintervention data using the candidate algorithms was modeled using fixed effects, as a function of baseline demographic and clinical covariates, EPIC domain/subdomain scores, and observed and predicted EQ-5D utilities from the baseline (estimation) data. According to the fixed effects model for EPIC domain data, lower observed and predicted baseline EQ-5D scores and time of assessment were significant predictors of the absolute prediction error (Table 5). For EPIC subdomain data, lower observed and predicted baseline EQ-5D scores, hormonal bother and function, and bowel function significantly predicted the absolute prediction error (Table 5).

TABLE 5.

Factors Associated With Absolute Prediction Error

DISCUSSION

This methodological assessment used OLS models to estimate the mapping algorithms using cross-sectional and longitudinal study designs. In the estimation phase, OLS models using combined longitudinal data from all assessment periods marginally outperformed the candidate OLS models using baseline data and the RE models, for all six model specification groups. In the validation, similar results were seen when only EPIC domain or subdomain data were modeled. When demographic or clinical covariates were added to the models, however, OLS models using baseline data demonstrated the best predictive ability. Although RE models performed well in the estimation phase, with the RMSE values marginally higher than the best performing model type, they exhibited very high RMSE values in the validation, when covariates were modeled. This suggests a poor model fit for RE models. OLS models using baseline data outperformed other model specifications overall in our study. It is, however, important to note that studies mapping a different instrument or conducted in a different population might have different findings.

Mapping algorithms have been mostly derived using the regression framework in baseline and pretreatment data from clinical trials.¹² It is important to examine the longitudinal validity of these algorithms to determine if the mapping algorithms are sensitive to treatment effects and whether they can be implemented with confidence in data sets where patients might or might not have received treatment. Kontodimopoulos et al¹² previously explored this question and examined the longitudinal validity of the Modified Health Assessment Questionnaire in a rheumatoid arthritis population. To our knowledge, the current study is the first to examine and demonstrate the longitudinal validity of mapping algorithms derived from the EPIC questionnaire.

Mapping a PBM to a disease-specific PROM generates a set of prediction errors, which reflect the difference between the observed and predicted utilities. Consistent with findings from previous mapping studies, the predicted utilities were underestimated for patients with milder health states and overestimated for those with more severe health states.²¹ Lower absolute prediction errors indicate better predictive performance. Kontodimopoulos et al¹² found that the MD in observed versus predicted utilities (prediction errors) in the postintervention samples in their study typically exceeded 0.03, which is a commonly reported minimal clinically important difference threshold for EQ-5D utilities. This indicates poor predictive performance of their algorithm in longitudinal data. However, the candidate algorithms using EPIC subdomains in our study were found to have a MD of 0.03 or lower at each assessment period, increasing confidence in their longitudinal validity. In an effort to identify drivers of the prediction error, this study modeled the absolute prediction error as a conventional linear function of observed and predicted baseline EQ-5D values, EPIC domain/subdomain scores, and patient demographics and clinical covariates. Observed and predicted baseline EQ-5D scores, time of assessment, hormonal function, and hormonal and bowel bother were found to significantly influence prediction errors.

The current study required complete information on EPIC domain and subdomain scores and EQ-5D utilities at all assessment periods, which resulted in a relatively small sample size. Larger sample sizes are desirable to detect minimally important effect sizes and identify statistically significant associations in multiple regression models.^22,23 However, the results from the OLS models using baseline data in this study are consistent with the findings from our previous mapping study with a larger sample size, suggesting that the reduction in sample size did not affect the performance of the estimated models, increasing the confidence in the overall findings of this study.¹⁷ Consistent with that study, the baseline OLS model with EPIC subdomains, patient demographics, and clinical characteristics outperformed candidate models from all other baseline OLS model specification groups.

Although the sample size was sufficient for the analyses conducted in this study, differences may exist between patients who completed the questionnaires in the trial and those who did not.²⁴ Consequently, the study findings may not be generalizable to the entire population. As the goal of the regression models in this study was prediction, and not estimation, we expected that removing the patients with missing data from the sample would not affect the predictive ability of the resulting models. A formal comparison of the characteristics of the responders and nonresponders was not within the scope of our investigation, and we acknowledge this as a limitation. However, the characteristics of the selected sample were consistent with the overall trial population; this increases our confidence in the generalizability of our study results.¹¹

To the best of our knowledge, this is the first study to formally investigate the impact of cross-sectional versus longitudinal study design on the predictive ability of mapping algorithms. EQ-5D utilities were modeled as a conventional linear function of EPIC domains/subdomains and covariates. As the end goal was prediction and not estimation, the OLS model was expected to provide unbiased predictions even with combined longitudinal data. However, to improve the predictive performance, this study also explicitly modeled the longitudinal nature of the data.¹⁸ The clinical trial data used in this study collected EPIC and EQ-5D data for each individual at various time points. The estimates of the relationship between these measures are conditional on the subjects and therefore cannot be generalizable to other subjects, making the choice of fixed effects specification inappropriate.¹⁸ A RE specification was therefore chosen, as it decomposes the error term into a subject- and measurement-specific error and a subject-specific, time-invariant error.¹⁸ Another important takeaway from this study is that OLS regression tends to perform better than theoretically more robust procedures, a finding consistent with our previous study and many other mapping studies in the literature.^3,14,25

In conclusion, to our knowledge, this study is the first to examine the effect of the type of data source (cross-sectional v longitudinal) on the predictive ability of the resulting algorithms. These findings can help future researchers undertaking a mapping study to choose appropriate study designs and statistical approaches on the basis of the available data sources. To our knowledge, this is also the first study to demonstrate the longitudinal validity of EPIC questionnaire, using data from a large randomized clinical trial, and builds upon existing research on longitudinal validity of mapping functions. Overall, the mapping algorithms derived from baseline data predicted utilities similar to the observed utilities in the postintervention data. The low MDs in prediction errors in this study demonstrate satisfactory performance of mapping functions in the longitudinal data, thereby increasing confidence in their use in economic evaluations in PC. Further testing in different data sets across various disease areas is necessary to corroborate our results and increase confidence in the study findings.

APPENDIX

TABLE A1.

Model Specifications

AUTHOR CONTRIBUTIONS

Conception and design: Rahul Khairnar, Howard M. Sandler, W. Robert Lee, C. Daniel Mullins, Francis B. Palumbo, Deborah W. Bruner, Fadia T. Shaya, Jeff M. Michalski, Mark V. Mishra

Provision of study materials or patients: W. Robert Lee, Amit B. Shah, Shawn Malone, Jeff M. Michalski, Ian S. Dayes, Samantha A. Seaward, Thomas M. Pisansky

Collection and assembly of data: Rahul Khairnar, Howard M. Sandler, Francis B. Palumbo, Ian S. Dayes, Samantha A. Seaward, Adam D. Currey, Thomas M. Pisansky, Yuhchyau Chen, Eric M. Horwitz, Mark V. Mishra

Data analysis and interpretation: Rahul Khairnar, Lyudmila DeMora, W. Robert Lee, Ester Villalonga Olives, Francis B. Palumbo, Fadia T. Shaya, Soren M. Bentzen, Amit B. Shah, Shawn Malone, Jeff M. Michalski, Ian S. Dayes, Michele Albert, Adam D. Currey, Thomas M. Pisansky, Eric M. Horwitz, Albert S. DeNittis, Felix Feng, Mark V. Mishra

Manuscript writing: All authors

Final approval of manuscript: All authors

Accountable for all aspects of the work: All authors

AUTHORS' DISCLOSURES OF POTENTIAL CONFLICTS OF INTEREST

The following represents disclosure information provided by authors of this manuscript. All relationships are considered compensated unless otherwise noted. Relationships are self-held unless noted. I = Immediate Family Member, Inst = My Institution. Relationships may not relate to the subject matter of this manuscript. For more information about ASCO's conflict of interest policy, please refer to www.asco.org/rwc or ascopubs.org/cci/author-center.

Open Payments is a public database containing information reported by companies about payments made to US-licensed physicians (Open Payments).