Negligible Effects of the Survey Modes for Patient-Reported Outcomes: A Report From the Childhood Cancer Survivor Study

Abstract

PURPOSE

This study compared the measurement properties for multiple modes of survey administration, including postal mail, telephone interview, and Web-based completion of patient-reported outcomes (PROs) among survivors of childhood cancer.

METHODS

The population included 6,974 adult survivors of childhood cancer in the Childhood Cancer Survivor Study who completed the Brief Symptom Inventory-18 (BSI-18), which measured anxiety, depression, and somatization symptoms. Scale reliability, construct validity, and known-groups validity related to health status were tested for each mode of completion. The multiple indicators and multiple causes technique was used to identify differential item functioning (DIF) for the BSI-18 items that responded through a specific survey mode. The impact of the administration mode was tested by comparing differences in BSI-18 scores between the modes accounting for DIF effects.

RESULTS

Of the respondents, 58%, 27%, and 15% completed postal mail, Web-based, and telephone surveys, respectively. Survivors who were male; had lower education, lower household income, or poorer health status; or were treated with cranial radiotherapy were more likely to complete a telephone-based survey compared with either a postal mail or Web-based survey (all P < .05). Scale reliability and validity were equivalent across the 3 survey options. One, 2, and 5 items from the anxiety, depression, and somatization domains, respectively, were identified as having significant DIF among survivors who responded by telephone (P < .05). However, estimated BSI-18 domain scores, especially depression and anxiety, between modes did not differ after accounting for DIF effects.

CONCLUSION

Certain survivor characteristics were associated with choosing a specific mode for PRO survey completion. However, measurement properties among these modes were equivalent, and the impact of using a specific mode on scores was minimal.

INTRODUCTION

For the past 4 decades, the 5-year survival rate of pediatric cancers in the United States has improved substantially, from 58% in the 1970s to 83% in the 2010s.^1,2 However, compared with the general population, the growing population of survivors of childhood cancer has an increased risk for late mortality, chronic health conditions (CHCs), high symptom prevalence, and poor quality of life.^3-8 Assessing patient-reported outcomes (PROs) on a regular basis may assist clinicians in identifying potential adverse health events for early interventions.⁹

Multiple survey methods are well established for PRO assessment. The Childhood Cancer Survivor Study (CCSS),¹⁰ one of the largest childhood cancer survivor cohorts worldwide, has used Web-based surveys alongside paper-and-pencil postal mail surveys and telephone interviews to collect PROs. The Web-based method has advantages of decreasing administration cost,^11-13 increasing response rate,^11,14 improving data entry errors,^15,16 and providing real-time score calculations.^12,17 Evidence suggests that survivors of younger ages were quicker to adopt an electronic or mobile health (mHealth) platform to receive health information and complete surveys,^18-20 whereas survivors of older ages or low socioeconomic status (SES) might use postal mail and telephone options.^21,22

Despite the advantages of multiple survey methods, using these approaches to assess PROs is not without concerns. In fact, each option applies a unique format to display PRO questions and depends on a distinctive physical or social context to collect data. In addition, different psychological interpretations may have been used by individuals to answer the questions, even if the same wording or content was included. In contrast to the postal mail option, participants of telephone or in-person interviews were likely to take social norms or personal privacy matters into consideration, potentially leading to social desirability bias.^23,24 However, this bias may have been less significant when a computer-based or paper-and-pencil method was used.²³ Thus, it was critical to test whether comparable quality was achieved across the survey modes.

Ideally, PROs collected from multiple modes should hold similar measurement properties.²⁵ Several methods were available to evaluate the quality of PRO surveys collected from different modes, including missing data and scale reliability, validity, and accuracy (eg, estimated PRO scores conditioned on mode effects). Differential item functioning (DIF) is a technique to test whether the measurement property of a PRO item performs equivalent given the same level of a PRO domain.²⁶ If properties across modes vary appreciably, PRO data collected through different modes should not be merged for analyses.²⁷

Using data collected from the CCSS, we examined the characteristics of survivors who completed a PRO survey through a postal mail, telephone interview, or Web-based option. We hypothesized that survivors who were male, had lower SES or poorer health status, or were treated with radiotherapy would be more likely to complete the PRO by telephone; in contrast, those who had higher SES, had better health status, or had not received radiotherapy would be more likely to use a postal mail or Web-based option. We then tested measurement properties of the survey for each mode and identified DIF items for a specific mode. We hypothesized that more DIF items would be related to telephone interviews in contrast to other modes. Finally, we tested the impact of modes on PRO score calculation with and without accounting for DIF effects related to individual modes.

METHODS

Participants and Data Collection

Study participants were adult survivors of childhood cancer from the baseline of the CCSS Expansion Cohort who had been treated for pediatric cancers or malignancies at 31 institutions in the United States and Canada between 1987 and 1999.⁵ The criteria for CCSS Expansion Cohort participation included the following: individuals who had been diagnosed with cancers or malignancies before the age of 21 years and were 5-year survivors at the time of CCSS enrollment.¹⁰ As of March 2015, 10,004 survivors consented to CCSS Expansion Cohort participation. However, we excluded 3,030 participants who were < 18 years old at the time of the baseline survey, died before the baseline survey, and/or did not self-report PROs, leaving 6,974 participants for the analysis (Appendix Fig A1).

Three mode options (postal mail, telephone interview, or Web based) were used to collect PRO data. We sent a package to eligible participants via postal mail that included a survey hardcopy and a Web link protected with a unique username and password. Eligible participants were asked to complete the survey via either paper hardcopy or Web site, per personal preference. If the survey was not returned within 3 weeks, we used multiple methods (postal letters, e-mails, and telephone calls) to remind survivors to complete the survey, followed by telephone interview to collect PRO data 3 additional weeks after the reminders. The CCSS was approved by the institutional review board at each of the 31 participating institutions; all participants provided informed consent.

PRO Measurement

The Brief Symptom Inventory-18 (BSI-18) was used to measure emotional distress during the past 7 days.²⁸ BSI-18 includes the domains of anxiety, depression, and somatization with 6 items on each domain. Distress on each item was rated using a 5‐point Likert scale (not at all, a little bit, moderately, quite a bit, and extremely), with higher scores indicating more distress. The same content of the BSI-18 was included in all 3 modes of survey administration.

Sociodemographic, Clinical, and Health Outcome Variables

Sociodemographic (eg, age at survey completion, sex, race and ethnicity, education, annual household income) and health outcome (CHCs and general health status) data were collected from the survey. For CHCs, consistent with previous CCSS studies,⁴ a total of 137 conditions were categorized, and each condition was graded as mild (grade 1), moderate (grade 2), severe or disabling (grade 3), or life-threatening or disabling (grade 4) per the Common Terminology Criteria for Adverse Events (version 4.03). Perceived general health status over the past 4 weeks was collected through a single item using a 5‐point Likert scale (excellent, very good, good, fair, or poor). In addition, outpatient visits and hospitalizations in the past 12 months and lifestyle variables (cigarette smoking and exercise) were collected from the survey. Childhood cancer diagnosis and treatment data (chemotherapy, radiotherapy, and surgery) were abstracted from medical records at each treating institution.^29,30

Statistical Analyses

Percentages or means (standard deviations [SDs]) of individual sociodemographic, clinical, and health outcome variables were reported by mode of administration. Analysis of variance tests for continuous variables and χ² tests for categorical variables were performed to examine differences across modes. The percentage of nonresponse to each BSI-18 item across modes was compared. Per significant variables identified from bivariate analysis and hypotheses stated in the Introduction, multivariable multinomial logistic regression was performed to identify sociodemographic, clinical, and health outcome variables that were associated with completing the telephone interview or Web-based mode compare with the postal mode (as the reference).

For each mode, standard psychometric properties for individual BSI-18 domains were evaluated. A Cronbach’s α coefficient was estimated to evaluate internal consistency, with a value ≥ 0.7 deemed acceptable. Confirmatory factor analysis (CFA) was performed to test the degree to which the underlying structure of the BSI-18 domains was preserved. The following 2 model fit indices were used to determine satisfactory construct validity: the root-mean-square error of approximation (RMSEA) < 0.6 and the comparative fit index (CFI) ≥ 0.9. As a standard approach, known-groups validity was tested through the relative validity metric,³¹ defined as an F value (squared t value) of BSI-18 domain scores between levels of overall health status on a mode divided by an F value estimated by the reference mode (the one with the lowest F value). A lower value represents poorer known-groups validity for one mode versus the reference mode.

The multiple indicators and multiple causes (MIMIC) technique was used to identify DIF items associated with the selection of a specific mode^32,33 by incorporating background variables (age, sex, race or ethnicity, educational attainment, treatment, years since diagnosis, and general health status). The MIMIC model comprised a measurement component representing the relations between a latent variable (a BSI-18 domain) and its indicators (items measuring a BSI-18 domain) and a structural component specifying the relationship among a latent variable and survey modes (Fig 1).³⁴ Theoretically, relationships of survey modes with individual BSI-18 items were explained when the corresponding BSI-18 domain was included in the modeling. However, a DIF phenomenon emerged when relationships between modes and individual items were statistically significant (ie, a dotted line in Fig 1), suggesting that survivors might under- or overrate an item of the BSI-18 if a specific mode was used, even though they possess the same level of psychological distress. We used a 2-step approach to evaluate items with DIF.^35,36 For a BSI-18 domain, step 1 constrained the relationship between modes and all items under a domain to be 0 and examined the modification indices to suggest how much the model fit would be improved if specific relationships were freely estimated; step 2 began with the item having the largest and most significant modification index and added items one at a time to the model for freely estimating the relationship with mode of administration until no modification indices were > 3.84 (df = 1). Finally, we tested the impact of DIF by comparing differences in domain scores across the 3 modes when including versus not including the DIF effects in the analyses.

FIG 1.

A conceptual model for testing differential item function related to the modes of survey administration. η, correlations between covariates; γ, associations of covariates with individual Brief Symptom Inventory-18 (BSI-18) domains; β, associations of the mode administration and individual items conditioning on underlying BSI-18 domain score (ie, differential item functioning effect); α, item factor loading; ζ, item thresholds.

Mplus 8.0 was used to conduct CFA and DIF analyses, and SAS version 9.4 (SAS Institute, Cary, NC) was used for the remaining analyses. All tests were 2-sided, and P < .05 was considered statistically significant.

RESULTS

Characteristics of Study Participants

Of the 6,974 survivors included in this study, 58% used the postal mail option, 27% used the Web-based option, and 15% used the telephone option. Mean age at survey completion and time since cancer diagnosis were 27.6 years (SD, 5.7 years) and 18.1 years (SD, 3.8 years), respectively; 50.5% of participants were women, 76.4% were non-Hispanic, and 39.9% earned a college or postgraduate degree. Approximately 25% of survivors were treated for leukemia, 25% for lymphoma, 22% for brain malignancy, and 28% for solid or other malignant tumors (Table 1).

TABLE 1.

Participant Characteristics Across 3 Modes of Survey Administration

Participant Characteristics Associated With the Use of a Specific Mode

The distribution of participant characteristics across the 3 modes of survey administration was unequal based on bivariate analyses. Overall, participants who were male (P < .001); had lower educational attainment (P < .001), lower annual household income (P < .001), higher severity grades of CHCs (P < .001), or poorer self-reported general health status (P < .001); or had been treated with cranial radiotherapy (P < .001) were more likely to complete a survey by telephone versus the other administration modes (Table 1).

Using postal mail as the reference group, multivariable analyses suggest that telephone interview completion was associated with being male (odds ratio [OR], 1.91; 95% CI, 1.60 to 2.29), being non-Hispanic black (OR, 2.46; 95% CI, 1.85 to 3.27), and having poorer self-reported general health status (OR, 2.12; 95% CI, 1.15 to 3.90). In contrast, Web-based completion was associated with being female (OR, 1.49; 95% CI, 1.31 to 1.70), being Hispanic (OR, 1.75; 95% CI, 1.42 to 2.16), having more years since cancer diagnosis (OR, 1.06; 95% CI, 1.04 to 1.08), having college or postgraduate education (OR, 2.80; 95% CI, 2.27 to 3.45), having a higher annual household income (OR, 1.27; 95% CI, 1.08 to 1.49), and not having experienced a second cancer (OR, 1.29; 95% CI, 1.06 to 1.56; Table 2).

TABLE 2.

Factors Associated With the Use of a Specific Mode of Survey Administration: Multivariable Multinomial Logistic Regression

Measurement Properties of the BSI-18 Across Individual Modes of Administration

Table 3 reports the scale reliability, construct validity, and known-groups validity for each domain of the BSI-18. Overall, α coefficients were above the satisfactory threshold (≥ 0.7) across the 3 survey options. Using RMSEA < 0.06 and CFI > 0.90 as criteria, construct validity on the 3 domains was satisfactory across the 3 modes, except for anxiety data collected by telephone interview and Web-based modes. For known-groups validity, compared with the web-based mode, the postal mail option had the highest and the telephone-based option had the lowest relative validity for each BSI-18 domain in differentiating general health status. In addition, missing data at the item level were negligible for all survey modes (Appendix Table A1).

TABLE 3.

Reliability and Construct and Known-Group Validity of BSI-18 Domains Between 3 Modes of Survey Administration

DIF Associated With Modes of Administration

Table 4 lists specific items identified as having DIF related to survey modes for each BSI-18 domain. For the depression domain, 2 DIF items related to the telephone versus postal mail were found. However, the DIF was in the opposite direction for these items (no interest: β = 0.063, P = .001; and blue: β = −0.044, P = .015). For the anxiety domain, 1 DIF item associated with the use of the Web-based mode versus the postal mail mode was identified (fearful: β = 0.072, P < .001). For the somatization domain, 5 DIF items related to the telephone-based mode versus the postal mail mode were identified. Three of the DIF items for somatization were in a positive direction (faintness: β = 0.049, P = .050; numbness: β = 0.107, P < .001; and feeling weak: β = 0.077, P = .002), and 2 items were in a negative direction (nausea: β = −0.085, P < .001; and trouble getting breath: β = −0.072, P = .012). In addition, 1 DIF item associated with the Web-based mode versus the postal mail mode was identified (nausea: β = −0.055, P < .01).

TABLE 4.

DIF Associated With Specific Modes of Survey Administration

Impact of DIF on BSI-18 Domain Score Calculation

Table 5 lists the difference in BSI-18 domain scores between administration modes considering and not considering DIF effects in analyses. Compared with the postal mail option, participants who completed a Web-based survey reported more depression, anxiety, and somatization symptoms, whereas those who completed a telephone-based survey reported fewer symptoms in these domains. However, the estimated differences in depression and anxiety scores between modes were not changed even when DIF effects were included in analyses (all P > .05). In contrast, the difference in somatization scores increased slightly after accounting for DIF effects (all P < .05).

TABLE 5.

Impact of Domain Score Calculation Among Modes of Administration With and Without Adjusting for DIF Effects

DISCUSSION

To our knowledge, this is the first study to evaluate the preference and impact of using different survey modes for PRO assessment among survivors of childhood cancer. We found that the measurement properties of the 3 different modes were equivalent, and the effect of choosing a specific mode toward PRO scoring bias was minimal. We especially elucidated that personal characteristics such as sex, race, self-reported health status, income, and cancer diagnoses were related to mode preference. In contrast to previous mode effect research that primarily reported response rates or estimated correlations of PRO scores between the modes,^37-39 we identified items having DIF related to mode selection and tested the impacts of DIF items on PRO score estimation. Our findings were both novel and highly relevant considering the increasing numbers of cancer research programs that have adopted mHealth platforms^22,40,41 to replace paper-and-pencil or telephone modes for collecting PRO data.

In our study, survivors who were male, had lower educational attainment or household income, had poorer health status, or were treated with cranial radiotherapy were more likely to complete a telephone-based survey, whereas those with higher educational attainment or household income tended to complete a postal mail or Web-based survey. In line with previous adult-onset cancer studies,^42,43 these findings suggest that low proficiency in reading comprehension and limited access to electronic devices or the Internet are barriers to using mHealth modes for PRO assessment among low socioeconomic class survivors.^21,22 Importantly, survivors treated with cranial radiotherapy or those with poorer health status were more likely to engage in a telephone-based PRO survey than the postal mail or Web surveys. Our finding underscores the need to retain the more costly and labor-intensive telephone mode option for survivors who are cognitively impaired or ill.

Compared with the postal mail survey, lower distress scores on all domains were reported by survivors using the telephone mode, whereas higher scores were reported by survivors using the Web-based mode after adjusting for the influence of covariates. In addition, 1, 2, and 5 items from the anxiety, depression, and somatization domains were identified with significant DIF, respectively, and these DIF items were related to the telephone interview. The DIF finding implies that criteria used by survivors for interpreting PRO questions may vary when different modes are involved, which might bias responses and scoring of the same question.^44,45 This phenomenon may be attributable to social desirability or contextual influence.²³ There is evidence that people who speak to interviewers (telephone or in person) often provide socially desirable answers (eg, more positive health status or behavior) versus those who self-complete a survey.⁴⁶ However, lower PRO scores have been reported by studies using a computer-administered mode versus a paper-and-pencil mode.^47,48 In our study, after adjusting for the influence of DIF, differences in estimated scores in the depression and anxiety domains remained similar between the telephone and postal mail modes but were slightly increased in the somatization domain. Therefore, using the telephone and other options interchangeably may not appear to bias the estimation of distress scores.

Poor PROs on symptom assessment may indicate the occurrence of late medical effects, which is the main reason that survivors of cancer seek health care services.^49,50 Ideally, survivorship care for pediatric cancers should collect PRO data on a regular basis to facilitate clinical decision making.^9,51 Our study has far-reaching implications for improving PRO assessment. On the basis of the equivalent and satisfactory measurement properties over 3 modes and the minimal DIF effects, the CCSS and other pediatric cancer survivor studies can safely transfer the postal mail survey to the Web-based option if the same wording and response options of PRO measures are used. Survivors should be directed to use either the postal mail or Web-based mode to complete a survey per their preference to increase the response rate and decrease missing PRO data. As an alternative, survivors may use the telephone option with a standardized procedure to minimize the DIF effects. Undoubtedly, mode effect research for PROs will remain relevant with the development of novel technologies (eg, virtual reality devices, artificial intelligence speakers, and smart mirrors).^52,53

Several limitations should be considered when interpreting our findings. First, the results may not be generalizable to all adult survivors of childhood cancer because of the relatively young age of participants (mean age, 27.6 years). Second, in this observational study, each participant merely completed a single survey mode and the 3 options were given in a fixed sequence. Future studies that apply a randomized controlled trial with a cross-over design for each participant over different modes are warranted.⁵⁴ Third, the use of a cross-sectional design limits the direct clinical application of our findings. It is critical to conduct longitudinal studies to test whether implementing different modes leads to a similar magnitude in PRO changes.

In conclusion, the survey mode preference (postal mail, telephone, or Web based) for PRO assessment varies by sociodemographic variables, and treatment characteristics of survivors of childhood cancer are associated with selecting a specific mode to complete a PRO survey. However, measurement properties across these modes are equivalent, and DIF effects related to modes are minimal. The data indicate that cancer survivorship studies can use multiple modes to collect PRO data without jeopardizing data quality.

Appendix

FIG A1.

Identification of study participants. CCSS, Childhood Cancer Survivor Study; PRO, patient-reported outcome.

TABLE A1.

Missing Data by Individual Items of the BSI-18 Across 3 Modes of Survey Administration

AUTHOR CONTRIBUTIONS

Conception and design: Melissa M. Hudson, Leslie L. Robison, Gregory T. Armstrong, Kevin R. Krull, I-Chan Huang

Financial support: Leslie L. Robison, Gregory T. Armstrong, I-Chan Huang

Administrative support: Leslie L. Robison, Gregory T. Armstrong, Kevin R. Krull, I-Chan Huang

Provision of study materials or patients: Leslie L. Robison, Gregory T. Armstrong, I-Chan Huang

Collection and assembly of data: Leslie L. Robison, Gregory T. Armstrong, Kevin R. Krull, I-Chan Huang

Data analysis and interpretation: Jin-ah Sim, Geehong Hyun, Todd M. Gibson, Yutaka Yasui, Wendy Leisenring, Melissa M. Hudson, Gregory T. Armstrong, Kevin R. Krull, I-Chan Huang

Manuscript writing: All authors

Final approval of manuscript: All authors

Accountable for aspects of the work: All authors

AUTHORS' DISCLOSURES OF POTENTIAL CONFLICTS OF INTEREST

The following represents disclosure information provided by authors of this manuscript. All relationships are considered compensated unless otherwise noted. Relationships are self-held unless noted. I = Immediate Family Member, Inst = My Institution. Relationships may not relate to the subject matter of this manuscript. For more information about ASCO's conflict of interest policy, please refer to www.asco.org/rwc or ascopubs.org/cci/author-center.

Open Payments is a public database containing information reported by companies about payments made to US-licensed physicians (Open Payments).

Melissa M. Hudson

Consulting or Advisory Role: Oncology Research Information Exchange Network, Princess Máxima Center, SurvivorLink

Kevin R. Krull

Patents, Royalties, Other Intellectual Property: Royalties from Wolters Kluwer

No other potential conflicts of interest were reported.