Feasibility and acceptability of using an Interactive Voice Response System (IVRS) to assess decision making about sun protection

Abstract

Objective:

We developed an Interactive Voice Response System (IVRS), an automated telephone survey technology, to assess real-time decision making about sun protection. We examined the feasibility and acceptability of IVRS in this electronic health (eHealth) context.

Methods:

Melanoma patients who underwent surgery referred their first-degree relatives (FDRs) for participation. Eligible FDRs were contacted twice daily (12:30 p.m.; 5 p.m.) over 14 consecutive days via IVRS to complete a survey about their sun protection behaviors and decisions about those behaviors.

Results:

Of the 81 eligible FDRs, 69 (85%) consented to the study, and 53 (77%) completed the study. We assessed adherence with the IVRS via the number and pattern of missing survey items across all answered IVRS calls. About 80% of scheduled IVRS calls were answered (1316/1652). Most surveys (93%) of the IVRS-answered calls were completed. To examine acceptability, we analyzed the program satisfaction survey data collected at the end of the study. Most participants viewed the IVRS to be highly acceptable and easy to use.

Conclusions:

These findings illustrate that use of real-time IVRS data collection regarding sun protection decision-making is feasible and acceptable to higher-risk research participants and could thus be used with time and location-sensitive eHealth support to enhance sun protection decision making.

Introduction

Understanding how people make decisions about sun exposure and sun protection are critical to the development of effective melanoma risk reduction interventions. Sun exposure is the primary modifiable risk factor for melanoma,^1–4 yet it tends to be practiced inconsistently, even among at-risk individuals, such as those with a family history of melanoma.^5–9 There is little research examining daily variation in decision making about sun protection, so understanding such decisions may shed light on why sun protection is inconsistently practiced. Feasible and acceptable strategies are needed to capture variation in sun protection decision making across contexts, and this would be best accomplished with electronic health (eHealth) strategies. Commonly used global self-report strategies that capture general trends or cumulative frequencies (e.g., “What do you usually do on sunny summer days?”) do not have the requisite sensitivity to elucidate the range of behaviors that may be practiced - and perhaps more importantly, the decisions that drive these behaviors - even by one individual across a variety of outdoor contexts. Ecological momentary assessment (EMA) is a data collection strategy designed to capture information about individuals’ daily behaviors, attitudes, and moods close in time to the experience and in the participants’ natural environment.¹⁰ As EMA captures real-time data, it can be easily collected via interactive voice response system (IVRS), which is an automated telephone system that allows callers to interface with a pre-programmed database through their telephone keypad or voice commands. Real-time data collection integrated with eHealth interventions, have utilized IVRS and in this study we used IVRS to try to understand the real-time decision making process that may influence behaviors.^11–14

We developed an IVRS to assess real-time decision making about sun protection to capture variations in sun protection decisions across time in melanoma first degree-relatives (parents, siblings, children, FDRs). Outcome findings are reported elsewhere.^15,16 In the current paper, we examine adherence to IVRS for three reasons. First, adherence can be a critical barrier to interpretation of IVRS data,¹⁷ and actual engagement with IVRS is understudied and warrants specific attention.¹⁸ Second, there are multiple aspects of study adherence, including study attrition, and adherence within and across telephone calls, that relay important detail that is relevant to future IVRS iterations. Third and finally, to our knowledge, IVRS has not been used in the setting of sun protection and skin cancer risk reduction, making feasibility and acceptability important areas for examination in this context. Accordingly, we examine feasibility (study attrition and adherence) and acceptability (satisfaction, study facilitators and barriers) of IVRS to examine real-time sun protection choices, as well as the decisions that drive these behaviors, across diverse outdoor contexts.

Materials and Methods

Sample & Recruitment

Melanoma patients who underwent surgery at a New York City comprehensive cancer center referred their FDRs for participation in this Institutional Review Board approved study (IRB Reference #09–137). Eligible FDRs were English-fluent, age 18 or older, reported at least some use of sun protection, and had to affirm that they would be outdoors for at least 1 consecutive hour in the morning, and again in the afternoon every day throughout a 14-day assessment period of their choice between May and October when people are more frequently engaged in outdoor activities in the Northeastern United States. Individuals were ineligible if they never used sun protection. We approached melanoma patients at their postsurgical follow-up appointments or by letter followed by a phone call to request permission to contact their potentially eligible FDRs. Each patient was offered the opportunity to refer more than one FDR, however only one FDR per patient could consent. With the patient’s permission, we then contacted their referred FDRs to assess study interest and eligibility. In brief, FDRs were told that the purpose of this study was to understand how people make varied decisions about sun protection so that we can improve our ability to help those who may be at risk for melanoma. Those interested were then consented to the study, and elected the 14-day assessment window of their choice for the study, between the months of May and October.

Procedure

Once consented, participants completed a brief demographic and cancer history questionnaire (i.e., age, race, gender, marital and educational status, any prior skin cancer diagnoses, the number of family members with a skin cancer diagnosis, prior experience with electronic equipment) by telephone. We also provided participants extensive IVRS training and support materials (available from the first author). Prior to the 14-day IVRS assessment period, all participants were mailed a packet of training and support materials including a copy of an IVRS training manual and survey, a wallet card of the telephone keypad numbers corresponding to survey responses, participant ID and PIN numbers, a sticker that listed a study phone number, and the IVRS phone number in case they missed a call. Participants received a 15-minute IVRS training call prior to their assessment period after which they completed an “IVRS Proficiency Checklist” created by study staff to ensure that they met all training goals and proficiency was at 100%. Participants received additional briefing until they reported 100% proficiency with the IVRS. After this was completed, each participant was called via IVRS twice daily (12:30 p.m.; 5 p.m.) during their scheduled 14-day assessment window. At the conclusion of the 14-day assessment period, participants completed a post-IVRS satisfaction survey via telephone with study staff to report their level of acceptability with study procedures. All participants who completed the study received $50 and National Cancer Institute information on skin cancer prevention and screening.

Assessments

The IVRS software application was developed in-house with the help of an external vendor Portal 724 (http://portal724.com).¹⁹ To answer the survey questions, participants pressed designated numbers on their telephone keypads. Participants were able to skip any of the IVRS questions if they preferred not to answer an item. If an IVRS call was not answered, the system was programmed to call up to two more times within a 1-hour period and leave a voice mail message stating that the participant would be called back again shortly. If the participant did not answer the third call for a given time period, they were left a voice mail message stating that they would be called at the next assessment period. Participants were able to call the IVRS within 1 hour of the first call attempt (either at 12:30 or 5 p.m.) to complete the survey. If a call was disconnected, the IVRS called the participant back and began the survey where it left off. To maximize study accessibility or in the case of technical problems, participants were given the opportunity to complete any assessments on paper if needed.

During each call, participants were asked to complete a 29-item survey expected to take about 6–8 minutes to complete. The survey was developed from prior qualitative work identifying sun protection decisions in melanoma FDRs.²⁰ The 29-item survey included four yes/no questions about the use of sun protection (sunscreen, hats, protective clothing, shade-seeking) during that time period (morning or afternoon), and then the final 21 questions (yes/no) asked about decisional factors (e.g., weather conditions, type of activity, social encouragement.²⁰ Four questions assessed perceived melanoma risk,^5–7,21,22 sun protection self-efficacy,²³ and satisfaction with sun protection use^24–26 on Likert-type scales. And the end of each of the 28 surveys, participants provided a brief audio-recorded voice narrative (≤ 2 minutes) to describe additional context regarding their decision to use or not use sun protection during the designated assessment period.

At the end of the 14-day assessment period, participants completed a telephone survey examining acceptability (i.e., perceived ease, comfort, and utility with the IVRS, participant burden, and overall study impressions) as well as facilitators and barriers to survey completion (16 items, Likert-style response categories) that was adapted from an eHealth intervention program satisfaction survey.²⁷

Data Analysis

We assessed feasibility across two domains, study attrition and adherence. We used descriptive statistics to examine attrition (the percentage of consented research participants who withdrew from the study prior to completion) and study adherence (the percentage of participants who answered all 28 scheduled IVRS calls). We also examined adherence to each survey item given that those items later in each call may be more likely to be skipped. To examine associations of demographic factors to attrition and adherence, statistical comparisons (Chi-square or t-tests) were used. No multiple comparison adjustments were applied to the p-values due to the descriptive nature of the study. We assessed acceptability across three domains, satisfaction, study facilitators and barriers), we utilized descriptive statistics examining items from the Program Satisfaction Survey. All analyses were examined using SPSS 22.0.

Results

Over the course of two summers we contacted 512 melanoma patients and 251 (49.0%) patients referred a total of 418 FDRs. From this pool of 418 FDRs, 284 (67.9%) FDRs screened ineligible (predominantly because they did not meet the requirement for daily sun exposure), 81 (19.4%) screened eligible, and 53 (12.7%) refused. Of the refusers, about half (n=29, 54.7%) were considered “passive refusals”, as they were never reached for screening. The remaining refusers reported being too busy (n=14, 26.4%) or not interested (n=10, 18.9%). Of the 81 eligible FDRs, 69 (85.2%) consented to participate in the study. An abbreviated report on study adherence was provided in our primary outcome study.¹⁶

Attrition

Of those 69 who consented to study participation, we had 23% (n=16) attrition. Ten individuals withdrew prior to starting the IVRS. Another six participants dropped out by the third day of receiving IVRS calls so did not complete the study in its entirety (including post-IVRS satisfaction survey). We compared the demographic characteristics between those who withdrew from the study (n=16) compared to those who fully completed it (n=53) and found that siblings were more likely to withdraw than children or parents, participants with a longer number of days from study consent to planned start of the IVRS were more likely to withdraw; and those consented in summer wave 2 were more likely to withdraw than those recruited in summer wave 1 (all ps<.05).

We considered our final study sample the 59 who consented to the study and completed at least some of the IVRS. Two participants completed one IVRS assessment on paper due to technical difficulties they experienced using the IVRS and 1 completed 8 IVRS assessments on paper in observance of religious practices, however this participant did experience a few technical difficulties as well. The sample was predominantly female (64.4%), on average age 48, and a majority were children of melanoma patients (66.1%), followed by siblings of melanoma patients (18.6%) and then parents of melanoma patients (15.3%), see Table 1.

Table 1.

Sample demographics.

Characteristic	Study Sample
	N=59
Gender
Male	21 (35.6%)
Female	38 (64.4%)
Age (mean, SD)	48.7, 15.3
Relationship to patient
Parent	9 (15.3%)
Sibling	11 (18.6%)
Child	39 (66.1%)
Race †
White	58 (100.0%)
Ethnicity †
Hispanic	1 (1.7%)
Marital Status †
Married	36 (62.1%)
Not Married	22 (37.9%)
Education †
High School or less	6 (10.3%)
Some College	9 (15.5%)
College Grad	22 (37.9%)
Graduate Degree	21 (36.2%)
Employment †
Employed	38 (65.5%)
Unemployed	20 (34.5%)
Family members with melanoma dx ‡
One	45 (78.9%)
Two	8 (14.0%)
Three	4 (7.0%)
Skin cancer diagnosis for self †	6 (10.3%)

^†n=58 for study sample

^‡n=57 for study sample

Adherence

We assessed adherence via the number and pattern of missing survey items in all answered IVRS calls. About 80% of the scheduled calls were answered (1316/1652) as seen in Figure 1. Most morning and afternoon surveys (93.3%) were completed in their entirety (i.e., all 29 questions were answered in each survey), leaving only 88 surveys (6.7%) partially completed. Notably, the likelihood of skipping an item within a survey call increased directly with item order in the survey, with later items successively more likely to be skipped (r=.96, p<0.001, Figure 2), suggesting that skipped items were likely the result of participants having to end the call early. Also as shown in Figure 2, the likelihood of ending the call early seems to increase after item 10. The mean number of calls answered per participant was 22.3 calls (SD=7.6) out of 28, and this was unrelated to the timing of the call (i.e., whether the call was made at 12:30 p.m. or 5:00 p.m.), whether the call occurred on a weekday or weekend, or the specific day of the week that a call was placed. For the surveys that were completed in a single phone call (i.e., without getting disconnected, 83%) the mean length of time for each IVRS call time, including the audio narrative, was 2.48 minutes, the range was 0.00 to 6.45 minutes. There were no demographic differences (gender, age, education level, employment and marital status, relationship to patient) among those who answered fewer than 28 calls compared to those who answered all 28 calls (all ps >.05).

Figure 1.

Scheduled IVRS calls answered by call across 14-day study period (N=59).

Figure 2.

Prevalence of skipped survey items in relation to numerical order of survey items (N=59).

Acceptability

To examine acceptability, we analyzed the program satisfaction survey data collected at the end of the study from the 53 participants who completed the entire study (the remainder did not complete the program satisfaction survey). Most participants viewed the study to be highly acceptable (See Table 2), reporting that the IVRS was easy to use for both the survey items and voice narratives (88.7% and 94.3%, respectively). A minority (22.7%) found the calls to be intrusive; about half (54.7%) indicated that the calls came in at inopportune times during the day. Most (81.1%) found the telephone-based training that staff provided to be extremely or very helpful. In general, most of the sample reported strong preferences for providing data through the IVRS. A majority (81.1%) favored an IVRS-based data collection approach compared to telephone surveys administered by study staff. Most held positive views about the study design overall, indicating that they would participate in a similar study again (88.6%) and would recommend IVRS data collection to others (83.0%). Most (83.0%) were comfortable with the survey time commitment, but a minority (28.3%) felt that the 14-day study period was too lengthy a timeframe in which to participate.

Table 2.

IVRS acceptability data.

Acceptability Item	Participants (n=53) †
How hard or easy was it to use the IVRS to complete the surveys?
Very easy or somewhat easy	47 (88.7%)
Somewhat hard or very hard	6 (11.3%)
How hard or easy was it to use the IVRS to complete the voice narratives?
Very easy or somewhat easy	50 (94.3%)
Somewhat hard or very hard	3 (5.7%)
How comfortable were you with using the IVRS?
Extremely comfortable or very comfortable	47 (88.6%)
Moderately comfortable	5 (9.4%)
A little bit comfortable or not at all comfortable	1 (1.9%)
How much of an inconvenience was it to answer the phone and complete the surveys and voice narratives?
Extremely inconvenient or very inconvenient	4 (7.6%)
Moderately inconvenient	18 (34.0%)
A little bit inconvenient or not at all inconvenient	31 (58.4%)
Did you think the length required to complete each survey call was:
Just right	44 (83.0%)
Too long	9 (17.0%)
Did you think the length of the entire project assessment, answering two calls per day for 2 weeks, was:
Too short	3 (5.7%)
Just right	35 (66.0%)
Too long	15 (28.3%)
How intrusive was answering the IVRS phone calls to complete the surveys?
Extremely intrusive or very intrusive	2 (3.8%)
Moderately intrusive	10 (18.9%)
A little bit intrusive or not at all intrusive	41 (77.4%)
Would you have preferred to answer the survey questions and complete the voice narratives directly with a research assistant over the phone?
Yes	10 (18.9%)
No	43 (81.1%)
I would recommend this form of research data collection to others.
Agree strongly or agree somewhat	44 (83.0%)
Neither agree or disagree	4 (7.5%)
Disagree somewhat or disagree strongly	5 (9.5%)
I would do a research study like this again in the future.
Agree strongly or agree somewhat	47 (88.6%)
Neither agree or disagree	4 (7.5%)
Disagree somewhat or disagree strongly	2 (3.8%)
Did you think the over-the-phone training session with research staff was:
Extremely helpful or very helpful	43 (81.1%)
Moderately helpful	3 (5.7%)
A little bit helpful or not at all helpful	7 (13.2%)

^†53 participants completed the program satisfaction survey.

Perceived facilitators and barriers to using IVRS.

The most commonly endorsed facilitators for study completion selected from a list of options among the 53 participants who completed the program satisfaction survey included keeping the study phone close by (62%), referencing the study training materials (i.e., sticker [28%], wallet card [25%], manual [21%]), and avoiding scheduling activities during study call times (30%). Other facilitators volunteered by participants included the extensive phone training provided (n=4), having a paper copy of the IVRS survey to familiarize themselves with the items (n=3), adding the IVRS calls into one’s calendar (n=3), and adding the IVRS phone number to one’s cell phone (n=2). The most important endorsed barrier to study completion involved being busy when the system called 25%). Participants offered other barriers in completing the phone surveys such as phone calls being cut off or the system hanging up on participants during the course of the surveys (n=12) and the IVRS not accepting the participant’s ID or PIN number (n=7), both of which were necessary telephone keypad entries in order to begin the surveys. Poor cellular reception (n=6) and feeling that either the survey questions were difficult to answer or that the survey response options were hard to use (n=6) were also noted as barriers to study completion.

Discussion

In this paper we examine the feasibility and acceptability of using IVRS to assess real-time decision making about sun protection in FDRs of melanoma patients. While this technology is feasible and acceptable in other behavioral contexts,^{11–13,28–30} this study confirms that it is also quite feasible and acceptable in this novel setting, to collect information about sun protection, as well as the decisions that drive sun protection choices. This is promising as it allows for data collection across multiple, diverse outdoor environments where these discrete, real-time decisions are made, and can be integrated into eHealth interventions.

We explored feasibility across two dimensions, attrition and IVRS adherence. Our attrition rate (23%) is consistent with prior work.¹³ There were two factors that were important covariates of attrition. First, we found that siblings of melanoma patients were more likely to drop out of the study than parents or children. Future studies in which a patient’s family members will use IVRS data collection may want to consider the nature of the family member’s relationship to the patient when developing strategies to minimize study withdrawal. Second, those consented in summer wave 2 were more likely to withdraw than those recruited in summer wave 1. Some slight variations in procedure may have caused this. For wave 1, we consented participants once they were able to identify a specific IVRS start date; whereas in wave 2 we consented participants when they agreed to participate, regardless of their IVRS start date. These procedural differences led to a longer mean length of time between participant consent and initiation of IVRS data collection for the later cohort, and, as we found, a higher rate of failing to follow through with the study. For all potential participants, initiating a participant’s IVRS data collection closer to the date of study consent may help to improve study retention.

In terms of adherence, most participants who answered at least one IVRS call followed through with the entire 14-day period, which resulted in minimal missing data. This outcome may in part be a result of the extensive training and support we provided to participants about the IVRS. Adherence rates to IVRS calls were consistent across a number of dimensions, including time of call (i.e., either 12:30 p.m. or 5 p.m.), the day of the week, and whether the call was placed on a weekday or a weekend. The completion rate of the 29-item survey was very high, yet expectedly we found a higher likelihood of missing data later in the survey. The mean length of time for each call was brief - 3.28 minutes - which may have also contributed to high adherence rates.

Clinical Implications

We found a range of addressable study barriers, including problems with the IVRS (i.e., both IVRS itself and cell phones) to person-specific barriers (i.e., being busy at the time of the call). The most commonly reported technical problems were with the IVRS not recognizing participants’ ID and PIN numbers as they attempted to log into the system. Another technological barrier not necessarily related to the IVRS itself was with calls being dropped during the surveys and phones being out of range during survey call times. Barriers that were not technical in nature included the length of time from study consent to IVRS initiation; the longer this length of time, the higher the attrition. Finally, many participants who answered at least one IVRS call reported being busy at the time of the IVRS calls and therefore were unable to provide study data at that time. Of note, several participants who withdrew from the study indicated being too busy with life demands. To address these barriers, we recommend allowing adequate time for developing an IVRS, testing the IVRS using multiple cell phone carriers to eliminate programming glitches specific to one carrier, and randomly altering the order of IVRS survey items across calls to avoid higher levels of missing data on items placed later in the survey. Lastly, and in agreement with Grzywacz and colleagues,¹³ we believe that comprehensive training is a critical study component that ensures high participant acceptability and study adherence. We programmed the IVRS to be flexible for participants to use without compromising the study design. Specifically, the system placed follow-up calls to participants if they did not answer initial calls at either 12:30 or 5 p.m., and it called participants back if the calls were cut off during survey completion. Overall, study acceptability was high, with most participants reporting that they were comfortable with the system, found it easy to use, and stated that they preferred reporting their sun protection use and decision making through the IVRS than by telephone to study staff.

Study Limitations

Study limitations included the modest sample size, and the fact that our at-risk population may have been more adherent to the study than the general population given the salience and importance of sun protection to their personal health. Our study was also limited in that most participants had high levels of educational attainment. We were unable to verify self-reported weather conditions that were the basis for sun protection decision making. We did not record whether participants used a landline or cell phone for the study; having a landline may have accounted for some of the missing data if they were not home to answer the phone. Finally, the IVRS did not account for time zone, although we only had a few (n=7) who were not living in the Eastern United States Daylight Saving Time (EDT) zone. These limitations certainly dictate the need for further research in larger, more diverse populations, possibly limiting participants to cell phone usage only, and including other objective measures of sun exposure, local weather conditions, and time zone differences into consideration. The relative accuracy of data collected through IVRS compared to that obtained through retrospective self-report or objective measures of sun exposure are also important research objectives. We also suggest extending this research to other geographical locations that are not constrained by seasonal variation in sunny outdoor activities.

Use of IVRS in eHealth behavioral intervention studies has many benefits, including access and cost effectiveness in diverse populations as most Americans own or have access to a cell phone,³¹ as well as real-time data entry and the ability to monitor compliance over time, automated follow-ups, the use of standardized interviewing, and reliability of the time stamp on data collection.^28,32 Other real-time data collection strategies provide equivalent data quality to IVRS.^28,33–35

In the current study, we examined important aspects of adherence that will be relevant to future IVRS iterations,¹⁷ and, more broadly, confirm feasibility and acceptability of IVRS in a novel setting related to sun protection and skin cancer risk reduction. We demonstrated that IVRS is a feasible, acceptable data collection method for sun protection decision making. Our participants were highly adherent to the IVRS data collection schedule and found the IVRS to be acceptable and easy to use. These findings provide important guidance for future research using real-time data collection methods to assess eHealth strategies to encourage sun protection decision making.