Maximizing Acceptability and Usefulness of an Automated Telephone Intervention: Lessons from a Developmental Mixed Methods Approach

Abstract

Objective

To describe the utility of mixed method techniques to inform the development of an automated telephone intervention that seeks to improve patients’ compliance with asthma medication.

Study Design

Mixed methods with qualitative and quantitative components.

Methods

As part of intervention development for a larger randomized clinical trial, we conducted 15 focus groups (n=53) to design and develop calls, and identify factors influencing intervention acceptability and usefulness. We piloted 4 call types and interviewed 64 participants to further improve call content and receptivity to the intervention.

Results

The feedback obtained from this mixed methods approach directly led to several changes to the intervention scripts and eventual calls. These included immediately identifying the call source/purpose; acknowledging the call’s use of computerized technology; and personalizing relevant messages that reflect local culture. In addition, participant feedback following an initial pilot of the intervention led us to drop one of the calls. During the pilot, we reached 43% of target participants; 74% of those stayed on the call until it ended. Call computerization was not a primary reason for declining or discontinuing calls.

Conclusion

This inductive developmental process using mixed methods provided key insights about the acceptance and use of automated calls for a medication reminder program, and may have broader applicability for the development of automated interventions designed to help patients manage a variety of chronic conditions.

Introduction

Automated telephone messaging systems (commonly called Interactive Voice Recognition, or IVR systems) have been used in multiple studies to collect health data, monitor patients, provide health education, and remind patients about appointments or health screening activities. [1] IVR systems can be used to provide health education, to increase compliance with childhood and influenza immunizations or with medication regimens, and to assist in management of cardiovascular disease and diabetes.[2–9] In a systematic review of IVR-mediated applications, Krishna et al. [10] concluded that automated telephone interventions can have a significant impact on both behavioral and clinical outcomes.

IVR systems use voice-processing technology to link individuals to their health information stored in a database. Pre-recorded voice files generally prompt the caller to press telephone buttons to answer questions or request information. The IVR system can then access the host computer’s database, and give information back to the patient. [11;12] Recent applications of IVR technology utilize advanced speech-recognition software that allows patients to respond to recorded queries with natural-language answers, thus potentially enhancing the acceptance and effectiveness of this telephone-based interaction. [13;14] Overall, IVR systems have the advantage of delivering tailored health information and scripts to large populations at low cost, as well as efficiently collecting data. [12;15] Thus, IVR systems may be a promising strategy to enhance chronic-disease management in managed care settings. [15;16]

Finding ways to maximize the acceptability, usability, and usefulness of IVR interventions to patients may be key to their success. Yet, although a variety of health-care settings seem to be increasingly utilizing IVR systems to improve patient care, [17] little is known about exactly how patients receiving these types of interventions respond to them. In particular, review of the literature suggests that few, if any, studies have explored participants’ reaction to, and use of, IVR and other automated phone technology during design and implementation stages of the intervention. [11;18;19]

We report our findings from a mixed-methods approach used to develop an IVR intervention using speech-recognition software. This paper describes the mixed-methods approach we used and summarizes the lessons it provided to us as part of our intervention development. The findings should have broader applicability for the development of automated interventions designed to help patients manage a variety of chronic conditions.

Methods

Study Overview

This study is based on the intervention development phase for a larger randomized clinical trial, “Phone Calls to Promote Adherence with Inhaled Corticosteroids” (PEANUT), testing the effectiveness of telephone outreach that uses speech recognition software to improve compliance with inhaled corticosteroids among patients with asthma.

The objective of the developmental evaluation reported here was to enhance the acceptability and usefulness to asthma patients of an IVR intervention to improve compliance with medication. We used a mixed-methods approach relying on three components that built upon each other: a) member focus groups; b) pilot testing of IVR calls; and c) qualitative debrief phone interviews in a subset of those who received pilot calls. Specifically, we wanted to obtain participants’ feedback about getting automated calls, including reasons for continuing or hanging up, and to evaluate areas where we could improve the intervention during the development phase of the study. The pilot test of calls provided a gauge of how many participants the intervention might reach, how long participants stayed on the call, how many accepted a transfer to the refill line or pharmacist, and points in the script where people commonly hung up.

Research Setting and Participant Recruitment

The study setting was Kaiser Permanente Northwest (KPNW) and Kaiser Permanente Hawaii (KPH). Kaiser Permanente (KP) is a group-model health maintenance organization (HMO) that provides comprehensive, prepaid health care service to about 450,000 members in KPNW and 230,000 members in KPH. KPNW serves a population that is largely Caucasian (~91%), while the KPH population includes about 25% Caucasians, 47% Asians, native Hawaiians, or Pacific Islanders, and 29% mixed heritage. Kaiser Permanente’s electronic medical record (EMR), which provides both administrative and clinical data, was utilized to identify potential study participants for recruitment.

We recruited focus group and pilot participants from a group of KPNW and KPH members who met the eligibility criteria for the larger PEANUT trial. Criteria included being 18 years or older, diagnosed with asthma, having had at least one dispensing of an anti-asthma medication, and health maintenance organization membership for past year (Figure 1).

Figure 1.

Patient Recruitment for the 3 Components of the Developmental Evaluation

Qualitative Methods

We utilized qualitative methods, including focus groups and interviews, because [20–22] they elicit participants’ perspectives, and are therefore useful in defining the range and variability of beliefs, behaviors, and experiences of study populations, as well as the natural language used to discuss these issues. [20–24]

Focus Group Methods

First, we conducted a series of 15 focus groups [25] (KPNW = 7; KPH = 8) to aid in call design and content development, and in creating an acceptable IVR intervention. We chose a two-phase approach so that changes to the call scripts, based on feedback from the first round of focus groups, could be “member checked” [20–24] with participants in the second round. Approximately 400 potential focus group participants received letters of invitation, followed by a phone call from study staff. Participants attending focus groups were provided with a meal and gift card. Participants attended two consecutive focus groups, approximately 2 months apart, each of which lasted approximately two hours. The first round of focus groups explored barriers participants experienced in both taking and filling asthma medications as prescribed. Participants also listened and reacted to sample telephone calls that used IVR technology. Data from the initial focus groups were used to create sample intervention telephone call scripts, and scripts were recorded to mirror what the intervention telephone calls would sound like in real time. Approximately two months later, participants were invited back, listened to the sample telephone calls and provided feedback, which was used to further refine the content and delivery of the scripts for the larger clinical trial.

IVR Pilot Call Methods

We used focus group data to assist in programming the phone intervention script content and branching logic. We tested four unique call types (welcome call, new-user call, simple refill reminder call, and tardy refill call) with 500 patients (KPNW = 250; KPH = 250). We selected equal numbers of participants by site and call-type eligibility. Pilot participants received an invitational letter and brochure, including an opt-out postcard. Those not opting out received an IVR pilot call in month one of the pilot, and may or may not have received further calls in the subsequent 3 months of the pilot, according to their eligibility as determined by monthly queries of dispensing data from the electronic medical record.

The four call types tested in the pilot included the following features, depending on call type. The average call lasted less than 2 minutes. Figure 2 shows a sample call flow and branching logic.

Figure 2.

• ○Welcome call: orientation to refill reminder service and active consent (included in all call types if the participant was interacting with the IVR intervention for the first time)
• ○Simple refill reminder: brief refill reminder and offer to transfer to refill line
• ○Tardy refill reminder: refill reminder, assess asthma control, provide medication education, offer suggestions for overcoming barriers to compliance with medication, and offer refill line and/or pharmacist transfer
• ○New user/restart call: first fill reminder, assess asthma control, provide medication education, offer suggestions for overcoming barriers to compliance with medication, and offer pharmacist transfer

Qualitative Phone Interview Methods

We contacted 64 of the pilot participants for telephone debrief interviews. All pilot participants who had received at least one of the 4 call types were eligible to be contacted for a debrief phone interview. We purposely recruited [22–24;26] individuals equally from among those who had received the different call types and we over-sampled [22–24;26] participants who hung up compared with those that completed a call in order to gain better understanding of reasons for hanging up. Participants were contacted approximately 5–7 days after receiving an IVR call, and telephone interviews lasted approximately 20 minutes. Participants who fully completed an IVR call were asked about their overall reaction to the call content and IVR technology, the perceived usefulness of the call, and their likelihood of taking future calls. Participants who hung up early in the call or declined participation were asked their reasons for discontinuing the call, as well as about their overall reaction to the call content and IVR technology, the perceived usefulness of the call, and likelihood of taking future calls.

Analysis Methods

Focus groups [25] and telephone interviews [27] were designed, conducted, and analyzed by the author (JS), a researcher trained in qualitative methodology, and by other study staff also trained in qualitative methods. A common interview guide was developed for both rounds of focus groups, and for telephone debrief interviews, to increase consistency across interview sessions while also allowing individual participants’ experiences to emerge. [24;24–26;26;27] Focus groups were audio taped and professionally transcribed for analysis. Telephone interviews were manually recorded in detail. We used a thematic-content analysis approach, guided by a specific set of standard qualitative data reduction, and reconstitution techniques that included coding passages of text. [28–31] For example, we developed a coding dictionary based on the interview guides and review of transcribed interviews. Transcribed interviews were then coded by marking passages of text with phrases indicating the content of the discussion. We utilized a qualitative research software package, ATLAS.ti 5.0 (Scientific Software Development, 1997) to electronically code the transcripts and generate summary reports of coded text. Using the report and query functions of ATLAS.ti, coded text was further reviewed through an iterative process, resulting in refined themes.

Pilot call data were analyzed by placing them into mutually exclusive groups based on the best call result of any call attempt made and how much call content was received by the participant during the call. The two primary, mutually exclusive groups identified were: confirmed target reached or not-reached. To help assess level of participant engagement in the call attempt, the confirmed target reached group was further divided into the following sub-categories: willingness to continue with the call after confirming identity; intervention began; normal wrap up/medication refill transfer option offered; or answering machine message left; and declined taking a message for confirmed target. For the confirmed target not-reached group, sub-categories of data included: hang-up; wrong number; and unreachable (busy or no answer).

Results

Focus Groups (Tables 1–2)

Table 1.

Examples of How Focus Group Results Guided Development of IVR Script Design and Content

Participant Identified Barriers to Asthma Medication Adherence	Attributes	Illustrative Quotes from Focus Group Participants	Sample IVR Script Content and Language Influenced by Focus Group Findings
Lack of Routine	- no routine at all	-“What do you do to remind yourself-because it’s not like a pill where you can take it on Tuesday, open it up and see that you haven’t taken it yet.”.	Tardy Call: “Many people with breathing concerns also find that they often forget to take their medication. Do you sometimes find it difficult to remember to take your [drug name]?”
	- routine for 1 dose but not the other dose (morning versus evening routines)	-“I have a routine in the morning – shower, comb hair, deodorant, sprayer [medication], brush teeth – in that order every morning. But the evening, there is no routine for the evening so I sometimes forget.”	IF YES“ Many people do find that it takes time to develop a routine for making medication a part of their daily lifestyle. You may find it helpful to think of it as something you need to do as part of some daily routine. You might try putting your [DRUG NAME] in a location where you will see it and remember to use it. For example, keep it by the bathroom sink so you can use it before you brush your teeth or in the kitchen so you can take it after one of your meals. If you feel comfortable doing so, you can always ask a family member or close friend to help make it a part of your routine. And please remember that your doctor and [HEALTHPLAN] pharmacy department also can be great resources for helping you find ways to remember to take your medication every day.”
	- difficulty “getting all the doses in” in one day	-“I try REAL hard not to miss both morning and night time doses, but I have some nights I am so tired I don’t have the energy to go through the process of taking it.”
Forgetfulness	- overwhelmed by responsibilities of a “busy life”	-“I forget to take it in the morning so I take it to work with me. But then I get busy at work and don’t have the time to take it.”	- All call types: “taking your [drug name] every day is one of the most important things you can do to help control your breathing.”
	-“in a hurry” during daily transition times	-“I just do it in the morning and sometimes I do forget because I am in a hurry to get out the door.”	-All Call types: Many people also find that they simply have trouble remembering to fill their prescriptions. Is this true for you?
			IF YES “then we hope you’ll find this Refill Reminder Program helpful.”
Lack of Understanding about Medications	- lack of clear understanding as to “why my body needs” the medication	-“I need a better understanding…If I have a clear understanding of, ‘You need to take THIS medicine because this is what’s going on internally and this is what the medicine will do to help you,’ then I’m going to be more likely to take it.”	- All call types: “…one of the most important things you can do to help control your breathing is to take your [DRUG NAME] every day as prescribed, even if your breathing is already in good control.
	- lack of understanding, education, and reinforcement regarding the purpose of the medication and differences between controller and reliever medications	-“I think they should tell you that even though you feel better you should still take the Qvar. I didn’t realize that until I talked to my provider again. And I think they REALLY need to stress that.”	-Tardy and New User call types: If Poor control indicated by participant response, “Nighttime wakening, interference with daily activities, and the need for frequent use of your reliever medication are all signs that your breathing could be in better control. So you may want to speak with your doctor about how your breathing difficulties are affecting your life right now. Also, keep in mind that the best way to stay in good control is to take your [DRUG NAME] every day, even when you are feeling well.
	- often receive education and reinforcement during “acute attacks” when less likely to absorb, retain, and understand information	-“I don’t think I got enough of what I need to know to manage it during my visit. So for me, I may not understand it totally…”	-New User Call[ICS education module] “Your [DRUG NAME] is what is known as a controller medication which is used to prevent symptoms from occurring by keeping your airways from becoming irritated. Keep in mind that when you first start taking [DRUG NAME] it may not work right away. In fact, most people don’t see an effect for up to two weeks of daily use. Also unlike steroid pills, such as prednisone which can have side effects, your [DRUG NAME] is considered to be very safe to take for long periods of time. And remember, this medication works best when you use it every day. Would you like me to repeat that?”
Feeling Good	-feel good/no noticeable symptoms so choose not to fill	-“My asthma was not bothering me, so I got lazy and ran out of it [medication].”	New User and Tardy Calls: “One common reason people give for not taking their breathing medications is that they’re feeling well and don’t think they need to take them. Is this sometimes true for you?”
	-feel good/no noticeable symptoms so forget to fill	-“The problem with Qvar is if you are feeling good and don’t take it, you don’t notice anything, like more coughing, for another week or two.”	IF YES [ICS education module] same as above: “Your [DRUG NAME] is what is known as a controller medication which is used to prevent symptoms….”

Table 2.

Key Factors to Increasing Acceptability and Usability of IVR Calls as Identified by Focus Group Participants

Key Factor	Patient Recommendations	Illustrative Participant Quotes
Immediate and Clear Identification of Call Source and Purpose	- name healthcare organization immediately to increase credibility of call and distinguish it from a telemarketing call	-“I think they should say the call is from [HEALTHPLAN] first, real quick and upfront, other wise people may hang up thinking it is a marketing type call.”
	- immediately identify who the call is for and provide a succinct and clear description of call purpose and benefit to recipient	-“I think they ought to start off and explain immediately who is calling and why they are calling so that the person recognizes it is a benefit.”
Personalization and Relevancy of Message	- correctly record pronunciation of recipients’ name	-“If my name is not said right, I might be more likely to hang up…”
	- specifically name any medications or conditions and avoid generic language wherever possible	-“It will get my attention more if the call is real personal…say exactly what the medication is I need to refill.”
	- use call recipients’ preferred phone number (e.g. home, work, cell)	-“Can I tell them what number I want the call on…’cause I’m more likely to use my cell than my home [phone].”
	- time the call to an event/experience meaningful to the recipient to increase utility (e.g. near medication refill date or during allergy season)	-“‘This is [HEALTHPAN] and we just noticed it’s time to renew your medication. Because we know it is cold and flu season we just wanted to remind you of…’ a simple call like that will get my attention [participant’s suggestion for a useful call].”
	- incorporate language/events reflective of the local/regional culture	‘I thought adding ‘aloha’ was a real nice touch. Will the call end with ‘mahalo’?…because if they start with call with ‘aloha’, they have to end it with ‘mahalo.’”
Acknowledgement of Automated Technology	- within first few seconds of call identify the call as “automated” or “computerized”	-“Say this is a computerized voice or recorded voice right away in the message.”
	- acknowledgement orients recipient to computerized nature and structure of the call (yes/no format)	-“Say it is a computerized message and tell us how we need to respond ‘yes’ or ‘no’ to the questions – that way it feels less patronizing.”
	- acknowledgement focuses recipients’ attention to call content rather than to wondering whether or not the caller is a “real person”	-“You have to identify the call as automated so you know where you are at with it. ‘Do you have this, yes/no?’…this will help you identify right away you are speaking to a computer and need to be a little more patient.”
Delivery Style of Call	- record content in a business-like style reflecting professionalism and friendliness	-“The voice needs to be more businesslike, you know. Friendly, but not overly friendly – still serious.”
	- avoid an overly cheerful or “sing- song” style of delivery and voice	-“I found the speaker’s voice hard to understand at times – the sing-song quality to it made it hard to focus on.”
	- record voice at a slightly slower speaking pace to assist comprehension, especially for older and non-native recipients	-“I really couldn’t understand the call. It went too fast. I couldn’t follow that fast of pace…slow it down a bit.”
Voice Personality	- use a voice personality reflective of local/regional culture to increase sense of friendliness and familiarity	-“If the voice on the call sounds more localish in styling, it’s more friendly…a mainland voice doesn’t have the natural rhythms…it doesn’t sound as friendly.”
	- local/regional voice makes call feel more personalized and increases credibility of call	-“If the call is going to be used in Hawaii, understand that not everyone here went past high school. And so most people feel more comfortable hearing these terms in a voice that sounds real local.”
Duration of Call	- keep all calls to under 5 minutes and say how long the call will last in first few sentences	-“Get the call down to 5 minutes or less and tell us that early in the call…I have a few minutes to give, but I don’t have 10 minutes!”
	- eliminate redundancy from call content	-“I became impatient with the redundancy. The recorded voice repeated herself on more than one point and I don’t have patience with that.”
Written Information	- supply accompanying information (letter/brochure) prior to receiving calls to explain call purpose and relevancy	-“Some written information could help explain some things, like if you don’t have time to answer the call, what will happen?”
	- explain in more detail why the calls use automated technology rather than a “real person”	-“Do you have a letter of introduction going to all the people saying that there will be these automated calls? I think that’s important…to make us aware of it in advance.”

A total of 53 individuals (KPNW = 31; KPH = 22) participated in the focus groups. The first round primarily focused on identifying common barriers to taking and filling asthma medications. The three most commonly cited barriers were: (1) forgetfulness/too busy; (2) lack of routine; and (3) not understanding the reason and purpose for the asthma medication. Regarding refill behavior, the primary barrier related to a sense of feeling good/symptom free, thus causing patients to choose not to refill or to forget to refill. These participant-identified barriers to taking and filling asthma medication were incorporated into the IVR scripts, providing guidance on call content, logic/structure of the call and the natural language used (see Table 1 – column 4). Participants also briefly listened to some sample IVR calls and provided feedback, particularly focusing on what could be improved to keep the participant engaged in the phone call.

During the second round of focus groups, participants listened and reacted to the newly designed and recorded IVR intervention calls. Analysis of participants’ reactions and feedback revealed 7 key factors to improving intervention acceptability and usability : (1) immediate and clear identification of call source and purpose; (2) personalization and relevance of message; (3) acknowledgement of automated technology; (4) delivery style of call; (5) voice personality; (6) duration of call; and (7) accompanying written information. These key factors and corresponding patient recommendations (Table 2) were incorporated into the IVR telephone scripts and intervention recruitment materials.

IVR Pilot Calls (Table 3)

Table 3.

Summary of pilot call outcomes

Pilot Sample: Total Calls Attempted (n=992)
Contact Rates	%	Cum %
Target participant reached n=465	46.88%	46.88%
Message left n=360	36.29%	83.17%
Target household not reached N=167	16.83%	100%
Pilot Sample: Target Participant Reached (n=465)
Exposure to Intervention Rates	%	Cum %	% of total calls attempted (n=992)
Full intervention exposure: stayed on to call conclusion N=147	31.61%	31.61%	14.82%
Heard some intervention but hung up before call conclusion N=53	11.40%	43.01%	5.34%
Hung up before intervention began (e.g. during introduction) N=265	56.99%	100%	26.71%
Transfer outcomes
Automated refill line or live pharmacist transfer offered N=123	26.45%		12.40%
Transfer offer accepted N=60	12.90%		a6.05%

We invited 500 eligible members to receive reminder phone calls for their asthma medications, and made 992 calls to them over a 4-month test period. Of attempted calls, 47% resulted in reaching the target health plan member and keeping him/her on the telephone long enough to hear “Hello, this is the Kaiser Permanente Breathe Easy Medication Reminder Program…” We successfully left a message in an additional 36% of attempted calls. The telephone was not answered or was busy in 17% of attempted calls. In 15% of attempted calls, the member interacted with the intervention to the fullest extent and had a normal wrap-up. An additional 5% interacted with the intervention to some degree, but hung up before the call concluded.

Of those reached, 57% hung up before the intervention portion of the call began (49% hung up and 8% declined participation in the program during the consent portion of their first call). Twenty-six percent were offered a transfer to either the automated refill line or a pharmacist, and of these who were offered, 49% accepted the transfer (this is 13% of the 495 calls resulting in reaching the target member and 6% of the total number of attempted calls).

Debrief Qualitative Phone Interviews (Table 4)

Table 4.

Participant Reactions to the IVR Calls: Synopsis of Key Findings from Qualitative Debrief Phone Interviews

Overall Call Reaction: Complete and Incomplete Calls (n=64)
Key Finding	Percent of Participants Interviewed	Illustrative Participant Quote
Clarity and Purpose of Call is Clear	63% (40 out of 64)	- “This is very useful and it helps me with routine.”
		- “It [the call] went smoothly and was very handy and convenient…I got two Qvar in the mail and the transfer to refill worked well.”
Automated SR Technology is Acceptable	63 % (40 out of 64)	- “The computer technology is better than most.”
		- “I liked the friendly voice of the computer and thought it asked logical questions.”
Willing to Take Future IVR Program Calls	52 % (33 out of 64)	- “I would take a call again because it gives me a benefit.”
		- “I’d take another call, especially if it reminds me to fill my medication.”
Incomplete Calls Only: Reasons for Hanging Up / Discontinuing Call (n=42)
Service Not Needed / Reminder in Place	31% (13 out of 42)	- “This is a good idea but I just don’t need the service at the moment…I do fine.”
		- “I’m not interested… I already have a system in place that reminds me ahead of time when to re-order my medications.”
Call Taken by Another in Home	19% (8 out of 42 )	- “I think I never got the call and that my older mother-in-law took it for me…but a call to remind me is a good idea because I know I’m running out of medicine soon.”
		- “Five people live in my household and one of them probably took the call and didn’t tell me.”
Too Busy / Bad Time for Call	17% (7 out of 42 )	- “I was changing diapers when the call came and wasn’t able to take the call.”
		- “I wasn’t feeling well that day from a cold and didn’t want to be bothered by a call.”
Not Living at Residence of Designated Phone Number	12% (5 out of 42)	- “My son is working away from home during the summer months and is not around to take the call.”
		- “The call was for my son who is a student and is living away from the home right now.”
Challenges with Automated IVR Technology	12% (5 out of 42)	- “I couldn’t say much or ask questions and I wanted to change my responses and couldn’t…”
		- “It took me a few seconds to realize it was a computer and figure out the yes/no pattern.”

Of the 64 telephone interviews conducted, 22 were with participants who had fully completed one of the IVR phone calls. Of these, 91% had a positive overall reaction to the call; 95% found the purpose of the call clear ; 82% found the call timely and useful; 91% reacted positively to the automated technology; 23% chose to use the transfer option; and 86% indicated a willingness to receive future IVR calls.

We also interviewed 42 participants who either hung up before the call conclusion, or declined participation shortly after the call began (Table 4). Of these, 31% were not interested in the IVR call due to already having a good medication reminder system in place; 19% believed they did not receive the call and assumed another member of their household took the call on their behalf; 17% indicated the call came when they were too busy or rushed to take it; 12% were not currently living at the designated phone number; 12% were confused by or struggled with the automated technology (mostly older participants); and 7% cited language barriers.

Regardless of whether participants completed a call, 63% (40/64) of participants interviewed indicated they found the purpose of the IVR calls clear and the automated technology acceptable. Furthermore, 52% (33/64) of participants interviewed indicated they would take future calls from the intervention even if they initially did not complete their first call experience.

In general, reaction to the calls was similar across all four call types. However, participants who received only the welcome call were more likely to be uninterested in the service than participants who received other call types. Additionally, approximately 10% of participants interviewed indicated that the name of the program (“Medication Reminder Program”) was potentially confusing by giving the impression that calls might also be about non-steroidal breathing medications or about other medications entirely. This feedback led us to change the program name on all materials to “Breathe Easy Medication Reminder Program.” Additionally, feedback from the debrief calls, in combination with pilot call data, led us to discontinue the welcome call, as participants did not perceive it as useful or meaningful.

Discussion

This 3-stage developmental intervention process led to numerous valuable findings that helped to shape the final intervention used in the main randomized clinical trial. The findings from the focus groups, pilot test, and qualitative phone interviews provided understanding about how participants may interact and engage with IVR calls relying on speech recognition technology for the purposes of improving compliance with asthma medication.

To our knowledge, previous health studies [11;18;19] utilizing qualitative methods to assess participants’ use of, and reaction to, automated phone calls have not reported relying on developmental focus groups. The qualitative data we gathered in the developmental focus groups helped identify important areas of script design and content that would increase call acceptability. These data led us to modifications we would not have otherwise considered, such as immediately identifying the call source, making calls more regionally/culturally appropriate, determining which type of voice and call style would be most effective, identifying the call as automated, and including additional written information (e.g. descriptive letter/brochure sent prior to call receipt).

Studies [11;18;19] using qualitative methods to assess participants’ reactions to IVR calls have found participants react negatively to the call when: content is repeated or lacks new information; they feel talked down to; interactions are inflexible; they are not understood by the technology; they are too busy; and when the content and tone of the call make them feel guilty. Our qualitative data identified similar issues. However, because we conducted formative focus groups, pilot tested the calls, and conducted debrief interviews prior to finalizing the intervention, we were able to clearly identify and modify upfront words or phrases that were repetitious or condescending, content areas that felt redundant, and points of branching logic in the call that created confusion. The two-phased focus group approach allowed us to identify these issues early in the design phase, make adjustments, and then re-check the altered calls in the second round of focus groups. We were than able to further refine the calls based on issues raised in the qualitative debrief telephone interviews and patterns observed during the pilot test.

Two of the key findings from our focus groups as reported in the results section included the importance of identifying the call as automated and exploring the delivery style and tone of the call voice. In designing calls, we were uncertain as to whether stating upfront that the calls were automated or computerized would increase initial call hang up. Focus group data informed us that participants want the call to be identified as automated because the call voice and aspects of the interaction sound human, yet the yes/no branching logic of the calls creates a formality in speech that does not necessarily replicate human interaction. We addressed this concern by including descriptive language in our one-time “opt-out” portion of the call that made it clear the call was automated, and by providing a more detailed explanation and reasoning for utilizing automated calls (versus a live person) in the written recruitment/informational brochure.

Additionally, the recorded voice and delivery style of the call elicited the most negative reactions during the focus groups. The voice used on the call was particularly important in call acceptability for Hawaiian participants. Participants in Hawaii generally reacted more negatively to the recorded voice, perceiving it as unfriendly and condescending. Participants indicated they would likely hang up immediately if the current recorded voice was used because the voice did not sound like them, and lacked familiarity, friendliness, and credibility. Using a local voice in the phone recordings for Hawaii, along with customary local greetings, was an important design feature that increased call acceptability and meaningfulness for this population. We perhaps would not have considered recording the Hawaii-based calls in a voice reflective of the local culture had this not been identified as an issue during the focus groups.

We believe our qualitative data from the telephone debrief interviews with pilot participants helps demonstrate that participants’ behaviors during the calls (hanging up before a call conclusion) had less to do with call content or issues with the automated technology, and more to do with other issues. The IVR technology itself does not appear to be a primary reason for discontinuing a call; participants who indicated they already had a good reminder system in place for filling and taking their ICS prescription were the most likely to indicate they might decline future intervention calls. However, participants who were too busy to take the call or did not believe they received the call due to someone else in the household taking it on their behalf, were likely to indicate a willingness and desire for future IVR calls. Every participant we interviewed who reacted negatively to the computerized technology (7 out of the 64 interviewed), was older and cited hearing difficulties that made interacting with IVR phone technology challenging.

Quantitative pilot data informed us that shorter calls allowed for our key messages to reach more people, and that shortening time spent in the call introduction increases the rate of intervention delivery. In the pilot, 57% of those reached by the automated reminder call prematurely hung up (before the intervention content began) after replying, “yes” to “may I continue?” This larger-than-expected proportion of “early hang-ups” led us to take a close look at the importance and relevance of content in the “first call module.” These pilot data, along with the interview debrief data, led us to shorten the description of the program, remove the opportunity for active consent on the call, and remove the welcome call from the program, since its main purpose was to obtain consent. Initial results from the 5 months of calling rounds during the full PEANUT trial using these modified scripts show that the early hang-up rate dropped 37% (from 57% of reached members to 36%) and the “normal wrap-up” rate increased 73% (from 15% in the pilot to 26% in the trial) (Full outcome data from the PEANUT trial will be reported in a future manuscript).

Limitations

We believe that many of the patient recommendations and learnings from this developmental process will be useful to organizations relying on speech recognition-based interventions to manage chronic conditions, facilitate prevention, and improve patient care. However, there are a few limitations to consider. For example, the formative focus group data captures patient reaction to IVR calls within a formal research setting rather than a natural setting such as the participant’s home, which may have shaped participants’ reactions to calls. We attempted to address this issue by guiding and encouraging patients to respond honestly to the played-back IVR phone calls, but we cannot determine how well this strategy worked. Participants’, however, were quite forthcoming in their critique of the calls during the focus groups. We also acknowledge the possibility participants’ may have felt uncomfortable during the focus groups sharing all of the reasons they may not take or refill their asthma medication regularly. Additionally, participants’ responses to our focus group questions may have been shaped to please the interviewer. Thus, we may not have captured additional medication compliance barriers that could have been potentially addressed in the call content and structure. Overall, we conducted a large enough number of focus groups, with a range of participant ages, gender and disease experience, to generate sufficient thick and rich description to meet qualitative standards of data credibility and transferability. [20; 21]

Regarding the pilot calls, we chose to conduct debrief phone interviews with participants rather than in-person interviews; as such phone interviews are less likely to generate as in-depth explorations of topics as face-to-face interviews. Additionally, some of the positive views of the calls expressed by participants who did not complete a call or hung up early may be due to self-selection (being willing to talk with us) or a desire to be polite to the interviewer.

Two factors may have contributed to incomplete recall by participants about their experience with IVR calls. First, because uploading call data is time consuming, and we made multiple attempts to reach participants at home for interview availability, interviews did not occur until approximately 5 to 7 days after the participant received an IVR call. Given this, participants might not have thoroughly recalled the call content and/or their reactions to the call. Second, the interview was a one-time event conducted after a participant had received one of their first IVR calls. While we asked during the debrief phone interviews if participants would be willing to take future calls from the program, we do not know how participants might respond to multiple IVR calls over time. However, our formative focus group data provide some insight into this question. Participants indicated that, given familiarity with the call purpose and content, they might start to hang up on the call more frequently, or earlier in the call, but that this would not necessarily indicate that the IVR calls were not useful; a familiar IVR call could still be a reminder to take action (e.g., refill medication) without the person listening to the entire call. Focus group participants advised that changing the IVR call content, even slightly, as the program continued, may help to retain people’s interest in listening to most, if not the entirety, of a call.

Overall, researchers wishing to explore participant behavior and reactions to IVR calls may want to conduct debrief interviews closer to the time of receiving the call, and should consider conducting face-to-face interviews to elicit more detailed and accurate data. Future explorations should also obtain feedback on how participants respond and react to IVR calls over time. Additionally, future research identifying ways that IVR interventions can be made more practical, acceptable, and user-friendly to older populations would be extremely useful.

Conclusion

Formative and developmental processes such as focus groups, pilot testing, and qualitative debrief interviews—and modifications to an intervention based on what is learned in these evaluations—appear to improve the acceptability and usefulness of an IVR intervention utilizing speech recognition technology. From a public health perspective, there is considerable potential value in identifying large-scale, low-intensity interventions that achieve even small improvements in patients’ compliance with medications on a population basis. Automated phone interventions can serve such a purpose. Understanding how patients receive and interact with these types of interventions, and improving their acceptance and usefulness, can help increase their benefit.