Early Usability Assessment of a Conversational Agent for HPV Vaccination

Abstract

With the emerging use of speech technology in consumer goods, we experimented with the application of conversational agents for the communication of health information relating to HPV vaccine. Research have stated that one-to-one contact between providers and patients have a variety of positive influences on patients’ perception towards vaccines, even leading to uptake, compared to paper-based methods. We implemented a Wizard of Oz experiment that counsels adults with children (n=18) on the HPV vaccine, using an iPad tablet and dialogue script developed by public health collaborators, and for early testing of a prospective conversational agent in this area. Our early results show that non-vaccine hesitant parents believed that agent was easy to use and had capabilities needed, despite the desire for additional features. Our future work will involve developing a dialogue engine to provide automated dialogue interaction and future improvements and experimentation for the speech interface.

1. Introduction

Human papillomavirus (HPV) is a virus that causes several types of cancers including cervical, head and neck, anal, penile, vaginal, and vulvar. The primary preventive measure is the HPV vaccine which is recommended for all adolescents at 11–12 years of age. In the state of Texas, the vaccination coverage rates are low, at 33% (39.7% for girls; 26.5% for boys), compared to the national rate of 43%, yet both are still lower than the Healthy People 2020 target of 80% coverage [1].

Provider vaccine counseling appears to be the most effective educational method [2,3] and preferred by some patients [4], but the amount of time and resources needed to effectively communicate is an impediment on busy health care providers [5].

Nonetheless, speech is the most natural way for humans to communicate [6] and consume information [7]. Conversational user interfaces, or conversational agents (CA), is an emerging technology trend [8]. Harnessing CA for health communication could consistently and systematically perform dialogue tasks while alleviating the communication burden of the provider and supplement the delivery of health information, specifically related to the HPV vaccine. But how would potential end-users perceive it? We propose the following questions:

• How would parents with a child under 18 assess the usability (ease of use, efficiency, and expected capabilities) of a voice user interface for HPV vaccine counseling application?

• Would the parents’ vaccine hesitancy have an impact on the usability of an HPV vaccine conversational agents?

• What are the features and requirements that users desire in order to be feasible?

Our aim is to gather preliminary assessment knowledge of a vaccine-centric CA to help refine our idea of utilizing an automated CA for HPV counseling at a clinical environment. For our early assessment and data collection process, we employed the Wizard of OZ protocol that simulates speech interfaces with a potential user who thinks they are interacting with an automated machine or robot [9]. It allows us the evaluate the usability and acceptability of the system rather than to measure the quality of an entire system.

2. Material and Method

2.1. Dialogue Script

Dialogue Script. We utilized an HPV vaccine survey called the Carolina HPV Immunization Attitudes and Beliefs Scale (CHAIS)[10] that contains survey items categorized by the Health Belief Model, a behavioral change model that has been used in numerous vaccine intervention studies. This provided us with an initial baseline of talking points to communicate HPV vaccine information to prospective users, and possibly enough information that could nudge the participant towards vaccine uptake. For the time being, the latter is an area we will explore at a later study.

Each survey item from CHAIS communicated a piece of knowledge about the HPV vaccine. For each piece of knowledge, we developed dialogue utterances that initiated a discussion about that piece of information. For example, in the survey, there is a question that asks, “How effective do you think the HPV vaccine is in preventing cervical cancer?” The question expresses the notion that the HPV vaccine prevents cervical cancer effectively, which helped us construct dialogue that initiates the conversation by expanding on that concept, e.g. “If your child is vaccinated with the HPV vaccine it will protect against various HPV viruses which causes many precancerous and cancerous lesions in males and females.”

After developing the first draft of the script, we conferred with public health experts and experts who work directly with patients at Texas Children’s Hospital to refine the script. The working draft used for the study segmented the dialogue into different sections of the health belief model and included an introductory section that included small talk to segue into the main counseling section. From advisement of our experts, we avoided complex information that would be best handled by the users’ provider. Additionally, for potentially complicated discussion about dosage, we kept it brief and emphasized that the user should seek their provider for more information.

There was also a “pursuit” section in the event that participants did not want to talk about the vaccine. An example of pursuit might be a provider stating, “If (s)he were my child, I would definitely go ahead with the vaccination,” as opposed to just accepting parental resistance (in which no vaccine is given) or mitigating their original recommendations (e.g., delaying the vaccine). Previous studies have found that when physicians continued to pursue their original vaccination recommendations toward vaccine-hesitant parents (VHPs), significantly more VHPs ultimately accepted the physician’s vaccination recommendations [11,12]. This indicates the importance of persistence in vaccine counseling, and we incorporated it into our dialogue to maximize the likelihood that the users would become receptive to the HPV vaccination.

In addition, we developed a dialogue sequence to handle questions during the session. The answers for prospective questions were based on our previous work of creating a patient-centric vaccine ontology knowledge [13] from VIS documents - a paper flyer provided to patients at the time of inoculation . These answers would be provided in simple triples of subject, predicate, object format, e.g. “HPV vaccine has 3 doses”. For any question that was outside of the scope of the knowledge base we resorted to canned responses that encouraged the user to ask that question to their provider.

2.2. Software System

We developed an iPad and Mac platform application system to perform the Wizard of OZ experiment. The iPad application (Figure 1 (l)) served as the user-facing speech interface that the participant will interact. The tablet communicated with a separate laptop application through Bluetooth connectivity. The laptop application was a GUI tool that enabled the operator to copy and paste from the script or type out utterances (for unanticipated user responses) to the iPad (Figure 1 (r)). The text is transmitted directly to the tablet, where the tablet speaks, using text to speech (TTS), to the participant. The iPad tablet can capture the participant’s speech using offline speech recognition where it is transcribed and sent to the laptop application. We utilized offline speech recognition to ensure if there were any personal information spoken that it would not be transmitted to any external server. The GUI laptop tool collects a chat log of the interaction between participant and the software system and save it in a secured drive. The tablet application was developed using proprietary Apple’s iOS SDK and the desktop application was also developed using Apple’s MacOS SDK.

Figure 1.

(l) Tablet speech interface using the Siri voice profile, (r) desktop application that provides the tablet’s utterances. Both applications were connected peer-to-peer via Bluetooth.

2.3. Experimental Method

Our study was approved by UTHealth’s Internal Review Board¹ and conducted from February to July of 2018. Flyers were posted across the campus to advertise for participants. During that time period, we recruited 18 participants, who were adults with at least one child under age of 18. This is primarily because the HPV vaccine is an adolescent vaccine that is administered between the ages of 11 through 18, and the parent is the decision maker for the child. Each participant was escorted to a private room by the data collector assistant and completed a pre-assessment survey that included basic information about the subject and a Parent Attitudes about Childhood Vaccines (PACV) survey[14] that measures vaccine hesitancy. Adjacent to the room, another researcher, the operator, was seated with the desktop application and the dialogue script.

After completing the pre-assessment survey, the experiment started, and the participants went through the simulated automated counseling system with the operator coordinating the interaction through the guidance of the dialogue script. At the end of the simulated counseling session, we administered a usability survey voice user interfaces provided by [15] that had three questions pertaining to the aforementioned usability variables, and we collected the free text comments from the participants which were later segmented by positive and negative comments.

3. Results and Discussion

Out of the 18 participants, one refused further participation and one experienced technical difficulty, and overall, the final count of participants was 16. Of these 16 participants, 6 spoke English as a non-primary language. There was an equal number of healthcare professionals or researchers and non-healthcare professionals or researchers. 9 of the participants attended had a graduate degree. Most people have children below 10 years of age.

The PACV survey measured parent attitudes about childhood vaccines. Each answer had a value and a raw score was obtained for all of the questions. That score was then converted to a score that measured vaccine hesitancy. A score of 0–50 represents not vaccine hesitant, 50–80 represents vaccine hesitant, and 80–100 represents very vaccine hesitant. All 16 of the participants were considered not vaccine hesitant, which prohibited us to measure the impact of vaccine hesitancy on the usability of the voice interface.

On a scale from 1–7², there was an average score of 5.4 (σ =1.59) for ease of use and an average of 4.5 (σ =1.46) for the expected capabilities. Most of the participants relatively disagreed with the statement that the system was quick and efficient with an average score of 3.3 (σ =1.85) on a scale from 1–7 ². On average, participants relatively agreed that the system was easy to use and had the capabilities that they expected. However, the perceived slowness was attributed to how fast the remote operator responded to the user’s utterance.

The Pearson’s correlation coefficient (r) was used to quantify the linear correlation between two variables in order to investigate if the score of one variable affected the other in a linear fashion. Correlation coefficient is a continuous number ranging from −1 to 1, with −1 stands for perfectly negative linear relation, and 1 stands for perfectly positive linear relation. For interpretation purpose, we classify the absolute values of correlation coefficients of less than 0.30 as “small or no correlation”, values of [0.30, 0.50] as “weak correlation”, values of [0.50, 0.70] as “moderate correlation”, and values of [0.70, 1.00] as “strong correlation”. Also, we calculated the Fisher’s z’ transformation to obtain a normal distribution for our correlation calculation.

As a result of this correlation analyses, we found the weak correlation between the ease of use and the efficiency of the system with an estimate of 0.34 (95% CI [−0.18, 0.72], with two-sided p-value of 0.197), the moderate correlation between the ease of use and the expected capabilities of the system with an estimate of0.63 (95% CI [0.08, 0.82], p-value=0.024), and the moderate correlation between the efficiency and the expected capabilities with an estimate of 0.55 (95% CI [0.20, 0.86], p-value=0.007), where the 95% CI were computed by proper use of Fisher’s z transformation [16]. In summary, these analyses suggested that the score given for the expected capabilities has a moderate effect on the scores given for the ease of use and efficiency.

The written comments from the participants can be separated into two broad categories, negative and positive. The negative comments mainly concerned five things: the response time, the repetitiveness, the lack of visuals, the need to humanize the system, and the inability to answer all questions. The biggest concerns were regarding the response time and the repetitiveness. Most people wrote that the system needed to improve its response time and that it repeated the same points. People also stated that adding graphics would help with the overall look, improve interaction, and make it less uncomfortable. Some people also stated that they would prefer speaking to a person or a system that sounded more humanlike. The positive comments mainly regarded four things: the interactivity, how informative it is, the accessibility, and the clarity of the system. Most people enjoyed the interactive aspect of the system and how it understands them and encourages them to ask questions. Many people also stated that the system provided useful information on the HPV vaccine and reinforced the important points. Not only did the system offer useful information, it also stated the information clearly, had a clear voice, and was very straightforward and easy to operate.

While we received some encouraging positive comments, there were some useful suggestions that could lead to an improved user experience. The responsiveness issue was an important aspect. Because we were copying and pasting or typing responses to be transmitted wirelessly that may have had a slight latency impact on perceived responsiveness. It also highlights that if the system were to be automated and utilized artificial intelligent components, the system would need to be relatively quick in responding to the user’s utterance. Another was better usage of graphics or visualizations to complement the dialogue. Recently, we have been experimenting with the use of visualizations of emotions which could be used to augment the interface [17,18]. Lastly, the aim of the speech system is to alleviate some of the communication challenges at a clinical environment. Some of the users expressed a desire to speak to a human. Our belief is that the important discussion points, like personal contextual health information as it pertains to vaccines should be handled by the provider to avoid confusion. Throughout the dialogue, we emphasized that user should confer with the doctor for more nuanced and specific information, especially since we envision this conversational agent to be stationed as a kiosk or tablet in the waiting room.

4. Conclusion

From our study of 16 participants, who were not vaccine hesitant, we determined that parents found that a vaccine conversational agent was relatively easy to use and had the capabilities. Most participants found the agent to be slow and this is mainly due to responsiveness of the WOZ remote operator. Nonetheless, the feedback highlighted the importance for automated vaccine conversational agents to be responsive with utterances and with the suggestions from users, we have collected future features and improvements to better develop an automated CA. Currently, work is underway in developing an ontology-driven dialogue engine and lightweight question-answering component to answer natural language questions.