Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2015 Nov 20.
Published in final edited form as: Online J Commun Media Technol. 2015 Oct;5(4):184–203.

Usability Testing of an HPV Information Website for Parents and Adolescents

Randall Starling 1, Jessica A Nodulman 2, Alberta S Kong 3, Cosette M Wheeler 4, David B Buller 5, W Gill Woodall 6
PMCID: PMC4654461  NIHMSID: NIHMS671986  PMID: 26594313

Abstract

Objective

Parents make the decisions regarding their children’s health care. Unfortunately, many parents are misinformed about HPV and HPV vaccines. In order to help parents make an informed decision regarding HPV vaccination for their daughter, the GoHealthyGirls website was created for parents and their adolescent daughters. Usability testing was conducted with members of the target population to refine the website prior to conducting an efficacy trial.

Methods

Parents with girls (n=9) between the ages of 11-13 and 11-13 year old adolescents (n=10) were recruited for usability testing. The testing consisted of completing twelve scenarios where participants were asked to find specific information on the GoHealthyGirls site. This was followed by a self-administered system usability scale—to determine ease of use and functionality of the website—and a user satisfaction survey.

Results

Both adult and adolescent participants were able to easily find the requested information and reported an increased positive opinion of HPV vaccines after visiting the website. Both groups of participants reported favorable evaluations of using the website.

Conclusion

The GoHealthyGirls website has the potential to help parents of adolescent daughters make an informed decision about HPV vaccination. A large scale efficacy trial will determine its usefulness.

Keywords: HPV, Internet, Vaccines, Adolescents, Usability, Website

Introduction

Human Papillomavirus (HPV) is the most common sexually transmitted infection in the United States and throughout most of the world (Hariri et al., 2011; Gillison et al., 2012; Mandic 2012). Research has identified about 40 types of HPV that can result in health problems, including low-risk HPV strains that do not cause cancer but do cause genital warts, and high-risk HPV strains which can cause cancer, including cervical, anal, vaginal, oropharyngeal, and others (Schmeink et al., 2013; Chaturvedi et al., 2011; Jayaprakash et al, 2011; Watson et al., 2008). The Centers for Disease Control and Prevention (CDC) indicates that HPV causes about 360,000 cases of genital warts and 10,000 new cases of cervical cancers in women each year in the United States (CDC, 2014d) However, 21,000 HPV-associated cancers could be prevented annually in the Unites States through HPV vaccination(CDC, 2014d)

Two vaccines to prevent HPV infection have been approved by the United States’ Food and Drug Administration (FDA, 2014). These vaccines (Gardasil® and Cervarix®) have been shown to be very effective against high-risk HPV strains that account for most cervical and anal cancers. Gardasil® also provides protection against low-risk HPVs that account for most genital warts (Koutsky et al., 2002; CDC, 2014a). Gardasil® has been approved for use in both males and females, whereas Cervarix® is approved only for use in females (FDA, 2014). Both vaccines require three doses over a 6-month period for completion (CDC, 2014a; 2013) and both the Advisory Committee on Immunization Practices (ACIP) and the CDC recommend HPV vaccination routinely begin for all kids at ages 11-12 so that recipients have time to develop an immune response prior to becoming sexually active (CDC, 2014a; 2013). Healthy People 2020 (U.S. Department of Health and Human Services, 2014a) has set a target to increase 3-dose HPV vaccination coverage to 80% among 13-15 year old females, but has not set an HPV vaccination goal for males equivalent to that for females. However, the 2013 National Immunization Survey (CDC, 2014b) found that only approximately 26% of 13 year old females, 32% of 14 year old females, and 39% of 15 year old females in the United States received all three recommended doses of HPV vaccine. The survey also found that only about 12% of 13 year old males, 14% of 14 year old males, and 15% of 15 year old males received all 3 doses of the recommended HPV vaccine.

Typically, for minors less than 18 years old, health care decisions are made by someone else on their behalf, usually by their parents or a legal guardian (Macina, 2007; Zawistowski & Frader, 2003). Despite an increase in clinician recommendations to parents of adolescents for HPV vaccines (Darden et al., 2013; Vadaparampil et al., 2014; Hensley, 2012), many reasons have influenced parental decisions against this vaccination (CDC, 2014c; Darden et al. 2013; Jacobson, Roberts, & Darden, 2013; Trim, Nagji, Elit, & Roy, 2012; Gerend, Weibley, & Bland, 2009). For example, parents may believe that their adolescent daughter is too young for the vaccine (Waller, Marlow, & Wardle, 2006; Darden et al. 2013; Jacobson et al, 2013), that the vaccine is not necessary for either boys or girls (CDC, 2014c; Darden et al. 2013; Grabiel et al. 2013; Jacobson et al., 2013), or that the vaccination may encourage early sexual behavior (Luedtke, 2008). Other research reveals that parents may be misinformed about HPV and HPV vaccine side effects (CDC, 2014c; Grabiel et al. 2013; Trim et al., 2012; Lai, Tinker, & Cheung, 2013; Rambout, Tashkandi, Hopkins, & Tricco, 2014). Considering the slow uptake of HPV vaccines among both female and male adolescents, and misinformation or concerns associated with the vaccines, health communication researchers need to address information concerns regarding HPV vaccination among parents of adolescents (U.S. Department of Health and Human Services, 2012). Over 1,000 studies have documented problems consumers have understanding health information. Generally these problems occur because the information provided greatly exceeds the comprehension of the average U.S. adult (Neuhauser & Paul, 2011). As such, usability testing—a method of engaging users in a website's design and testing—is essential in order to make risk communication more understandable.

With funding from the National Institute on Allergies and Infectious Diseases, our research team developed an interactive website to be used by both parents of 11-13 year old girls and their daughters to help them make an informed decision about HPV vaccines. At the time the project was funded, the use of Gardasil® had not yet been approved for adolescent males so the focus was placed on adolescent girls and their parents. The process of developing the HPV vaccination website involved conducting separate focus groups with parents of girls and their 11-13 year old daughters to understand their information needs and preferences regarding HPV vaccine information. The results were used to develop the content in the theoretically-guided HPV vaccination website, GoHealthyGirls, which was designed to help parents of adolescent females and their daughters make an informed decision about receiving HPV vaccinations (Nodulman et al., 2014). The project website was then subjected to usability testing with adolescents aged 11-13 and parents of girls aged 11-13 to determine how well the website functioned for members of the target population. In this article, the results of usability testing with these two groups are reported and discussed.

Methods

Website Overview

The website contains four main modules. “Get Answers!”, discusses parent’s questions and concerns about HPV and HPV vaccines, including risks/benefits, and social and health organizations that recommend HPV vaccines. The second module, “Let’s Talk!,” addresses ways for parents to communicate with their daughter, how to talk with other family members, and advice on how to talk with their physician about HPV vaccination. The third module, “Info for Teens”, was designed specifically for adolescent users and contains an interactive quiz show game regarding HPV and HPV vaccines, a texting simulation, and a Frequently Asked Question area specifically for adolescent users. The last module, “Preteen Vaccine”, contains an adolescent vaccination schedule, a section where users can put in their zip code to find clinics offering HPV vaccines in their area, and an email reminder system to remind parents of HPV vaccination follow-up appointments. Additional description of GoHealthyGirls may be found in Starling et al., 2014a.

Procedures

The purpose of usability testing the GoHealthyGirls website was to gather opinions and user feedback about the website from members of the target group of decision makers about HPV vaccination: parents of 11-13 year old adolescents. GoHealthyGirls was developed primarily for parents since research shows they generally make health decisions for their children (Macina, 2007; Zawistowski et al., 2003), but includes information targeted to adolescent females; therefore, we included usability testing with 11-13 year old adolescent females as well. Further, we included adolescent males in the usability testing to gain their insight into how a similar website might function for them as well, given that Gardasil® was recently approved for boys. Prior to conducting any research activities, approval was obtained from the University of New Mexico Institutional Review Board (IRB) and informed consent was obtained from adults and informed assent from adolescents. Eligible parents did not have to enroll in the study with their eligible child in order to participate. Participants were a convenience sample recruited through program contacts in New Mexico and Colorado and through a middle school outside of a metropolitan area in New Mexico. Testing was conducted individually between a trained member of the research staff and a participant in Albuquerque, NM and Golden, CO in late 2012.

Usability testing was conducted via 1) protocol analysis (Zimmerman, Akerelrea, Buller, Hau, & LeBlanc, 2003), 2) a self-report on a system usability scale (Bangor, Kortumb, & Miller, 2008; 2009; Bangor, Joseph, Sweeney-Dillon, Stettler, & Pratt, 2013), followed by 3) survey questions on user satisfaction. Participants met individually with a trained member of the research project to complete the testing which generally ran for 45 minutes to an hour. Parents were asked two screening questions prior to participating: “Do you have a daughter between the ages of 11-13?” and “Has your daughter ever received any dose of an HPV vaccine?” Only those who responded yes to the first question and no to the second question were allowed to participate. Participating adolescents were asked two screening question prior to participating. They were asked, “Are you between the ages of 11-13?” and “Have you ever received a dose of an HPV vaccine?” Adolescent participants that were between the ages of 11-13 and had not received any HPV vaccine were eligible for participation in the study. Both adult and adolescent participants received a $30 gift certificate for their participation in the study.

Protocol Analysis

To begin testing, the research team member met individually with each participant and explained the HPV informational website project and showed the participant the website homepage. Participants were then provided with 12 scenarios—worded for adults and again for adolescents— related to finding information on the website; e.g., for the parent, “where on the website would you go to find out if your child can get the vaccine?” or for the adolescent, “where on the website would you go to find out if you can get the vaccine?” Participants were encouraged to talk out loud while they searched for the answer to the prompts. The research members wrote down participants’ statements, reactions, and how s/he navigated through the website for each of the prompted scenarios. See Table 1 for the list of scenarios.

Table 1.

Protocol Analysis Scenarios

1.GoHealthyGirls is a website designed to help young women and their parents to get vaccinated for HPV. Knowing this, what kinds of information would you expect to find on the website? 7. Let’s say you aren’t sure how to go about getting the HPV vaccine for your child (or yourself, for adolescent participants). Where on the website would you go to get help?
2. Where would you go to find out if your child (or you, for adolescent participants) can get the vaccine? 8. Imagine you want to talk to your child (or parent, for adolescent participants) about getting the HPV vaccine but aren’t sure how to go about it. Can you find help with this on the website?*
3. Pretend that you are wondering if the vaccination is the right choice for your child (or you, for adolescent participants). Where would you go first? 9. Imagine you were curious about whether the HPV vaccine had been tested and whether it was safe. Where would you find this information on the website?
4. Let’s say that you have heard that the vaccination can cause some side effects. Where would you go to find out more about potential side effects? 10. If you had never heard of HPV and were not sure what negative outcomes (i.e., diseases) result from being infected with it, can you find this information on the website?
5. Pretend you have an appointment set to see your doctor, but you aren’t sure how to talk to them about the HPV vaccine. Where would you go to get help with this? 11. There are probably a lot of organizations whose opinions you’d like to know about the HPV vaccines. Where would you go to find out whether these organizations approve of the HPV vaccines?
6. Where would you go to get help discussing HPV and the vaccine with your child (or parent, for adolescent participants)? 12. Image you wanted to know the benefits of the HPV vaccine. Where would you find this information on the website?
Open in a new tab

System Usability Scale and User Satisfaction

Developed by Brooke (1996), the system usability scale (SUS) provides a reliable assessment of subjective usability of various products and services (Bangor et al., 2008; 2009; 2013). Containing only 10 items, it is a very easy scale to administer and can be used on small sample sizes to differentiate between usable and unusable systems (U.S. Department of Health and Human Services, 2014b). Questions regarding user satisfaction with the website were developed by the research team members and attached to the SUS.

Measures

Protocol analysis scenarios were developed by trained research staff by adapting usability testing scenarios used in previous projects (Buller et al., 2013; Starling et al., 2014b). The research staff member would count how many sections of the website participants went to in order to complete each scenario. Upon completing protocol analysis testing, the SUS was administered immediately to participants. Participants answered the 10-item SUS, using Likert-type response categories (1=strongly agree to 5=strongly disagree).The SUS has repeatedly been demonstrated as a reliable and valid instrument (U.S. Department of Health and Human Services, 2014b).

In addition to the SUS, participants also provided information regarding their satisfaction with the website. The questions consisted of a 7-point Likert-type question on the user-friendliness of the product (1=worst imaginable to 7=best imaginable) followed by open ended questions regarding what each participant liked most/least about the website, what they learned from the website, and whether the website influenced their opinion about HPV vaccines. The qualitative questions were developed by the research staff for the project by adapting questions used in other website usability testing (Buller et al., 2013; Starling et al., 2014b). Data from the SUS and quantitative portion of the user-satisfaction items were then entered into SPSS statistics software version 19. Information from protocol analysis, SUS, and user-satisfaction questions were used to further refine GoHealthyGirls prior to conducting a beta-test of the website.

Statistical Analysis

Data were analyzed employing SPSS version 19. Basic descriptive statistics of participant’s efforts during the quantitative portion of GoHealthyGirls website usability testing were calculated.

Results

Participants

Nineteen people (n=19, 79% female, 21% male) participated in usability testing of GoHealthyGirls; nine parents of 11-13 year old adolescents (n=9) and ten 11-13 year old adolescents (n=10, mean age=12.7). Approximately 79% of the participants self-identified as Anglo, 10%, as Native American, and 10% considered themselves Hispanic or Latino. Specific ages of adult participants were not recorded. Table 2 reflects the ethnicity and gender of participants.

Table 2.

Participant Demographics

Race/Ethnicity Parents Adolescents Total
Male N (%) Female N (%) Male N (%) Female N (%)
Do you consider yourself to be Hispanic? 0 1 (5.2) 1 (5.2) 0 2 (10.5)
What race do you consider yourself to be?
Native American 0 0 2 (10.5) 0 2 (10.5)
Asian 0 0 0 0 0
Black 0 1 (5.2) 0 0 1 (5.2)
Native Hawaiian 0 0 1 (5.2) 0 1 (5.2)
White 1 (5.2) 7 (36.8) 0 7 (36.8) 15 (78.9)
Total 1 (5.2) 8 (42.1) 3 (15.7) 7 (36.8) 19
Open in a new tab

Parents were asked how long they had been using the Internet to which all replied “more than 5 years.” They were also asked how comfortable they were using websites, to which all replied “very comfortable.” Age and socioeconomic status of parents was not recorded. Among adolescent participants, five participants (50%) reported they had been using the Internet 1-5 years, whereas the other participants reported they had been using it more than 5 years. Six participants (60%) reported that they were “very comfortable” using websites while 4 participants (40%) reported that they were “somewhat comfortable.”

Protocol Analysis

Protocol analysis testing, a test to see how participants navigated through the website to find specific content, took approximately 45 minutes for each participant to complete. All parents and adolescents answered each scenario (Table 1), except scenario seven where one adolescent did not complete the task, and scenario 8 (“Imagine you want to talk to your child (or parent) about getting the HPV vaccine but aren’t sure how to go about it. Can you find help with this on the website?”), where no adolescents, and only four adults, completed it due to the perceived similarity to scenario 6 (“Where would you go to get help discussing HPV and the vaccine with your child (or parent)?”. Scenario one (“GoHealthyGirls is a website designed to help young women and their parents to get vaccinated for HPV. Knowing this, what kinds of information would you expect to find on the website?”) did not ask for participants to complete a task, but rather briefly explain what information they believed would be available. Parental participants typically answered with “What the HPV vaccine is for,” “Risks and benefits of vaccine,” “Where to get the vaccine,” and “Ages to get the vaccine.” Adolescent participants answered with “What HPV is,” “What the vaccine is for,” and “How it works.”

The protocol analysis revealed that participants had little trouble finding information requested in each scenario. Adult participants demonstrated the most difficulty with scenario 4 (“Where would you go to find out more about potential side effects?”). Participants went through almost three sections (M=2.90) in order to find the information. Adult participants were able to find the requested information in other scenarios within two sections of the website. Similar to adults, the majority of adolescents found the requested information within the first two sections they examined. A few problems were uncovered, such as one adult participant discovered that she could not get back to the home page from the FAQ section and another adult participant had trouble locating descriptive pictures in a website area where one may click a picture for additional information. Table 3 shows the scenarios and the mean number of sections participants went through to complete the each requested task.

Table 3.

Protocol Analysis Scenario Results

Scenario Parents Adolescents
1.GoHealthyGirls is a website designed to help young women and their parents to get vaccinated for HPV. Knowing this, what kinds of information would you expect to find on the website? NA NA
2. Where would you go to find out if your child (or you, for adolescent participants) can get the vaccine? 3.2 1.6
3. Pretend that you are wondering if the vaccination is the right choice for your child (or you, for adolescent participants). Where would you go first? 2.00 1.6
4. Let’s say that you have heard that the vaccination can cause some side effects. Where would you go to find out more about potential side effects? 2.90 1.6
5. Pretend you have an appointment set to see your doctor, but you aren’t sure how to talk to them about the HPV vaccine. Where would you go to get help with this? 1.90 1.6
6. Where would you go to get help discussing HPV and the vaccine with your child (or parent, for adolescent participants)? 1.30 1.2
7. Let’s say you aren’t sure how to go about getting the HPV vaccine for your child (or yourself, for adolescent participants). Where on the website would you go to get help? 1.50 1.2
8. Imagine you want to talk to your child (or parent, for adolescent participants) about getting the HPV vaccine but aren’t sure how to go about it. Can you find help with this on the website?* .90 .00
9. Imagine you were curious about whether the HPV vaccine had been tested and whether it was safe. Where would you find this information on the website? 1.80 1.8
10. If you had never heard of HPV and were not sure what negative outcomes (i.e., diseases) result from being infected with it, can you find this information on the website? 1.60 1.4
11. There are probably a lot of organizations whose opinions you’d like to know about the HPV vaccines. Where would you go to find out whether these organizations approve of the HPV vaccines? 1.20 1.4
12. Image you wanted to know the benefits of the HPV vaccine. Where would you find this information on the website? 1.70 1.2
Open in a new tab
*

Only 4 adults and no adolescents completed this task due to redundancy to scenario 6

SUS

After completing the protocol analysis, participants were asked to complete the system-usability scale (SUS). Participants gave favorable evaluations of the GoHealthyGirls website on the SUS (Table 4). Specifically, both parents and adolescents felt it was easy to use, most people could learn to use it quickly, it could be used without technical support from someone else, and they were confident using it. They also perceived the functions of the website were consistent and well integrated.

Table 4.

Results of System Usability Testing Survey

Usability Questions Parents (n=9) Adolescents (n=10) Combined (N=19)
I think that I would like to use this product frequently 2.25 2.33 2.26
I found the product unnecessarily complex 4.08 4.44 4.32
I thought the product was easy to use 2.00 1.44 1.74
I think that I would need the support of a technical person to be able to use this product 4.91 5.00 4.95
I found the various functions in the product were well integrated 2.45 2.33 2.37
I thought there was too much inconsistency in this product 4.45 4.11 4.26
I imagine that most people would learn to use this product very quickly 2.18 1.22 1.79
I found the product very awkward to use 4.09 4.00 4.11
I felt very confident using the product 2.55 1.67 2.11
I needed to learn a lot of things before I could get going with this product 4.82 4.67 4.74
Open in a new tab

User Satisfaction with the Website

After completing the SUS, participants were asked to rate the user friendliness of the website on a 7 point Likert-type scale from 1=Awful to 7=Best Imaginable. An average among all participants resulted in approximately a 6 (Excellent; adults M=5.8, adolescents M=6.0). Participants were also asked “Did the website influence your opinion about the HPV vaccine?” Overall, 68% of participants (60% of adults and 89% of adolescents) responded “yes”. When asked how it had influenced them, 50% of adults stated that they were more inclined to have their daughter vaccinated after viewing GoHealthyGirls, 33% said they learned more about vaccine safety, and 40% reported learning the vaccination schedule. Among adolescents, 12% said they learned about HPV vaccine safety from the website, 62% said they learned the purpose of the vaccine, and 25% said the website influenced them to tell their friends to get the vaccine.

During the qualitative section of the user-satisfaction portion, participants recommended very few changes to the website. Adolescent participants suggested that the title of the adolescent girls section of the website be changed from “For Girls” to “Info for Teens” so that adolescents would understand that that section of the website is designed for them. Adolescents also stated that sound effects in the website were louder than voice levels of the actors in the website. Adult participants suggested that the Frequently Asked Questions (FAQs) page should include hyperlinks that direct users to additional information within relative areas of the website. They also suggested that the “What is HPV?” hyperlink should be established as “Get Answers” in order to help users better understand that they could find a variety of information about HPV and HPV vaccines in that section.

The results of the usability testing were summarized and discussed by the members of the research team. The research team agreed with the changes requested by the participants and these changes were delivered to the project web-development team. The suggested changes were incorporated into the website by the web-developers, and the research team verified the changes had been successfully made. The information gained from usability testing was used in order to refine the website prior to initiating beta-testing and a later project efficacy trial.

Discussion

Usability testing of GoHealthyGirls found many compelling results. Testing was divided into multiple parts. First, participants completed a protocol analysis which consisted of completing hypothetical scenarios on the website. Next, they completed a system-usability scale for the website. Last, they answered a few brief open and closed- ended questions regarding their satisfaction with the website and influence it had on them. Protocol analysis revealed that participants were mostly able to easily find the requested information with very few problems reported. Overall, adolescent participants were able to find the requested information from scenarios quicker than the adult participants.

The protocol analysis portion of the usability testing of GoHealthyGirls was significant because it demonstrated areas of the website that participants could easily navigate to and those that they had more difficulty finding. Due to the protocol analysis, our research team was able to find what areas proved difficult to participants, and we were then able to make the necessary changes to better the site. It was also interesting that adolescents were able to find information quicker than adults. All of the adult participants reported that they were “very comfortable” using the Internet whereas 40% of teens reported being only “comfortable” using the Internet; however, teens found the information faster. This may be due to parents having more cognitive complexity, therefore, thinking there could be many different options to solve a task, whereas adolescents went with the first option they encountered.

Results from the system-usability scale (SUS) yielded positive evaluations—ease and confidence of use, ability to learn information quickly, consistent and integrated website design— from participants. These findings add to previous research (Atkinson et al., 2009; Pak, Prince, & Thatcher, 2009) that highlight the importance of testing health webpages prior to implementation to make sure they meet the needs of the consumer. By administering the open and closed-ended user satisfaction questionnaire, we were able to supplement the favorable results from the SUS with more specific information about users’ opinions of the website. For example, participants’ indicated that the website positively influenced their opinion of HPV vaccines. Most participants stated that they learned that HPV vaccines can help prevent genital warts and cervical cancer, in addition to learning the HPV vaccination schedule. Many others stated they learned more about HPV itself and what it does.

The goal of usability testing is to learn both the positive aspects to the design of a site, but also uncover any challenges users faced and learn how participants would recommend changing the site. Participants recommended a few changes to the website. Adolescents suggested changing the title of the section for them and smoothing out the sound levels in another section. Adults suggested adding hyperlinks in the FAQs so that additional information could be easily obtained. They also suggested changing the title of one section. As there were few suggestions for changing the website from participants, making the changes was relatively easy.

The study confirmed the research by Neuhauser et al. (2011) in using usability testing with targeted consumers in designing and modifying health information websites. Many other health information websites that have conducted usability testing confirm the usefulness of this tactic (Hinchliffe & Mummery, 2008; Korn, Murray, Morrison, Reynolds, & Skinner, 2006; Kushniruk et al., 2014; Johnson & Shaw, 2012; Choi & Bakken, 2006). Failure to incorporate usability testing with members of the targeted demographic prior to launching can result users feeling lost when navigating the website (Carusi & Alvao, 2012; de Matos et al., 2013).

Limitations

Although results appeared positive, study limitations should be considered. One significant limitation is sample size, which is often the case with usability testing since it is time-intensive and employs qualitative and quantitative methods. The small sample also limits generalizability, as did lack of diversity in the sample. Participants were selected based on convenience sampling which did not provide for a highly diverse ethnic and/or racial group of participants. Lastly, a possible social desirability effect may have occurred during testing. As usability testing was conducted individually with participants and a research team member, the possibility exists that the participant wanted to respond in a positive manner to the SUS and user-satisfaction questionnaire as it was completed in front of the research team member.

Suggestions for Future Research

The GoHealthyGirls website is now ready for evaluation and the efficacy trial has been launched. We propose to conduct a randomized controlled trial of 550 New Mexico parents of adolescent girls who have not received a dose of HPV vaccine. Half of the parents will receive the website and the other half will receive an electronic PDF about HPV and HPV vaccines. As our trial specifically targets parents of adolescent girls, future research could include using technological methods to address HPV vaccine informed decision making among parents of adolescent males and their sons. As technology becomes more mobile (i.e., smartphones, tablets/iPads), additional research could include using advanced technology to target both adolescent boys and girls, in addition to their parents.

The information gleaned from usability testing of GoHealthyGirls has been incorporated into the website for the upcoming project efficacy trial. The project efficacy trial will examine the impact of the website on actual uptake of HPV vaccines among participants. An opportunity exists to use technology to help parents make informed decisions regarding HPV vaccines for their adolescent children, and this study provides an important glimpse into how researchers and practitioners can test the usability of such technology to better deliver health communication efforts in a mediated context.

Acknowledgments

The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Allergies and Infectious Diseases Institute or the National Institutes of Health. This research is funded by the U.S. National Institutes of Health (U19 AI084081).

Contributor Information

Randall Starling, Center on Alcoholism, Substance Abuse, and Addictions University of New Mexico, 2650 Yale Blvd. SE MSC11-6280, Albuquerque, NM 87106, Phone: (505) 925-2318, Fax: (505) 925-2301, starling@unm.edu.

Jessica A. Nodulman, Department of Communication Studies, Augustana College, 639 38th Street, Rock Island, IL 61201, Phone: 217-721-8998, jessicanodulman@augustana.edu.

Alberta S. Kong, Department of Pediatrics, University of New Mexico School of Medicine, Dept. of Pediatrics, Division of Adolescent Medicine, MSC10 5590 1 UNM, Albuquerque, NM 87131, Phone: 505-272-5152, Fax: 505-272-6845, akong@salud.unm.edu.

Cosette M. Wheeler, Center for HPV Prevention, University of New Mexico Health Sciences Center, Department of Pathology, MSC08 4640, Albuquerque, NM 87131, Phone: 505-272-9151, Fax: 505-277-0265, cwheeler@salud.unm.edu.

David B. Buller, Klein Buendel, Inc., 1667 Cole Suite 225, Golden, CO 80401, 303-565-4340, 303-565-4320, dbuller@kleinbuendel.com.

W. Gill Woodall, Center on Alcoholism, Substance Abuse, and Addictions, University of New Mexico, 2650 Yale Blvd. SE MSC11-6280, Albuquerque, NM 87106, Phone: (505) 925-2316, Fax: (505) 925-2301, gwoodall@unm.edu.

References

  1. Atkinson NL, Saperstein SL, Desmond SM, Gold RS, Billing AS, Tian J. Rural eHealth nutrition education for limited-income families: An interative and user-centered design approach. Journal of Medical Internet Research. 2009;11(2):e21. doi: 10.2196/jmir.1148. [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Bangor A, Kortumb PT, Miller JT. An empirical evaluation of the System Usability Scale. International Journal of Human-Computer Interaction. 2008;24(6):574–594. [Google Scholar]
  3. Bangor A, Kortum P, Miller J. Determining what individual SUS scores mean: adding an adjective rating scale. Journal of Usability Studies. 2009;4(3):114–123. [Google Scholar]
  4. Bangor A, Joseph K, Sweeney-Dillon M, Stettler G, Pratt J. Using the SUS to help demonstrate usability's value to business goals. Proceedings of the Human Factors and Ergonomics Society Annual Meeting; 2013. pp. 202–205. [Google Scholar]
  5. Brooke J. SUS: a "quick and dirty" usability scale. In: Jordan PW, Thomas B, Weerdmeester BA, McClelland AL, editors. Usability Evaluation in Industry. London: Taylor and Francis; 1996. [Google Scholar]
  6. Buller DB, Berwisk M, Shane J, Kane I, Lantz K, Buller MK. User- centered development of a smart phone mobile application delivering personalized real-time advice on sun protection. Translational Behavioral Medicine. 2013;3(3):326–334. doi: 10.1007/s13142-013-0208-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Carusi A, Alvao CM. Usability testing with children: An application of Pedactice and Ticese methods. Work. 2012;41:822–826. doi: 10.3233/WOR-2012-0248-822. [DOI] [PubMed] [Google Scholar]
  8. Centers for Disease Control and Prevention. FDA Licensure of Bivalent Human Papillomavirus Vaccine (HPV2, Cervarix) for Use in Females and Updated HPV Vaccination Recommendations from the Advisory Committee on Immunization Practices (ACIP) 2010 May 28; Retrieved June 20, 2014, from http://www.cdc.gov/mmwr/preview/mmwrhtml/mm5920a4.htm. [PubMed]
  9. Centers for Disease Control and Prevention. Human Papillomavirus: HPV vaccines. 2013 Feb 5; Retrieved August 2, 2014, from http://www.cdc.gov/hpv/vaccine.html.
  10. Centers for Disease Control and Prevention. HPV vaccine – Questions & Answers. 2014a (2014, August 6). Retrieved August 21, 2014, from http://www.cdc.gov/vaccines/vpd-vac/hpv/vac-faqs.htm.
  11. Centers for Disease Control and Prevention. National and state vaccination coverage among adolescents aged 13-17 Years – United States, 2013. 2014b (2014, July 25). Retrieved August 25, 2014, from http://www.cdc.gov/mmwr/preview/mmwrhtml/mm6329a4.htm.
  12. Centers for Disease Control and Prevention. Barriers to human papillomavirus vaccination among adolescents in the United States. 2014c Retrieved August 25, 2014 from http://www.cdc.gov/cancer/hpv/what_cdc_is_doing/hpv-barriers.htm.
  13. Centers for Disease Control and Prevention. Genital HPV infection - fact sheet. 2014d Retrieved August 25, 2014, from http://www.cdc.gov/std/hpv/stdfact-hpv.htm.
  14. Chaturvedi AK, Engels EA, Pfeiffer RM, Hernandez BY, Xiao W, Kim E, et al. Human papillomavirus and rising oropharyngeal cancer incidence in the United States. Journal of Clinical Oncology. 2011;29(32):4294–4301. doi: 10.1200/JCO.2011.36.4596. [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Choi J, Bakken S. Heuristic evaluation of a web-based educational resource for low literacy NICU parents. Studies in Health Technology and Informatics. 2006;122:194–199. [PubMed] [Google Scholar]
  16. Darden PM, Thompson DM, Roberts JR, Hale JJ, Pope C, Naifeh M, Jacobson RM. Reasons for not vaccinating adolescents: National Immunization Survey of Teens, 2008-2010. Pediatrics. 2013;131(4):645–651. doi: 10.1542/peds.2012-2384. [DOI] [PubMed] [Google Scholar]
  17. de Matos P, Cham JA, Cao H, Alcantara R, Rowland F, Lopez R, Steinbeck C. The Enzyme Portal: a case study in applying user-centred design methods in bioinformatics. BMCBioinformatics. 2013;14:103. doi: 10.1186/1471-2105-14-103. [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Gerend MA, Weibley E, Bland H. Parental response to human papillomavirus vaccine availability: Uptake and intentions. Journal of Adolescent Health. 2009;45(5):528–531. doi: 10.1016/j.jadohealth.2009.02.006. [DOI] [PubMed] [Google Scholar]
  19. Gillison ML, Broutian T, Pickard RK, Tong Z, Xiao W, Kahle L, et al. Prevalence of oral HPV infection in the United States, 2009 2010. Journal of the American Medical Association. 2012;307(7):693–703. doi: 10.1001/jama.2012.101. [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Grabiel M, Reutzel TJ, Wang S, Rubin R, Leung V, Ordonez A, et al. HPV and HPV vaccines: The knowledge levels, opinions, and behavior of parents. Journal of Community Health. 2013;38(6):1015–1021. doi: 10.1007/s10900-013-9725-6. [DOI] [PubMed] [Google Scholar]
  21. Hariri S, Unger ER, Sternberg M, Dunne EF, Swan D, Patel S, Markowitz LE. Prevalence of genital human papillomavirus among females in the United States, the National Health and Nutrition Examination Survey, 2003–2006. Journal of Infectious Diseases. 2011;204(4):566–573. doi: 10.1093/infdis/jir341. [DOI] [PubMed] [Google Scholar]
  22. Hensley S. National Public Radio. Pediatricians recommend HPV vaccination for boys. 2012 Feb 27; Retrieved August 25, 2014, from http://www.npr.org/blogs/health/2012/02/27/147508943/pediatricians-recommend-hpv-vaccination-for-boys.
  23. Hinchliffe A, Mummery WK. Applying usability testing techniques to improve a health promotion website. Health Promotion Journal of Australia : Official Journal of Australian Association of Health Promotion Professionals. 2008;19(1):29–35. doi: 10.1071/he08029. [DOI] [PubMed] [Google Scholar]
  24. Jacobson RM, Roberts JR, Darden PM. Parents' perceptions of the HPV vaccine: A key target for improving immunization rates. Expert Review of Clinical Immunology. 2013;9(9):791–793. doi: 10.1586/1744666X.2013.824673. [DOI] [PubMed] [Google Scholar]
  25. Jayaprakash V, Reid M, Hatton E, Merzianu M, Riqual N, Marshall J, et al. Human papillomavirus types 16 and 18 in epithelial dysplasia of oral cavity and oropharynx: A meta-analysis, 1985 2010. Oral Oncology. 2011;47(11):1048–1054. doi: 10.1016/j.oraloncology.2011.07.009. [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Johnson CM, Shaw RJ. A usability problem: Conveying health risks to consumers on the Internet. AMIA Symposium Proceedings, 2012; 2012. pp. 427–435. [PMC free article] [PubMed] [Google Scholar]
  27. Korn D, Murray M, Morrison M, Reynolds J, Skinner HA. Engaging youth about gambling using the Internet: The YouthBet.net website. Canadian Journal of Public Health. 2006;97(6):448–453. doi: 10.1007/BF03405226. [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Koutsky LA, Ault KA, Wheeler CM, Brown DR, Barr E, Alvarez FB, et al. A controlled trial of a human papillomavirus type 16 vaccine. New England Journal of Medicine. 2002;347(21):1645–1651. doi: 10.1056/NEJMoa020586. [DOI] [PubMed] [Google Scholar]
  29. Kushniruk A, Kaipio J, Nieminen M, Hyppönen H, Lääveri T, Nohr C, et al. Human factors in the large: Experiences from Denmark, Finland and Canada in moving towards regional and national evaluations of health information system usability. Contribution of the IMIA Human Factors Working Group. Yearbook of Medical Informatics. 2014;9(1):67–81. doi: 10.15265/IY-2014-0023. [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Lai JY, Tinker AV, Cheung WY. Factors influencing the willingness of US women to vaccinate their daughters against the human papillomavirus to prevent cervical cancer. Medical Oncology. 2013;30(2):582. doi: 10.1007/s12032-013-0582-z. [DOI] [PubMed] [Google Scholar]
  31. Luedtke S. Human papillomavirus vaccine: Are the concerns unfounded? American Journal of Health-System Pharmacy. 2008;65(22):2150–2151. doi: 10.2146/ajhp080392. [DOI] [PubMed] [Google Scholar]
  32. Macina RP. Health care decisions in minors: Ethical dilemmas. The Journal of Lancaster General Hospital. 2007;2(1):25–26. [Google Scholar]
  33. Mandic A. Primary prevention of cervical cancer: Prophylactic human papillomavirus vaccines. Journal of the Balkan Union of Oncology. 2012;17(3):422–427. [PubMed] [Google Scholar]
  34. Neuhauser L, Paul K. Readablity, Comprehension, and Usability. In: Fischhoff B, Brewer NT, Downs NT, editors. Communicating Risks and Benefits: An Evidence-based User's Guide. Silver Spring, MD: Food and Drug Administration; 2011. [Google Scholar]
  35. Nodulman JA, Starling R, Kong AS, Buller DB, Wheeler CM, Woodall WG. Investigating stakeholder attitudes and opinions on school-based HPV vaccination programs. Manuscript submitted for publication to. Journal of School Health. 2014 doi: 10.1111/josh.12253. [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. Pak R, Price MM, Thatcher J. Age-sensitive design of online health information: Comparative usability study. Journal of Medical Internet Research. 2009;11:e45. doi: 10.2196/jmir.1220. [DOI] [PMC free article] [PubMed] [Google Scholar]
  37. Rambout L, Tashkandi M, Hopkins L, Tricco AD. Self-reported barriers and facilitators to preventive human papillomavirus vaccination among adolescent girls and young women: A systematic review. Preventive Medicine. 2014;58:22–32. doi: 10.1016/j.ypmed.2013.10.009. [DOI] [PubMed] [Google Scholar]
  38. Schmeink CE, Massuger LF, Lenselink CH, Quint WG, Witte BL, Melchers WJ, Bekkers RL. Prospective follow-up of 2,065 young unscreened women to study human papillomavirus incidence and clearance. International Journal of Cancer. 2013;133(1):172–181. doi: 10.1002/ijc.27986. [DOI] [PubMed] [Google Scholar]
  39. Starling R, Nodulman JA, Kong AS, Wheeler CM, Buller DB, Woodall WG. Beta-test results for an HPV information website: GoHealthyGirls.org–Increasing HPV vaccine uptake in the United States. Journal of Consumer Health on the Internet. 2014a;18(3):226–237. doi: 10.1080/15398285.2014.931771. [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. Starling R, Helme D, Nodulman JA, Bryan A, Buller DB, Donohew RL, Woodall WG. Testing a risky sex behavior intervention pilot website for adolescents. Manuscript accepted for publication to California Journal of Health Promotion. 2014b [PMC free article] [PubMed] [Google Scholar]
  41. Trim K, Nagji N, Elit L, Roy K. Parental knowledge, attitudes, and behaviours towards human papillomavirus vaccination for their children: A systematic review from 2001 to 2011. Obstetrics & Gynecology International, 2012. 2012 doi: 10.1155/2012/921236. 921236. [DOI] [PMC free article] [PubMed] [Google Scholar]
  42. U.S. Department of Health and Human Services. Office of Disease Prevention and Health Promotion. Healthy People 2020. 2014a Aug 14; Retrieved August 25, 2014, from http://www.healthypeople.gov/2020/default.aspx. [PubMed]
  43. U.S. Department of Health and Human Services. System usability scale. 2014b Aug 25; Retrieved August 25, 2014 from http://www.usability.gov/how-to-and-tools/methods/system-usability-scale.html.
  44. U.S. Food and Drug Administration. HPV (human papillomavirus) 2014 Apr 24; Retrieved August 1, 2014, from http://www.fda.gov/forconsumers/byaudience/forwomen/ucm118530.htm.
  45. Vadaparampil ST, Malo TL, Kahn JA, Salmon DA, Lee JH, Quinn GP, et al. Physicians’ human papillomavirus vaccine recommendations, 2009 and 2011. American Journal of Preventive Medicine. 2014;46(1):80–84. doi: 10.1016/j.amepre.2013.07.009. [DOI] [PMC free article] [PubMed] [Google Scholar]
  46. Waller J, Marlow LAV, Wardle J. Mothers' attitudes towards preventing cervical cancer through human papillomavirus vaccination: A qualitative study. Cancer Epidemiology, Biomarkers & Prevention. 2006;15(7):1257–1261. doi: 10.1158/1055-9965.EPI-06-0041. [DOI] [PubMed] [Google Scholar]
  47. Watson M, Saraiya M, Ahmed F, Cardinez CJ, Reichman ME, Weir HK, et al. Using population-based cancer registry data to assess the burden of human papillomavirus-associated cancers in the United States: Overview of methods. Cancer. 2008;113( S10):2841–2854. doi: 10.1002/cncr.23758. [DOI] [PubMed] [Google Scholar]
  48. Zawistowski CA, Frader JE. Ethical problems in pediatric critical care: Consent. Critical Care Medicine. 2003;31(S5):S407–S410. doi: 10.1097/01.CCM.0000065274.46402.DB. [DOI] [PubMed] [Google Scholar]
  49. Zimmerman DE, Akerelrea CA, Buller DB, Hau B, LeBlanc M. Integrating usability testing into the development of a 5 a day nutrition website for at-risk populations in the American Southwest. Journal of Health Psychology. 2003;8:119–134. doi: 10.1177/1359105303008001448. [DOI] [PubMed] [Google Scholar]

RESOURCES