Development of an Internet Intervention to Address Behaviors Associated with Skin Cancer Risk among Young Adults

Abstract

Purpose

Skin cancer is the most common cancer in the US, and its incidence is increasing. The major risk factor for skin cancer is exposure to ultraviolet radiation (UV). Young adults tend to expose themselves to large amounts of UV and engage in minimal skin protection, which increases their skin cancer risk. Interventions are needed to address risk behaviors among young adults that may lead to skin cancer. The nternet offers a cost-effective way to widely disseminate efficacious interventions. The current paper describes the development of an online skin cancer risk reduction intervention (UV4.me) for young adults.

Procedures

The iterative development process for UV4.me followed best-practice guidelines and included the following activities: individual interviews, focus groups, content development by the expert team, acceptability testing, cognitive interviewing for questionnaires, quality control testing, usability testing, and a pilot randomized controlled trial. Participant acceptability and usability feedback was assessed.

Principal Results

The development process produced an evidence-informed intervention that is individually-tailored, interactive, and multimedia in nature based on the Integrative Model of Behavior Prediction, a model for internet interventions, and other best-practice recommendations, expert input, as well as user acceptability and usability feedback gathered before, during, and after development.

Major Conclusions

Development of an acceptable intervention intended to have a significant public health impact requires a relatively large investment in time, money, expertise, and ongoing user input. Lessons learned and recommendations are discussed. The comprehensive process used may help prepare others interested in creating similar behavioral health interventions.

1. Introduction

Skin cancer is the most common cancer, with nearly five million cases treated annually in the US; its incidence has been increasing in recent years (Donaldson & Coldiron, 2011; Gordon, 2013; Nikolaou & Stratigos, 2014; Tuong, Cheng, & Armstrong, 2012; USDHHS, 2014). Most skin cancers are non-melanomas, which have been estimated to be increasing by 2.6% in incidence per year (1999–2006) (Donaldson & Coldiron, 2011), and melanomas have been increasing at a rate of 1.5–6% per year (2002–2011) (USDHHS, 2014). Known risk factors for skin cancer include personal or family history of skin cancer, fair skin, and ultraviolet radiation (UV) from the sun and/or indoor tanning (Goldberg et al., 2007; Lazovich et al., 2010; Markovic et al., 2007; Nikolaou & Stratigos, 2014; Psaty, Scope, Halpern, & Marghoob, 2010; Qureshi, Zhang, & Han, 2011; Siskind, Aitken, Green, & Martin, 2002; Vishvakarman & Wong, 2003). It is common for young adults (e.g., aged 18–25 years) to expose themselves to large amounts of UV without proper skin protection (e.g., wearing adequate sunscreen) (Buller et al., 2011; Coups, Manne, & Heckman, 2008; Heckman, Coups, & Manne, 2008; Stanton, Janda, Baade, & Anderson, 2004). For example, US adolescents have had the lowest skin protection rates of all age groups (Stanton et al., 2004), with only 39% applying sunscreen when going outdoors in the summer (Cokkinides et al., 2006). US adolescents also engage in increased exposure to natural and artificial UV as they move into adulthood (MacNeal & Dinulos, 2007). Invasive skin cancer is the second most diagnosed cancer among young adults (Bleyer & Barr, 2009). For these reasons, it is important to have interventions that are effective in addressing skin cancer risk behaviors among young adults. Indeed, the Surgeon General has recently published a call to action to prevent skin cancer (USDHHS, 2014).

Numerous skin cancer prevention interventions have been developed. Systematic reviews and meta-analyses have found some of these interventions to be efficacious for specific populations (Horsham, Sendall, et al., 2014; Lin, Eder, & Weinmann, 2011; Rodrigues, Sniehotta, & Araujo-Soares, 2013; Saraiya, Glanz, et al., 2004; Williams, Grogan, et al., 2013). However, many of these interventions have two key limitations. First, many have emphasized only education and awareness, which tend to result in limited changes in actual behavior, particularly changes that are sustained over time (Aarestrup et al., 2014; Hart & Demarco, 2008; Horsham et al., 2014; Keeney et al., 2009; Roberts & Black, 2009). However, one type of intervention that has shown promise in modifying behaviors associated with skin cancer development is those that focus on the negative effects on appearance of UV exposure and lack of protection (Williams, Grogan, Clark-Carter, & Buckley, 2013). For example, a series of studies by Mahler and colleagues found that taking and showing young adults photos of their existing facial UV damage resulted in decreased UV exposure and increased skin protection (Gibbons, Gerrard, Lane, Mahler, & Kulik, 2005; Mahler, Kulik, Gerrard, & Gibbons, 2013; Mahler et al., 2005). Another limitation of many skin cancer prevention interventions is that they are often delivered in-person, which can be difficult to scale up for wider dissemination. Internet interventions, alternatively, can be cost-effective and disseminated widely. Since 97% of US young adults aged 18–29 years use the internet (Pew_Research_Internet_Project, 2014), internet-based interventions may have considerable potential to appeal to young adults and provide a greater likelihood of being utilized than other types of interventions. Although a couple of studies have reported on the development and/or testing of online skin cancer prevention interventions (Bowen et al., 2012; Kimlin & Parisi, 2001), these have focused on children or melanoma patients and their family members. To our knowledge, this is the first paper describing the development or formative assessment of an online intervention to address risk behaviors among at-risk young adults that may lead to skin cancer.

Internet interventions more extensively informed by a theoretical framework tend to be more efficacious in modifying health behaviors than those that use theory less extensively (Webb, Joseph, Yardley, & Michie, 2010). The theoretical framework for the current intervention was adapted from the Integrative Model of Behavior Prediction (IM) (Fishbein, Hennessy, Yzer, & Douglas, 2003). The IM includes background variables such as demographics; cognitive variables such as beliefs, attitudes, norms, and self-efficacy; intentions; and behavior. We also emphasized appearance concerns, which is a major factor associated with tanning behavior among young adults (Cafri et al., 2006; Danoff-Burg & Mosher, 2006; Hillhouse, Turrisi, Stapleton, & Robinson, 2008). The intervention is targeted to young adults, individually-tailored based on IM variables, and interactive utilizing multiple media formats, characteristics that have been shown to be components of effective health behavior interventions (Bewick et al., 2008; Bock, Graham, Whiteley, & Stoddard, 2008; Davies, Spence, Vandelanotte, Caperchione, & Mummery, 2012; Griffiths, Lindenmeyer, Powell, Lowe, & Thorogood, 2006; Lustria et al., 2013; van den Berg, Schoones, & Vliet Vlieland, 2007).

The purpose of the current paper is to describe the systematic and comprehensive process of development of UV4.me, an online intervention to address risk behaviors that may lead to skin cancer among young adults. This description provides 1) an example of the application of best-practice guidelines, 2) our acceptability and usability findings, as well as 3) more specific lessons learned for others contemplating or preparing for internet health behavior intervention development.

The development of the web intervention and preparation for a randomized controlled trial (RCT) involved a comprehensive process including several mixed-method steps incorporating input from stakeholders (i.e., multi-disciplinary experts and the target population) over a three-year period. The process was guided by intervention development and assessment guidelines for behavioral therapy, web-based interventions, and health communications programs including health literacy best-practices (Barak, Klein, & Proudfoot, 2009; Danaher & Seeley, 2009; Fleisher et al., 2014; National_Cancer_Institute, 2002; Ritterband & Tate, 2009; Ritterband, Thorndike, Cox, Kovatchev, & Gonder-Frederick, 2009). In preparation for future efficacy and effectiveness trials, the phases of intervention development and assessment included intervention planning, content and website development, initial assessment and revision, and pilot testing. Prior web health intervention development projects have used various aspects or combinations of these development steps (Berry et al., 2010; Fergus et al., 2014; Michie et al., 2012; Moore et al., 2013; Pachankis, Lelutiu-Weinberger, Golub, & Parsons, 2013; Riiser, Londal, Ommundsen, Sundar, & Helseth, 2013; Villegas et al., 2014; Wolpin et al., 2014), but few have reported on all of them.

2. Methods and Results

The current project was developed by a multidisciplinary team, which included individuals with expertise in skin cancer prevention, young adults, internet interventions, psychology, qualitative and quantitative research methods, health literacy/communication, computer programming, psychometrics, and instructional design. The four developmental phases of this mixed-methods project are illustrated in Table 1. The methods and results from each step are described below. This project was approved and monitored by a cancer center Institutional Review Board, and informed consent was obtained from research participants.

Table 1.

The Developmental Phases of the UV4.me Project

Phase	Activities	n	Dates
1. Intervention Planning	Individual Interviews	25	Nov and Dec 2009
	Obtaining grant funding	NA	Feb 2010 – Mar 2011
	Focus Groups	14	July – Dec 2011
2. Content and website development	Synthesis of Material	NA	Jan–July 2012
	Avatar Development	NA	Jan–Oct 2013
	Health Literacy Evaluation	NA	April 2012 and April 2013
3. Preliminary assessment and revision	Cognitive Interviewing	20	Aug – Dec 2012
	Acceptability Testing	26	Aug – Nov 2012 and July – Sept 2013
	Usability Testing	12	Dec 2013 – Mar 2014
	Quality Control Testing	NA	Oct 2013 – Mar 2014
4. Pilot Testing	Pilot Testing	53	Mar and April 2014

2.1. Phase 1. Intervention Planning

Individual interviews

We recruited a convenience sample of 25 young adults participating in an in-person skin cancer prevention randomized controlled intervention trial from a university campus in Philadelphia (average age = 23 years [range from 20 to 25 years], 64% white, 72% female). The purpose of these individual interviews was to have participants rate proposed topics and web features on a 5-point Likert-type scale (See Table 2) and offer suggestions and feedback prior to online development and to support an application for funding. Participants attended an in-person session in which cognitive interviews were conducted. Participants were asked to “think aloud” (Willis, 2005) as they responded to the draft online baseline survey and reviewed a tailored skin cancer prevention pamphlet from a prior successful intervention (S. Manne et al., 2010). The tailored color pamphlet focused on constructs previously found to predict cancer prevention behaviors such as perceived risk, salience and coherence (the perception that a behavior is consistent with beliefs about how to protect health), benefits and barriers, social influence (e.g., doctor’s recommendations), self-efficacy, and intentions. Additionally, images under consideration for inclusion in the intervention that focused on appearance concerns (e.g., UV damage and age-progression photos) were presented. Participants were asked a series of questions about what they liked, did not like, and what suggestions they had about the survey, pamphlet, and study, for example, about games to incorporate into the website, potential incentives, recruitment, and strategies to minimize attrition. Responses were summarized by the first author. Feedback at this stage was positive (See Table 2). This feedback helped inform decisions about recruitment/enrollment, surveys, web intervention topics and features, and incentives. For example, we decided to include an avatar activity, and a decision was made against the use of Twitter or text-messaging due to their low ratings by participants. Although we were interested in using Facebook, we decided it would not be feasible for the current project given time and budget constraints but would be considered for future projects.

Table 2.

Phase 1: Intervention Planning - Ratings from Individual Interviews

	Mean Ratings (n = 25)
Topics	(1=dislike, 5=like)
Photo-aging information	4.88
Symptoms of skin cancer	4.84
Sunburns, Skin self-exams	4.80
How UV damages skin	4.76
Web Features
Images of twins, one aged	5.00
Personalized feedback	4.96
Incentives ($20 per hour)	4.94
UV photos	4.92
Email recruitment	4.90
Images of skin damage	4.88
Draft survey	4.40
Twitter	2.68
Text messages	2.42

Focus Groups

Once project funding was secured, a convenience sample of young adults recruited from the sponsoring institution and the local community participated in two sets of focus groups of 4–8 people each, to assist in shaping the content of the web program. As material differed across the groups, some individuals participated more than once. The first set of focus groups (n = 4) met twice and reviewed potential images and provided suggestions for general web program topics and specific activities that were identified during the individual interviews. Preliminary content was drafted and some images chosen in preparation for the second set of focus groups, which were held with groups of 6–8 young adults during four meetings. This set of focus groups reviewed other images and provided more specific suggestions about how to target and tailor material for young adults. The groups reviewed initial mock-ups of the intervention introduction, and a number of key areas of focus of the intervention, including why people tan, why people should not tan, indoor tanning, skin damage, skin cancer, and decreasing tanning. The groups also provided feedback and suggestions for the theme of the website and how to engage young adults with a skin cancer prevention website, in response to a series of interview questions. Feedback and suggestions about the interactive avatar activity that would demonstrate the effects of UV exposure and protection on an animated character were also gathered (e.g., what should it look like, should it be individually tailored, how should it work, how should it be integrated into the website). Qualitative feedback was grouped into themes by a research staff member and summarized for the research team. The focus group findings helped us select and refine the initial theme, topics, content, images, and activities to be used in the web program as well as plans for how to make the web program engaging to young adults. The findings also helped to identify areas of content that participants might have difficulty understanding (e.g., complicated figures).

2.2. Phase 2. Content and Website Development

Synthesis of Material into Website Content

Based on a synthesis of the individual interviews, focus groups, our expertise, and the literature (Barak et al., 2009; Ritterband & Tate, 2009; Ritterband et al., 2009), the multi-disciplinary team collaborated to create the design, modules, and other activities to be included in the web program (See Table 3). The intervention was intended to be interactive, tailored, utilize multiple media formats, and maximize participant engagement while minimizing burden. Twelve modules were created, each with content related to a specific topic (e.g., sunscreen, indoor tanning, skin cancer; See Figure 1) determined to be important in terms of risk or protective behaviors and their correlates. In addition, several other more general website sections (e.g., avatar, MyStuff – a printable summary of tailored goals and recommendations) were developed. Each module was expected to take about 10 minutes to review, and it was assumed that many participants would not view all the modules, and in fact, may only use the website once, so an attempt was made to have each module stand alone and be as focused as possible on encouraging behavior change.

Table 3.

Phase 2: Outline of Content and Features of the Web Intervention

Web Modules	Web Topics	Sample Interactive and Tailored Features	Sample IM Constructs Addressed
Introduction	Purpose, features How to use the website Tailoring questions		NA
Exposure
Why do people tan?	Change in tanning trends over time What do you know about tanning? Who tans in your circle?	Personalized responses (e.g., feedback to responses regarding “what do you know about tanning” quiz items)	Norms Knowledge
Indoor tanning	The truth about indoor tanning Risky behaviors	Personalized responses (e.g., feedback to responses regarding indoor tanning true/false quiz items)	Knowledge Behaviors
To tan or not to tan	Why people tan Reasons for tanning Tanorexia	Personalized responses (e.g., feedback regarding user’s own personal reasons for tanning) Tanorexia quiz	Self-efficacy Intentions Behaviors
Effects
UV and health	UV and its effects on the skin (e.g., vitamin D, acne) and immunity Sunburns Skin type and sunburn risk Skin damage	Skin type and sunburn risk UV damage photo of similar individual (See Figure 2)	UV related Beliefs
UV and looks	What is UV? Celebrities and tanning Photo-aging Feeling good about yourself without tanning	Age progression photos of similar individual (See Figure 3)	Attitudes toward appearance
Skin damage	Skin cancer risk factors Melanoma rates around the world New cases of skin cancer Impact of global warming on UV levels	Personal risk calculator Personalized feedback on melanoma quiz	Risk perception Knowledge
Skin cancer	Skin cancer types/prevalence Videos about skin cancer Skin cancer risk factors Treatment Skin cancer and darker skin	Personal risk calculator	Risk perception
Protection
Sunscreen	Sun Protection Factor (SPF) How to use sunscreen Reasons for not using sunscreen	Sunscreen SPF and reapplication calculator Personalized responses (e.g., feedback regarding which sunscreen is best for user’s skin)	Self-efficacy Norms Intentions Behaviors
Clothes	Protective clothes Reasons for not wearing protective clothes	Personalized responses (e.g., feedback regarding user’s habit of wearing protective clothes)	Self-efficacy Intentions Behaviors
Sunless tanning	Reasons people sunless tan Have you tried sunless tanners before? Better sunless tanning	Personalized feedback and responses to sunless tanning quiz and reasons for not using sunless tanner	-Self-efficacy Norms Intentions Knowledge Behaviors
Shade	Staying in the shade Reasons for not being in the shade	Personalized responses (e.g., feedback regarding user’s personal reasons for avoiding shade)	Self-efficacy Intentions Behaviors
Skin exams	Checking for skin cancer ABC’s of skin cancer Reasons for not doing skin exams	Personalized responses (e.g., feedback regarding user’s personal reasons for not doing skin exams)	Self-efficacy Intentions Behaviors

Figure 1.

UV4.me Home Page

Tailoring algorithms were created to direct participants to focus on certain modules first based on their responses to a few initial questions (e.g., the indoor tanning module was recommended if participants said they tanned indoors). Throughout the web program, participants were asked questions and were provided with tailored feedback (e.g., “Do you know people who tan? If so, how likely are they to affect your choice to tan or not?”). On some pages, additional information, such as definitions, was available by clicking on or hovering over an icon, image, or text. Once the content was finalized, many images were purchased from online stock photo websites, and online searches were conducted to identify some of the included videos (n=7) and comics (n=7) related to the topic areas (permission was obtained to use identified videos and comics). All video authors granted permission for use for free, and all comic authors charged a fee ranging from $25 to $45. Several photos were also taken in-house of female and male young adults with varied skin types ranging from fair to dark, and then two software programs were used to create a library of photos simulating current UV damage (Mirror - www.canfieldsci.com) and UV damage with age progression (https://www.aprilage.com/ageme) for tailoring purposes (Figures 2 and 3). In addition, the research team created a series of brief videos of a young physician who had tanned indoors and survived melanoma.

Figure 2.

Tailored Photos Showing Simulated Current UV Damage

Figure 3.

Tailored Age Profession Photos Showing Simulated UV Damage

A number of interactive elements were created to increase engagement in the web program. For example, at the end of each module was a goal-setting section in which participants could choose to set a pre-specified goal for the next two weeks or not (e.g., “For the next two weeks, I will not use a tanning bed, booth, or sunlamp.”). Another section of the website contained printable versions of all of the participant’s goals from each module and links to other online skin cancer prevention resources (called “MyStuff”). The web program also included a personalized avatar (See below). An instructional designer reviewed the material and activities and helped redesign some activities to make them more user-friendly, interactive, and to increase the potential for understanding and retention (Hilgart, Ritterband, Thorndike, & Kinzie, 2012).

Avatar Development

We worked with an existing web-based avatar system (doppelme.com) and incorporated it into the web program homepage (See Figure 1). A number of relevant animated items were added to allow for further personalization of the avatar by the target population and for the topic of focus (e.g., sunscreen with different SPFs, hats with different types of brims, different types of sunglasses). Based on responses to a few questions in the introductory section of the web program, a participant’s avatar was tailored to his/her gender and phenotype (e.g., skin type, eye color, hair color). When participants completed a module, additional items were made available for use with their avatar (e.g., sunscreen, hat, tanning bed). Depending on whether participants chose to use protective or non-protective items on their avatar, the avatar would become red or not, and participants would see a tailored message (“Ouch, you have a sunburn! …”).

Participants from the acceptability, usability, and pilot testing provided feedback on the avatar (Table 4). Users spent an average of 1.5 to 3 minutes exploring the avatar. Feedback from acceptability and usability testing showed that users enjoyed the avatar customization activity and expressed the desire to complete more modules in order to unlock additional features. Changes made as a result of feedback included the user’s eye, hair, and skin color being provided as the default upon beginning with the avatar and clarifying some of the associated text. Software “bugs” were identified during acceptability and usability testing, such as the avatar failing to “burn” at the appropriate times, and were resolved prior to pilot testing.

Table 4.

Phase 2: Avatar Feedback

TYPE OF FEEDBACK	SAMPLE QUOTE(S)	N
Positive	Like the number of options it provides to customize/personalize the avatar	17
	I liked the fact that some items are locked. This will encourage people to go through the website more to unlock them.	7
	It seems fun	6
	(Miscellaneous) Good. Nice. Liked it. Good visual representation. Made sense. Laid out well.	6
	People my age may like it	3
	The avatar draws the attention first	3
	Easy	3
	The avatar was one of the things I liked best	3
	(What would you do if you saw your avatar start to burn?) I don’t know. I guess it’s kind of a game. I would put on a hat, sunscreen, maybe move it to a shadier place.	3
	Interactive. I’m interested in seeing how to interact with it. I want to make him look nice.	3
	Overall - exciting	2
	Interesting twist to care for something	2
	Cute	2
	I like the explanation on what needs to change with the avatar so it does not get burnt	1
	Total Positive	61
Negative	Not sure why certain things are locked. Not sure how to use unlocked items.	11
	Younger people will be more motivated to unlock things for their avatar, not people of my age	5
	I feel as though the avatar is a little pointless	5
	The avatar was least helpful at understanding skin cancer risks and prevention	4
	Wouldn’t take the time to use it	4
	(Miscellaneous - Concept) Initially was not sure what the avatar was about. There should be more to do with the avatar. The avatar is just something extra to do on the website. What is the picture doing here? (without a guide)	4
	Avatar was one of the things I liked the least	3
	(Participant was unable to complete the task due to technical difficulties. E.g., Tried to save changes and was logged off.)	3
	(What would you do if you saw your avatar start to burn? Was confused by the meaning of the question.) Would try to figure out why the avatar was burning. Perhaps the avatar burned based on my responses to questions.	3
	Why doesn’t it have hair?	3
	(Miscellaneous - Appearance) The font explaining the avatar doesn’t stand out. Add the word ‘Home’ with the homepage icon at the top of the screen. Cannot change colors for sunglasses and hats.	3
	Is there any information, or is it just the avatar?	2
	Maybe a moving (animated/talking) avatar would be more interesting	2
	I am not into internet personification	2
	Arrows on either side of the items (when there are no more options on the next page for that particular category) do not make sense; remove them.	2
	It’s basic with some customization; not many options for skin tone. Participant suggested adding more options, not just summer clothes.	2
	There is no sunscreen option to put on the avatar. (Participant didn’t notice the sunscreen option.)	1
	Total Negative	59
Neutral
	It reminds me of a Facebook app in which you have to create an avatar	2
	Can you change the skin color?	2
	(Noticed certain options were locked.) I guess you would have to complete modules to unlock options.	2
	Have avatar available first, then the modules	1
	I want the avatar to look like me	1
	(Participant never used arrows to view more options for the avatar)	1
	Total Neutral	9

Health Literacy Evaluation

To ensure that the content of the website was understandable to a broad audience, health literacy experts (LF and SR) systematically evaluated all website text using software and their health literacy expertise. The health literacy experts were the director of the office of health communications and health disparities (LF) and the director of the resource and education center (SR) at a comprehensive cancer center. A software program, Health Literacy Advisor (Health Literacy Innovations, LLC), was used to evaluate and identify complex terms, complex health terms, polysyllabic words (i.e., words with more than three syllables), and long sentences (i.e., sentences with 15 or more words). A readability score was also calculated by the program using the Precise SMOG (“Simple Measure of Gobbledygook”) Index formula. This formula assesses and provides a reading grade level, which research indicates is more reliable than other measures for health-related materials (Wang, Miller, Schmitt, & Wen, 2013). The findings from the health literacy software were evaluated by the health literacy expert and the research team, and the text was simplified to improve readability and comprehension. One of the health literacy experts then reviewed the revised text and suggested any final edits. Other components of the website were also evaluated that are known to contribute to overall readability, such as the way content and images are designed, displayed, and organized, as well as ease of navigation (e.g., font color, size, and style). Suggestions were provided based on health literacy best practice for creating easy-to-use websites (Redish, 2007; U.S. Department of Health and Human Services, 2010).

2.3. Phase 3. Preliminary Assessment and Revision

Cognitive Interviewing

A smaller team of content, questionnaire, and health literacy experts (SR and a staff member who had been trained by her) developed the initial questionnaire using existing items/scales from the literature and created some new ones. The “assessment team” revised the initial questionnaire in order to make it more simple, readable, personal, consistent, and less ambiguous. We then pre-tested the questionnaire using cognitive interviewing (Willis, 2005). A convenience sample of 20 young adults recruited at the sponsoring institution and from the local community completed the questionnaire and provided immediate oral feedback in-person. See Table 5 for demographic characteristics. Cognitive interviewing was conducted in-person in two ways. Half of the participants (n = 10) responded to all the items and provided general feedback afterwards. The other half of participants (n = 10) were asked to use “think aloud” procedures as they read and responded to a subsample of items that were developed or revised significantly by the team and then responded to specific probes about these items. Based on the cognitive interviewing, items were removed, moved, or added, and again made simpler, more specific, personal, and consistent, as well as less ambiguous. Additional detail about the methods and results of the cognitive interviewing including the final measures will be provided in a separate paper.

Table 5.

Phase 3: Demographics of User Testing (Cognitive Interviewing, Acceptability and Usability Testing) (N = 47)^a

Variable
	[M(SD)]
Age in years	22.3 (2.3)
	n (%)
Female Sex	34 (72.3)
Race
White	44 (93.6)
African-American or Black	2 (4.3)
Asian American	1 (2.1)
Hispanic	1 (2.1)
Skin color
Very fair	7 (14.9)
Fair	28 (59.6)
Olive	12 (25.5)

^aSome users who completed cognitive interviewing (n = 20) also completed acceptability (n = 26) or usability testing (n = 12)

Acceptability Testing

Acceptability testing was modeled after that used by the US National Cancer Institute for an internet-based program for breast cancer patients (Atkinson et al., 2007). Acceptability testing was conducted to assess and increase the suitability of content for the intended audience (i.e., young adults at moderate to high risk of developing skin cancer) including attractiveness, comprehension, appropriateness, and persuasion. One research staff member with experience in qualitative data collection administered a structured interview, while a second staff member with similar experience video recorded the in-person interview and took notes. Two separate rounds of acceptability testing of the initial web program were conducted with a convenience sample of young adults recruited at the sponsoring institution (but unknown to research staff) and from the local community. The first round (n = 15) was conducted after all the basic content and format was in place, and the second round (n = 11) was conducted after the interactive activities (e.g., avatar, damage and aging photos, goal-setting) were in place. The second round was divided into participants who were asked to focus on specific interactive activities (n = 6) and those (n = 5) who were asked to explore the website freely. The latter individuals viewed an average of four modules and spent 35 minutes exploring the website. Participants in each round provided both positive and negative feedback about the overall web program and about specific content and activities. Participants rated items created by the team regarding likeability, usefulness, and personal applicability on a scale from 0–10, with 10 being the best (See Table 6). In both rounds, some participants thought that the program was geared toward adolescent, female, fair-skinned tanners. They also provided additional feedback about the homepage, information, format, icons, length, images, videos, quizzes, games, goal-setting, the avatar, modules, and functionality. In the second round, some participants provided additional feedback on the enrollment process, estimated how long they might spend on the website, and the content in MyStuff.

Table 6.

Phase 3 - Acceptability Ratings

	Mean (SD) for Round 1 (n = 15) (0 = worst, 10 = best)	Mean (SD) for Round 2 (n = 11) (0 = worst, 10 = best)
Likability	7.5 (1.0)	7.7 (1.8)
Usefulness	8.5 (1.3)	8.7 (1.5)
Personal applicability	8.2 (1.6)	7.8 (2.3)
Web Features
Features liked best	Videos; age progression photos; interactive sections; informational pop-up boxes; educational information - especially that associated with skin exams	Educational information; interactive sections; tanning trends timeline; videos from physician with personal skin cancer history; SPF calculator; age progression photos
Features liked least	Format (e.g., appearance, large blocks of text)	Tutorial; avatar; length of some videos

Usability Testing (n = 12)

In usability testing, users are asked to perform typical tasks with a product, or simply explore it freely, while their behaviors are observed and recorded to identify design flaws that cause user errors or difficulties (Bastien, 2009). For testing of the UV4.me intervention, one staff member administered an in-person structured interview (USDOHHS, usability.gov), while a second staff member video recorded the interview and took notes. Two separate rounds of usability testing were conducted with a convenience sample of young adults recruited at the sponsoring institution and from the local community. In one round of testing (n = 6), we assigned users to complete certain tasks (e.g., use the avatar, view a video, use the goal-setting activity) and monitored how easily they were able to complete them and what questions and problems they had. Users spent an average of 2–3.5 minutes on each task. Usability was also assessed with the System Usability Scale (Brooke, 1986), a ten-item robust measure designed to assess perceived effectiveness as well as efficiency of and satisfaction with a system. The website, as tested in the first round of usability testing, received a score of 86.7, with a standard deviation of 13.5. A score above 68 indicates above-average usability (Brooke, 1986). Users estimated that, if using the system on their own (at home and without being monitored), they would spend an average of 40 minutes using the website, with a range of 15 minutes to 2 hours. When asked how much they liked the website on a scale of 1 to 10, the website was rated with an average score of 7.5 (SD = 2.4).

In the second round of usability testing (n = 6), users were asked to explore the site from start to finish (i.e., from completing the screener to completing the final follow-up assessment). The purpose of this testing was to ensure that the programmed automation (e.g., email notifications, tailored content) was working properly. Though the purpose of this testing was not to review specific content and activities within the modules, users were offered the opportunity to explore this content if interested and as time permitted.

Quality Control Testing

Throughout the development process and particularly at the end prior to the pilot trial, the project manager and programmers at BeHealth Solutions, Inc. (the company that programmed the website, assessments, and data management system) conducted quality control testing in order to ensure that all intervention and data collection and management features were functioning properly. This consisted of testing the auto-enrollment process (eligibility, online consent, randomization), assessments, intervention including tailoring and the avatar, email reminders, data capture, and data reports. Testing was conducted by reviewing the content of the entire site as well as creating a number of mock subject profiles and taking them through a variety of scenarios to see if they worked properly (e.g., completing the screening form as an ineligible subject to see if he/she was deemed ineligible and received appropriate notifications).

2.4 Phase 4. Pilot testing

After completion of usability testing, a pilot trial was initiated using all of the planned procedures to determine whether the automated enrollment, website, and data management programs were working properly, finalize our data management plans, and establish recruitment and eligibility rates for the main RCT (Lackey & Wingate, 1986), which tested the effects of the interventions on self-reported UV exposure and protection behaviors at 3-weeks and 12-weeks after baseline. Participants were recruited from an existing internet research panel by Survey Sampling International (SSI). SSI panelists were exposed to brief web banner ads about the study from which they could click to link to UV4.me. Eligibility criteria included being 18–25 years old, at moderate to high risk of developing skin cancer based on the Brief Risk Assessment Tool (Glanz et al., 2003), and without a personal history of skin cancer. Using the automated enrollment procedures, 222 people were screened, 96 were eligible, 53 then provided informed consent and submitted completed baseline surveys that had been cognitively tested, and 46 submitted follow-up surveys after three weeks.

Participants who completed the baseline survey were randomized to the experimental UV4.me intervention program, the existing Skin Cancer Foundation website, and to an assessment-only condition. Out of the subjects who were randomized to UV4.me and completed the follow-up survey (n = 15), thirteen (93%) accessed the intervention modules. We estimated, based on log-in and module completion times, that subjects spent on average about 90 minutes using the website (standard deviation of 105 minutes, with a range of 0 to 344 minutes) and logged into the website about eight separate times (standard deviation of 9.6 times, with a range of 0 to 35 times). Among those who accessed the modules, perceptions were positive overall, with ratings ranging from 4.4 (on a scale of 1 to 5) to 4.7 (SD = 0.5–0.7). Specifically, participants reported satisfaction with the website [M(SD) = 4.5 (0.5]. The highest rated item was that the website was useful [M(SD) = 4.7 (0.5)], followed by easy [M(SD) = 4.6 (0.5)], convenient [M(SD) = 4.5 (0.7)],and helpful [M(SD) = 4.4 (0.7)]. These participants reported that the website increased their knowledge about skin protection [M(SD) = 4.5 (0.7)], their confidence to protect their skin [M(SD) = 4.5 (0.7)], and their knowledge about skin cancer [M(SD) = 4.4 (0.8)]. Participants also reported that they would be likely to recommend the website to others [M(SD) = 4.5 (0.7)].

Though no problems were reported with the website itself, about half of the subjects who were provided with access to the website reported that not having time to visit the site was at least “a little problem,” with two subjects indicating that this was “a big problem.” Open-ended feedback indicated that the information presented in the modules was most helpful, as well as the short length and interactivity of the modules, and that a few subjects found the avatar to be the least helpful part of the website and/or did not understand its purpose. One user noted that it would be helpful for the website to be available in other languages, such as Spanish. As a result of the pilot, we refined the study screener and assessment items, clarified instructions on some of the introductory pages, and updated some of our data management procedures.

3. Discussion

This manuscript documented the process used for development and preliminary assessment of a tailored internet intervention to address skin cancer risk behaviors among young adults. The process for the current project involved four phases: intervention planning, content and website development, initial assessment and revision, and pilot testing. To summarize, the overall lessons and recommendations are as follows: 1) gather input from multi-disciplinary experts and the intended population during the entire development process; 2) use mixed methods to provide the most complete and useful information; 3) include the planned population and procedures in formative and pilot testing; 4) plan and manage the time, money, and expertise needed for the project carefully prior to and throughout the project; and finally, 5) be willing and able to revise plans, procedures, the website, and potentially the budget and timeline as the project progresses. The challenges of conducting this type of work are, in part, the time, money, and expertise required to conduct such a project. A formal cost analysis was not conducted, but such analysis could be useful. However, it is important to note that the costs of initially developing an intervention would be considered “sunk” costs, and what may be more pertinent to consider for the future are the costs of eventual dissemination and sustainability, which we have not yet assessed. Additionally, it can be difficult to obtain feedback and responses from participants that apply to the full-fledged intervention using prototype materials; however, developing the full-fledged intervention prior to obtaining some feedback would most certainly require later modification to meet user needs. More specific practical lessons from each of the phases of the process that we hope will be useful to other internet intervention developers and researchers are discussed below.

3.1. Intervention planning

Individual Interviews

For efficiency’s sake and in preparation for future funding applications, it can be helpful to “piggy-back” qualitative interviews onto an ongoing trial. A structured interview incorporating both qualitative and quantitative feedback allows one to gather both general and specific data both systematically and efficiently. We recommend including some sample materials for participants to react to while also being open to creation of new activities.

Focus Groups

Young adults expressed interested in providing feedback and serving as research participants, but they are somewhat transient across time due to changes in school, work, and living situation. Thus, flexibility when contacting them and scheduling group meetings is required. Other methods such as online focus groups could be considered.

3.2. Content and website development

This was the most labor-intensive aspect of the development process. Our goal was to make the material and activities as engaging and appealing as possible to our target population. Translating conceptual ideas for interactive activities and processes into feasibly programmable ones was challenging at times. However, during this time, we learned to prioritize what material, activities, and processes would be included in the program. Content was developed and edited extensively and repeatedly over time. The development of a specific timeline, prioritization of content and activities, and an open and ongoing discussion between content, intervention, instructional design, health literacy, and programming experts can facilitate efficient use of time and avoid wasted or redundant effort. The timeline should include simultaneous (rather than sequential) work tasks and be somewhat flexible with extra time built in if possible in order to utilize time efficiently and accommodate unexpected setbacks and technical difficulties.

Avatar Development

Although they liked the avatar and said they were motivated to access more items for use with their avatar, users tended to like other website features better and used the avatar for only a few minutes during acceptability and usability testing. Despite our efforts, some users continued to say that they perceived the avatar activity and appearance to be appropriate for adolescents rather than young adults and that they didn’t see the point or wouldn’t use it themselves but that others might. Unfortunately, we did not have the funds to make additional, more extensive changes to the avatar. Our experience with the avatar might be an example of the incongruence between what people think or say they want, like, will use, and will actually use and benefit from, that sometimes occurs when developing and assessing behavioral health interventions. Thus, pre-testing important intervention components is critical to the ultimate success of intervention efforts.

Health Literacy Evaluation

It is well-known that scientists do not tend to write at the level of the average citizen (Eichler, Wieser, & Brugger, 2009). Recommended best-practice strategies to create understandable reading materials, interventions, and web interventions exist (Redish, 2007; U.S. Department of Health and Human Services, 2010). These include simplifying words, phrases, and sentences and attending to the way content and images are designed, displayed, and organized, as well as the ease of navigation. Our initial website content was revised so that it would be more understandable to a broad audience of young adults.

3.3. Preliminary assessment and revision

Cognitive Interviewing

Unless measures are developed with the assistance of assessment experts and cognitively tested with the target audience, they may not be reliable and valid. Cognitive testing improved the format and instructions for scales and made items simpler, more personal and consistent, as well as less ambiguous.

Acceptability Testing

Acceptability testing assisted us in making the web program content, format, media, activities, and functionality more appropriate and acceptable to the target population. Participants provided feedback about who they thought the website was designed for so that we could revise the site to make it more applicable to our target population. This helped us identify which sections and videos were too long and complex and which did not provide suitable information. It also became clear that the young adults did not want to engage in a somewhat lengthy tutorial before using the website and that some aspects of the navigation were confusing despite our instructions. Thus, we attempted to make navigation as similar to other commonly-used websites and as intuitive as possible.

Usability Testing

Usability testing allowed us to identify and correct specific problems, such as email notifications not firing properly. Overall, usability testing revealed bugs with the website and misunderstandings on the part of the users, such as with the avatar and with the goal-setting process. This prompted us to correct these issues, clarify instructions, and attempt to make activities more intuitive. Some “usability” testing can be conducted by staff themselves, but the use of potential participants provides invaluable information for future intervention programs and research trials.

Quality Control Testing

Although necessary, this type of testing has rarely been described in the literature, perhaps because it is often conducted by programmers rather than investigators or interventionists. However, it is useful for as many members of the team and “pilot” participants as possible to participate in this type of testing both by reviewing content and programming if possible and creating mock subjects to test various participation scenarios. This testing identified several technical glitches that were able to be remedied by the programmers prior to actual participant enrollment.

3.4. Pilot testing

Conducting a pilot test helped us accomplish the following: identify additional problems in a small sample before full release of the intervention trial in a larger RCT; develop, refine, and practice our data management procedures; assess eligibility, enrollment, and follow-up rates in order to finalize the larger RCT recruitment plan and sample size. The pilot was helpful in facilitating the activities of both study participants and research staff for the main RCT. Pilot trials should be conducted as similarly as possible to the main trial in order to be best prepared for the larger RCT. Otherwise, the main trial is likely to identify additional problems in a larger sample that could have been avoided or minimized.

The strengths of this project are the systematic development of an internet intervention using multiple methods and input from both experts and young adults. Though these efforts were extensive, we believe that they are necessary to develop an efficacious web intervention that an intended population will actually use. However, other projects may be able to include fewer rounds of assessment or numbers of individuals assessed. The key is to reach saturation so that additional feedback will be redundant. Once this occurs, further feedback is unnecessary. One limitation is that we used a convenience sample of young adults rather than an online panel for much of the formative steps so that we could work with them in person rather than online or over the telephone. Use of identical recruitment strategies and populations for the formative, pilot, and randomized controlled phases would be preferable. Another limitation is the small sample size for some of the formative steps. Although within the range of recommendations for formative assessment and qualitative research in order to result in saturation (Bastien, 2009; Willis, 2005), a few more participants in some of the phases may have provided additional useful information.

After the pilot trial, a fully-powered RCT was initiated in the spring of 2014 to compare the experimental UV4.me intervention program to the existing Skin Cancer Foundation website and to an assessment only condition. Reports on the RCT are forthcoming. A dissemination trial assessing various strategies to engage young adults in the intervention is also in the works.

4. Conclusions

The preceding report serves as an exemplar of the process of developing a health behavioral internet intervention. It is important to publish detailed descriptions of intervention development procedures in order to help prepare other researchers for the development process and so that reports of intervention trials can be interpreted properly (Barretto, Bingham, Goh, & Shope, 2011). Readers might also want to refer to alternative recent comprehensive approaches to web intervention development including the use of Intervention Mapping (Bartholomew, Parcel, Kok, Gottlieb, & Fernandez, 2011), instructional design (Hilgart et al., 2012), the PRECEDE model (Kattelmann et al., 2014), the process map of Elwyn and colleagues (2011), and the approaches of Barretto and colleagues (2011) and Chee and colleagues (Chee, Lee, Chee, & Im, 2014).