Is Your WebLitLegit? Finding Safe and Good Health Information on the Internet

Abstract

Background:

There exists a paucity of literature about teenager health literacy in general and teenagers are likely to turn to the internet for health information. Therefore, they need good e-health literacy to properly understand and apply the information obtained. Yet, many have limited e-health literacy, lacking the knowledge and skills to filter and distinguish reliable from unreliable health information and searches return large amounts of information, making it difficult to recognize whether information is reputable and raising concerns regarding teenagers' safety.

Brief Description of Activity:

We developed a toolkit in collaboration with community-based organizations serving teenagers and teenagers themselves usable with brief training to present a 1-hour, interactive workshop. We transformed current adult information for locating and appraising online health information into a teenager friendly format using relevant health topics to engage participants.

Implementation:

We met teenagers in teenager-friendly settings where they already gather to engage them and leverage the relationship fostered within those settings to bridge positive and negative social determinant influences on health literacy and e-health literacy as well as cross potential cultural, economic, political, and demographic barriers. Using the “train the trainer” method to build sustainability, we trained teenagers and group leaders to use the toolkit to run workshops with teenagers and placed the components in an easily available online format.

Results:

After completing the workshop, teenager participants expressed a high level of confidence in using Medline Plus, locating health information online, identifying Truth versus Trash and making health decisions. Most teenagers reported they would recommend the WebLitLegit workshop to their friends and it improved their ability to find credible online health information.

Lessons Learned:

The workshop's practical application provided participants with real-life examples for evaluating online information using the “LEGIT” acronym. The integration of this community-based program fostered relationships between the teenager participants, community organizations, and university students and faculty. All of the organizations involved benefited through exposure to health literacy concepts and knowledge of evaluation criteria, which may help expand e-health literacy in the community because the students, teenagers, and community partners are able to sustainably share the toolkit within their social network. [HLRP: Health Literacy Research and Practice. 2022;6(2):e151–e158.]

Plain Language Summary:

Teenagers use the internet to find health information but have difficulty deciding if the information is correct and safe. WebLitLegit workshops help teenagers find correct and safe information to make health decisions. Teenagers completing the workshop thought their ability to find correct information and make good health decisions improved. This best practice adds to the literature by addressing needed teenager e-health literacy.

There exists a paucity of literature about adolescent health literacy (Caldwell & Melton, 2020). Ghaddar et al. (2012) estimated that 48% of adolescents in their study had low health literacy. Earlier studies mostly focused on associations of health literacy with social determinants of health. Adolescent health literacy has been associated with race, indicating that adolescents from underrepresented racial and ethnic populations had lower health literacy scores than their White counterparts (Caldwell & Melton, 2020; Manganello & Sojka, 2016). In terms of income, adolescents living in lower income households were more likely to be assessed with low health literacy than those from higher income families (Caldwell & Melton, 2020; Manganello & Sojka, 2016) and were more likely to experience social determinants of health that set them up for poor chronic illness management, less ability to use health information, and difficulties following medical instructions into adulthood (Chisolm et al., 2015). Adolescent health literacy has also been positively associated with parent education and parent health literacy. Lower health literacy scores were found in adolescents with parents who only achieved a high school diploma or less education and parents who themselves had low health literacy scores (Chisolm et al., 2015).

Adolescents are likely to turn to the internet for health information (Sansom-Daly et al., 2016). Therefore, they need good e-health literacy to properly understand and apply the information obtained. Norman and Skinner (2006, p. 2) defined e-health literacy as “the ability to seek, find, understand, and appraise health information from electronic sources and apply the knowledge gained to addressing or solving a health problem.” Many adolescents are highly connected and feel comfortable using the internet to find health information (Stellefson et al., 2018) while 70% of youth use the internet as their first source of health information (Sansom-Daly et al., 2016). Yet, many have limited e-health literacy, lacking the knowledge and skills to filter and distinguish reliable from unreliable health information on the internet (Freeman, et al. 2018; Jain & Bickham, 2014; Stellefson et al., 2018). Locating health topics on the internet is challenging because searches return large, robust amounts of information, making it difficult to recognize whether information is reputable (Park & Kwon, 2018) and raising concerns regarding safety because using inaccurate information could potentially cause harm from wrong treatments or actions (Stellefson et al., 2018).

Youth reported difficulty understanding online health information, ascertaining the quality, truth, or correctness of the information and whether information online was valid and reliable (Esmaeilzadeh et al., 2018). For example, in a recent systematic review, Park and Kwon (2018) found adolescents were confused by the information they found online about sexual health; those with low health literacy mistakenly rated misinformation as accurate more frequently in comparison to their counterparts with adequate health literacy. Also, Manganello & Sojka (2016) determined that adolescents with low health literacy may use additional unreliable sources to obtain health information. Consequently, they may obtain poor quality or even misinformation about health for themselves, their family members, or friends. As the internet and social media continue to be avidly used by adolescent populations, so does the potential spread of misinformation and possible harmful outcomes when this information is applied to adolescents and their health. Adolescents indicated they may search online for information about alcohol, conflict resolution, vaping, healthy relationships, peer pressure, suicide, bullying, cutting, eating disorders, homicide, marijuana, and pregnancy (Null, 2019). These findings are concerning given existing health inequities (Centers for Disease Control and Prevention [CDC], 2020), which may be compounded by a lack of understanding of health information given the seriousness of the identified topics important to adolescents (Null, 2019).

Customized instruction that improves e-health literacy for adolescents is needed (Stellefson et al., 2018). Overall, current literature recommends that improving health instruction in schools could improve e-health literacy in adolescents, helping them distinguish between good and bad information (Manganello & Sojka, 2016; Stellefson et al., 2018). WebLitLegit, an interactive, adolescent-driven toolkit that empowers adolescents to discern health information Truth versus Trash on the internet addresses this important need. The toolkit offers customizable instruction for adolescent populations both locally and nationally while meeting the Healthy People 2020 objectives to increase the proportion of online health information seekers who report easily accessing health information. WebLitLegit empowers adolescents through knowledge and skills gained despite potential social determinant barriers to obtaining reliable, usable information on adolescent health. The interactive tools and workshop provide instruction empowering adolescents with the ability to locate and understand reliable health information found on the Internet, appraise the resources found and effectively apply the information to make informed health decisions.

Brief Description of Activity

Development

Our goal was to develop a toolkit that anyone with brief training could use to present a one-hour, interactive workshop. We garnered a team of volunteer adolescent and adult community consultants from supporting organizations: (1) A youth serving organization for those in need; (2) A Museum of Science and History Research and Learning Center; (3) A Pediatric Medical Center Library; (4) An Independent School District; and (5) A Children's Medical Center. Depending on the setting, adolescents and/or parents were provided information about the project inviting the adolescents to participate. Supporting organizations integrated the project into existing teaching curriculum. With assistance from current undergraduate, graduate, and medical students studying with the researchers, each organization participated in semi-structured focus groups, brainstorming, vetting ideas, and information. They also helped with developing creative materials and providing feedback throughout the development of the toolkit including the acronym (LEGIT; see Figure 1), slogan (Truth versus Trash), hashtag (#knowtruthfromtrash) and logo. To ensure that we were meeting adolescents' needs, we collaborated with adolescents and operationalized their feedback throughout the development process.

Figure 1.

Finding good health information on the internet.

Additionally, we assessed and transformed current adult information for locating and appraising online health information into an adolescent friendly format. Utilizing the MedlinePlus Evaluating Health Information webpage and the National Institutes of Health (NIH) National Center for Complementary and Integrative Health Finding and Evaluating Online Resources, the authors conducted adolescent focus groups to inform the overall development of the WebLitLegit workshop from the adolescent perspective. These focus groups intended to promote appeal to the adolescent audience. The workshop can be presented in-person or virtually with use of technology (tablets or their own devices) to give adolescents a feel for searching for reliable health information online.

Toolkit

The team operationalized heutagogy, a form of self-determined learning that develops learner capabilities through an active and proactive process, while developing the WebLitLegit materials and workshop (Blaschke, 2012). Workshop trainers facilitated the learning process by providing guidance and resources on how to locate and appraise health information on the internet, but the adolescents actively utilized them to build their own autonomy, capacity, and capability (Blaschke, 2012). An INSERM (French public research institute dedicated to health) advisory group (2001), noted when teaching health information to youth, improved knowledge does not necessarily lead to changes in behavior. To encourage new behaviors, education must not only be relevant topics for youth, but also the methods of instruction must be diverse. Thus, each component of the workshop had an objective to present information using a variety of teaching/learning strategies that targeted different learning styles and that emphasized active learning.

To engage adolescents, we needed to use adolescent relevant health topics. The CDC identified four focus areas including sexual behavior, high-risk substance use, violence victimization as well as mental health and suicide (CDC Division of Adolescent and School Health [DASH], n.d.). Adolescents have also identified safety issues for which they may even be missing school including electronic bullying or bullying at school (CDC DASH, n.d.). Similarly, findings from the National Youth Risk Behavior Survey identified mental health issues possibly resulting in harm or death continue to increase at an alarming trend with thoughts surrounding suicide and actual attempts rising steadily from 2007 to 2017 (CDC, 2018). An informal survey by the Boys & Girls Clubs called “Matters of a Minor” conducted with adolescents in the 6th through 12th grade that attended the club confirmed that locally adolescents were reporting similar topics to those identified in the national surveys (Null, 2019).

To address these adolescent priority health issues, we identified credible health information websites to use as examples such as Medline Plus, WebMD Health, Planned Parenthood (https://www.plannedparenthood.org/learn/teens/relationships/relationships-101), and Teens Health (https://kidshealth.org/en/teens/sexual-health/and https://kidshealth.org/en/teens/bullies.html). To provide contrast to these credible sites, we located non-credible website exemplars that addressed similar health topics.

WebLitLegit YouTube video. We started with a YouTube video knowing that videos support the learning of new content, concepts, and ideas by building background knowledge (Alber, 2019). Our team created the 3-minute video featuring diverse adolescents responding in a question-and-answer format to give an interview style feel. Adolescents provided answers to how they found health information, what health concerns they and their friends have looked up, what they found when they looked up health information on their phone, and how they knew whether information was reliable. The adolescents shared that they used their phones to search for health information, finding vast number of ads for medications, many pages of information to sort through including, “sketchy websites” and the difficulty of determining what was Truth versus Trash. Some adolescents stated that they would rely on the first link that was listed and admitted that they could not really tell whether a website provided reliable information. Adolescents acknowledged their confusion, feelings of being overwhelmed and skepticism of the information they found online. The video closes with the adolescents verbalizing the acronym LEGIT as their newfound way to appraise health information as Truth vs Trash on the internet.

Our goal with the video was to present the difficulties adolescents might experience when searching for health-related information on the internet. The approach was aimed at showing those participating in the workshop, that their peers, as presented in the video, experienced the same difficulties finding reliable health information on the internet, applied the LEGIT acronym to test for reliability or Truth versus Trash through a step-by-step evaluation operationalizing the content each letter represents (Figure 1) for evaluation to conclude the reliability of the information.

Engaging slide show. We developed a 20-slide PowerPoint presentation to introduce the concept of e-health literacy, five easy steps for evaluating health websites, using the LEGIT acronym with the meaning of each letter and examples of Truth versus Trash websites to point out the red flags of noncredible websites. Trainers have the option of using a 5-minute pre-narrated or an unnarrated version, depending on their preference for leaving discussion for the end or interspersing discussion throughout. Although the presentation is dense with information, we kept it short to avoid “talking at” the participants and to instead foster engagement.

Interactive component: Locating and evaluating truth versus trash health sites. To reinforce learning, adolescents applied the LEGIT acronym to evaluate websites as Truth Versus Trash with two exercises. First, using QR codes that linked to websites, adolescents used their phone cameras to get to examples of Truth Versus Trash websites. With the LEGIT acronym displayed either on a screen or a handout, adolescents evaluated the credibility of each site. After allowing time for them to look through the websites, adolescents returned to a whole group discussion to determine if the website was Truth Versus Trash. Second, adolescents looked up health topics on their own. Knowing that adolescents may be sensitive to choosing and discussing a topic of their own choosing, we had a jar with a variety of pre-determined health topics; most adolescents chose to pick a topic at random from the jar, eliminating embarrassment for picking birth control or anxiety, for example. After selecting a topic, adolescents used their phones to locate and evaluate health websites. The trainer circulated the room to have individual check ins and discussion about located sites. Adolescents shared their findings with the group. These interactive exercises reinforced the concept of LEGIT, Truth Versus Trash, as adolescents actively applied their skills to locate and appraise health information on the internet.

Jeopardy and Kahoot. We developed a Jeopardy game to again reinforce LEGIT and how to tell Truth Versus Trash health information websites. After playing the Jeopardy game, adolescent partners at a local high school developed an accompanying Kahoot game. Both the Jeopardy and Kahoot games can be played with individuals or teams, depending on the size of the workshop group.

Evaluation

Adolescent participants provided feedback at the end of the workshop. Using a QR code, adolescents used their phones to complete the evaluation survey. Participants answered questions evaluating the workshop and self-assessing their knowledge and confidence in skills searching for health information on the internet. Some questions were standard items for National Library of Medicine data collection. Adolescents used a 4-point Likert scale to respond to survey items selecting from Likely, Somewhat Likely, Somewhat Unlikely, and Unlikely.

National Library of Medicine item examples:

• “How likely are you to recommend the WebLitLegit workshop to your friends?”

• “I plan to start using at least one resource or tool that I learned about in this training.”

• “I plan to tell others about at least one resource or tool that I learned about in this training.”

Other items sought for adolescents to assess their confidence and behavioral intent using a 4-point: Likert scale to respond to survey items selecting from Completely, Pretty, A Little, and Not Confident.

For example:

• “How confident do you feel about locating health information on the internet?”

• “How confident do you feel about telling reliable from unreliable health information on the internet?”

• “How confident do you feel about making health decisions using health information on the internet?”

Implementation

Trainings

Our plan for uptake and sustainability met adolescents in community partners' adolescent-friendly settings where they already gather to engage them in both toolkit development and taking the training. Thus, the plan leveraged the relationship fostered within those settings. “Involving young people at every phase of the development process helps ensure that the intervention will be valued and sustainable” (World Health Organization, 2020, p. 12).

Using the “train the trainer” method, we aimed at training adolescents and group leaders to use the toolkit to run workshops with adolescents. The goal was to train 17 trainers at the original five partner sites; we encouraged community partners to identify interested adolescents who wish to be trained in the workshop to offer to their peers. Each partner site committed to conduct workshops in partnership with the project team or on their own for a total of at least 30 workshops that aimed to reach 300 to 400 adolescents in the North Texas area between October 2019 to April 2020.

To ensure translation and sustainability, the online toolkit was made available for trainers. It provides easy access to the materials used in the WebLitLegit Workshop. In the training, trainers were “walked through the workshop:” YouTube video, PowerPoint, how to use QR codes in the interactive exercise to locate and appraise the #truthversustrash using LEGIT, the Jeopardy and Kahoot games, and the evaluation. Trainers practiced coaching skills and how to complete a teach-back demonstration with an adolescent locating and appraising a health website.

Results

Process Objectives

Overall, we trained 20 teenager trainers and 18 adult trainers at our original partner sites and four additional participating venues with 2 to 4 trainers trained per site. The original project time frame was extended due to the onset of coronavirus disease 2019 (COVID-19) and the need to convert workshops to a virtual format. For those who completed the evaluation, behavioral and learning outcomes are depicted in Figure 2.

Figure 2.

Outcomes by percentage of participants.

Each partner site completed their committed workshops in collaboration with the project team or on their own for a total of 31 workshops held with adolescents in the North Texas area between October 2019 to October 2020. One hundred forty-eight adolescents attended the WebLitLegit workshops with 93 completing the evaluation at the end of the workshop. Both trainers and participants became engaged in the interactive lesson and reported that they forgot the evaluation. After learning this, we embedded the evaluation into the lesson to prompt trainers to instruct adolescents to fill out the evaluation at the end of each workshop. Adolescents with limited phone access were also unable to evaluate. The grade level of the participants completing the evaluation (n = 93) ranged from 6th to 12th grade with 6th to 8th grade having 1 (1%) participant each, 9th grade having 2 (2%), and 10th grade having 7 (8%) participants. The remaining participants who completed the evaluation in the workshops were in the 11th grade with 16 (19%) participants and in 12th grade with 38 (45%). There were 18 (21%) who did not indicate grade level and 9 (10%) skipped this question on the evaluation.

Outcome objectives. Through the WebLitLegit program, we achieved both outcome-based and process objectives. Outcome objectives were set at a goal of 50% achievement of participants' perceived, self-assessed greater confidence than prior to the workshop of being able to use Medline Plus, locating health information on the Internet, using the LEGIT checklist to ascertain Truth Versus Trash and applying the credible health information they find for making informed health decisions. After completing the workshop, participants expressed a high level of confidence (completely or pretty confident) in using Medline Plus (85%), Locating Health Information Online (82%), Identifying Truth Versus Trash (85%), and Making Health Decisions (76%).

Most adolescents reported that they would recommend the WebLitLegit workshop to their friends (91%), thought workshop was right amount of time (98%), reported that workshop introduced them to at least one new health information resource that they intend to use (96%), and strongly agreed or somewhat agreed that they planned to share at least one resource they learned (93%). All adolescents reported that they learned a new skill that they planned to use in the future and almost all reported that the workshop improved their ability to apply a resource they already used (98%) and strongly agreed or somewhat agreed that the WebLitLegit workshop improved their ability to find useful online health information (97%) (Table 1).

Table 1.

Process Results

Process Objective	Number Achieved
Teen and adult community members involved in development	Total 65
	Adults 8 (13%)
	Teens 57 (87%)
Count of WebLitLegit trainers in the community	Total 38
	Adults 18 (47%)
	Teens 20 (53%)
Count of sites that hosted	8
WebLitLegit sessions
Social marketing efforts, toolkit website placement	112,867 social media clicks during health literacy month
Teens reporting effective teaching methods	90 (97%)
Teens that reported they would use the information	93 (100%)a
Teens reporting useful online health information	90 (97%)

^aOne hundred forty-eight adolescents attended the WebLitLegit workshops with 93 completing the evaluation at the end of the workshop. Thus, 100% of those completing the evaluation reported they would use the information.

Most adolescents reported the workshop used effective teaching methods (97%) and that they would use the information they learned (100%). Adolescents also deemed the online health information provided in the toolkit useful (97%) for their everyday lives.

Lessons Learned

This article adds to the paucity of literature on adolescent health literacy and hopefully bolsters the literature in support of interactive workshops and tools to address the general lack of adolescent e-health literacy (Stellefson et al., 2018). Health professions students and faculty, community organizations, and workshop leaders operationalized community engagement and partnerships to support heutagogy learning principles as powerful tools for improving adolescents' health information and e-health literacy (Blaschke, 2012). The workshop's practical application provided participants with real-life examples of how to evaluate online information using the “LEGIT” acronym. The integration of a National Library of Medicine (NNLM) grant-funded community program fostered relationships between the adolescent participants, community organizations, and university students and faculty.

The undergraduate, graduate, and medical students who helped develop the workshop gained valuable skills in program design and implementation, community building, curriculum planning, project execution and evaluation. Adolescent participants benefitted from the workshop or the opportunity to conduct peer-to-peer education workshops through community groups including libraries, schools, Boys & Girls Clubs, museums, and other youth-serving venues in the Dallas-Fort Worth area. The WebLitLegit workshop equipped adolescents with the knowledge, skills, and self-efficacy to locate, appraise, and apply health resources on the internet, using reliable NNLM and NIH adolescent health information sites as a backdrop. These resources provided through the workshop enabled real-life interaction with the opportunity to receive direct feedback allowing students to practice and apply their skills. Working with adolescent-serving community organizations to develop the course greatly expanded the relatability of the workshop for adolescents.

The community partners contributed to the program's success through their interest in improving adolescents' health information literacy and e-health literacy. Although the intent of the workshop originated to empower self-sufficiency in community organization execution to future adolescent groups, many continued to depend on the faculty partners to execute rather than executing on their own. However, the connections made helped the organizations recognize academia as a valuable partner. Upon completion of the project, the faculty wrote this article and presented the work and lessons learned in different formats to a variety of audiences throughout health literacy month.

All of the organizations involved benefited through exposure to health literacy concepts and knowledge of evaluation criteria, which may help expand e-health literacy in the community because the students, adolescents, and community partners are able to share the toolkit sustainably within their social network. One of the partnering children's medical centers has further adopted the WebLitLegit workshop into their adolescent to adult transition program. Coupled with interactive activities, the workshop yields effective and exciting experiences for teenager participants that may impart health information seeking and e-health literacy skills to evaluate online health information sources for credibility. The toolkit located on a publicly available website, lends as an ongoing resource to a wide range of adolescent settings for delivery by a variety of diverse educators such as school librarians, public librarians, secondary school teachers (health, science), school nurses, museum educators, community programs and community volunteers. Additional groups including medical student educators, health care providers and community health workers have used the workshop to educate themselves and their constituents for work within the community. We also found that the workshop was easily converted to a virtual format for use during the pandemic. Most recently, the toolkit resources have been used to evaluate COVID-19 information sources. One limitation included that those adolescents without smart phones or with limited data, lacked the ability to participate on their own in the workshop. In this instance, they looked on with other participants or borrowed facilitator's phones. This reflects greater barriers to looking up health information in general, but also disparities of accessing credible health information. Moving forward, having adolescents partner within each workshop could alleviate any shame associated with this factor as well as having paper copies available for the evaluation.

Aligning with the mission of the NNLM, WebLitLegit met stated goals to teach and promote health information literacy and e-health literacy of adolescents in North Texas. All participants reported that they learned a new skill that they plan to use in the future. WebLitLegit also aligns with the Healthy People 2030 (Office of Disease Prevention and Health Promotion, n.d.) adolescent objectives which focus on helping adolescents stay safe and healthy. Another program, HackHealth, targeting youth ranging from age 11 to 13 years provides an immersive experience to improve e-health literacy in a 12-week afterschool program in collaboration with school librarians (Subramaniam & Casciotti, 2015; Subramaniam et al., 2015). In contrast, for community-based youth organizations lacking the resources for a longer program, WebLitLegit offers a brief introduction and practice of e-health literacy skills easily accessed and disseminated. Future studies might evaluate the effectiveness of the WebLitLegit program. The 1-hour workshop helps youth stay safe through avoiding trash information that might harm their health and keep them healthy by using truth or appropriate information to make informed health-related decisions. According to Healthy People 2030 (Office of Disease Prevention and Health Promotion, n.d.), “Adolescents are at risk for many preventable health problems, including substance use disorders, sexually transmitted infections, and injuries from motor vehicle crashes.” Additionally, WebLitLegit aligns with Healthy People 2030 communication objectives to increase the health literacy of the population. WebLitLegit lends to national scale-up as it provides an innovative, easily disseminated and implemented option for e-health literacy education to help achieve these Healthy People 2030 objectives.