User engagement with web-based genomics education videos and implications for designing scalable patient education materials

Abstract

Genomic medicine has created an urgent need for scalable genomic education. One promising approach is self-guided learning platforms. Understanding how these platforms are used is critical to guide their effective development and implementation. This study contributes a log-based method to study user engagement with online genomic educational videos among participants in a genomic screening study. We collected baseline demographics, logged participant usage and compared pre- and post-education genomic knowledge. Participants (N=390) who chose website access differed from those who declined access (N=81) and were more likely to be non-Latino, English speaking, younger, and have higher educational attainment. Only 45% who accessed the website viewed at least one video. The average video exposure time was 12 minutes. Longer exposure was not associated with an improvement in the user’s genomic knowledge. Our study and future studies of user analytics should be used to guide the development of effective, scalable genomic education methods.

Introduction

The integration of genomic medicine into research and healthcare has created a need for effective, efficient, and scalable genomic education. Traditionally, genomic education is provided by a healthcare provider, often a genetic counselor, in a one-on-one, face-to-face session. With the growing practice of genomic medicine, this model is not sustainable due to workforce limitations, including lack of access to genetic counselors¹, limited expertise among non- genetic providers^2-4, and stringent time constraints for all healthcare providers. Additionally, in the research setting, there is growing consensus that researchers have an obligation to return individual genetic research results though they may not have the knowledge or infrastructure to provide the necessary genomic education⁵. These changes and accompanying challenges raise the need to not only develop alternative genomic education methods, including web- based video methods, but also approaches to measuring the use and effectiveness of these platforms.

Studies have begun to demonstrate the potential effectiveness of alternative genomic education methods. Several studies have shown videos and interactive web-based methods to augment or provide equivalent genomic education to traditional genetic counseling. Web-based tools are also positively endorsed by providers and patients^6-12. A study of non-patient users demonstrated improved genetic knowledge after video education on genomic sequencing¹³. Studies have also demonstrated videos to be effective for non-genetic medical education and associated with improved knowledge, retention of information and patient satisfaction^14-16. With the exception of one study that measured the amount of time a participant spent watching the video¹⁵, in all of these studies, participants’ usage of the tool was known because they accessed the tool in the clinical setting or because they followed prompts to the end of the web- based education; no more detailed user analytics were collected. There is one study using public YouTube videos on hereditary cancer syndromes that collected frequency of views but had only limited information on a fraction of the people who viewed the video and had no measure of effectiveness or satisfaction¹⁷.

While emerging research demonstrates the potential utility of web-based genomic video education, these studies have lacked data on how individual participants interact with these platforms and content, and how this use is or is not associated with a measure of effectiveness (e.g. change in genomic knowledge, participant satisfaction, etc.). A more complete understanding of participants’ use including the minimal user engagement is needed, especially when the education materials are self-guided and administered outside of a clinical setting.

We developed a log-based analysis to measure individual engagement with web-based education videos among the participants enrolled in a study of genomic screening for Ashkenazi Jewish and Latino individuals. We also explored the associations of participant usage of the videos with demographics, change in genomic knowledge and participant perceived benefits of the educational videos. Our study informs how user data can be collected and should be used to guide the development, assessment and use of future genomic education methods in the research and clinical settings.

Methods

We conducted a secondary analysis of IMAGene participants’ study website and video usage. The IMAGene (Individualized Medicine through the Application of Genomics) (www.imagenestudy.org) study is part of the Electronic MEdical Record and Genomics (eMERGE) consortium. eMERGE is currently in Phase III, and in this phase approximately 25,000 people enrolled in parallel consortium studies are receiving results from genomic sequencing¹⁸. The IMAGene study is an arm of the Columbia eMERGE study which offered genomic screening using targeted gene sequencing/genotyping to adults who self-identified as Ashkenazi Jewish or Latino/a, who could read and speak English or Spanish and had received care at NewYork Presbyterian Hospital – Columbia University Medical Center. The restricted IMAGene study sample addressed the larger eMERGE 3 goal of assessing the ability to interpret genomic data from minority populations. The IMAGene cohort is a non-randomized, convenience sampling of individuals who met study eligibility criteria, actively invited through letters and phone calls or passively invited through flyers and community engagement events. Participants were invited to complete the study via a website, on paper through the mail, by phone or in-person. This analysis focuses on those participants who chose to complete the study and genetic education through the website. The study was approved by the Columbia University Medical Center Institutional Review Board.

Following informed consent, all participants completed a baseline questionnaire, which included demographic questions and a 15-question genomic knowledge scale (GKS). The GKS included questions that were adapted from published scales ^19-21. After completion of the baseline questionnaire, participants who elected genetic education via the website were invited to view the content of the study website including the study videos. Participants accessed the secure study website at their convenience using their email address and an assigned password. The participant self- determined when they had sufficient genetic education and were not required to view or watch any content on the website. They were then prompted to complete the GKS again (post-education GKS). Participants had two attempts to answer 12 or more of 15 questions on the post-education GKS correctly. If they were unable to do this, a genetic counselor contacted them before they were able to proceed with the study and have genomic screening.

We developed educational videos and documents for this study to provide self-guided genomic education through the study website https://www.imagenestudy.com. The videos and website content were developed by a team of geneticists, psychologists, genetic counselors, research assistants and community members over the course of 12 months. The videos were reviewed on three occasions by the development team and iterative improvements were made. The website hosts eight videos (Table 1); videos 2-7 were revised to improve clarity on January 27, 2017. Video 5 was removed after March, 2017 because of a change in the study protocol, and analysis of this video is not included in this study. In this analysis, we present user data from the old and new videos together unless otherwise specified. The video topics included genes, inheritance and genetic mutations, genomic screening for genetic conditions, interpretation of different types of screening results (e.g., personal genomic results and carrier screening) and personal, familial and reproductive implications of screening results. The average video duration for the English language videos was 3.5 minutes. The total time to view all videos was 26 minutes. The videos’ Felsch-Kinkaid reading grade level ranged from 9.0-12.0 and 6.4-10.6 when the genetic terms that were defined in the video were removed. The Spanish language video covered all topics and was almost 10 minutes long. It was viewed by only four participants, and therefore no analysis for this video was completed.

Table 1.

Educational videos on the IMAGene Website.

Video ID	Video Title	Video Length (minutes)		Language	Felsch-Kinkaid Reading Level
		Old	New		Full Text	Without Defined Terms
1	Introduction to Genetics	3.50	3.50	English	9.0	6.4
2	Introduction to Genomic Screening	3.53	2.77	English	11.9	7.8
3	Personal Genomic Risk	2.03	3.68	English	11.2	8.6
4	Carrier Screening	2.75	3.53	English	12.0	9.6
5	Pharmacogeneticsa	5.38	2.43	English	12.0	9.7
6	Possible Genomic Screening Results	2.95	7.07	English	12.0	10.2
7	Reproductive Options	1.53	3.58	English	12.0	10.6
8	Introducción a la genética y IMAGene	9.82	9.82	Spanish	n/a	n/a

Videos were updated on January 27, 2017. “Old” indicates the videos before the update, and “New” indicates the updated videos.

^a Video 5 “Pharmacogenetics” was removed from the website in March 2017.

Participants who elected not to have website access received a DVD of the website videos and paper materials that included the same information as that on the website. Since we were unable to measure usage of these offline materials, these participants were not included in our usage analysis. All participants were notified of the option to contact study personnel or the study genetic counselor by phone or email.

The website was accessed by participants on their own devices at their convenience from July, 2016 to August 2018 and unobserved by study personnel. We logged and measured user specific data including website access and duration and type of video viewed and performed a retrospective analysis of the log files of the website with focus on log data for video playing patterns. The atomic unit of the log files is event. Each event corresponds to a type of action. The analysis was performed using Python2.7, and the scripts are available upon request.

We measured the number of participants (i.e., users) who accessed the website, the total number of times they accessed the website, and the length of these accesses. Website exposure is the sum of all the website accesses for each participant. We measured the number of videos users accessed, number of videos users completely viewed, and the total time of video access. Video access is defined by the observation of a “video playing” event. Repetitive accesses to one video counted as one access. The video exposure time is the total length of time for all video accesses including repetitive views. The videos are composed of pages of content which are similar to animated slides in Microsoft PowerPoint, and this internal structure is referred to as a “page” in this analysis. The last page of each video displays study contact information without audio. Video completion was defined as reaching the last page of video. The events recorded in the log files were segmented into sessions to analyze users’ actions on each video. Each event is associated with a page view ID. We created sessions by grouping events based on page view ID prefix, and a session is dedicated to the access of one video from one user.

To enable the evaluation of associations between exposure time and participant characteristics, participants’ video exposure times were divided into three categories: (1) minimal - those who had web access but had less than 120 seconds of video exposure time, (2) moderate - those who had 120 to 600 seconds of video exposure time and (3) maximal - those who had over 600 seconds of video exposure time.

Summary statistics are presented in frequencies, means and ranges. Unadjusted analysis included Chi squared analysis and Fisher exact tests to evaluate categorical variables, and two sample t-tests and ANOVA to evaluate continuous variables. Forward multiple regression models were built to further examine the relationships between video exposure time and change in GKS score. In the simple analysis, demographic and baseline variables associated at an alpha <0.1 with predictor, video exposure time, and the outcome, change in GKS score, were defined as confounders. The final models adjusted for the identified confounders. We completed the analysis for old and new video exposure times separately and combined exposure times. The results did not differ, so we present the combined analysis. Statistical analysis was complete in SAS 9.4²². Given the exploratory nature of this analysis, we did not assign a threshold of statistical significance.

Results

A total of 471 participants were enrolled, and 390 elected access to the website. The participants who elected access differed demographically from those who chose not to access the website. Participants who elected website access were more likely to be non-Latino/a, US born, English speaking, younger than 45 with a greater-than-high-school education, and privately insured (Table 2). Only 14 participants without website access proceeded with the study, and therefore it is difficult to assess differences between these two groups for these variables. The participants who had access to the website performed better on the 15 pre-education GKS (no website access: average 9.6 range 0-15, website access: average 11.8, range 2-15; p-value <0.0001).There was no difference in the change in the GKS from baseline to post education between the two groups (no website access: average 5, range 1-5, website access average 1 range -6-11; p-value 0.19).

Table 2.

Demographics of participants who requested website access and those who declined. N (n is noted next to the variable if data are missing), column percentage. Chi square and Fisher

	No Website Access		Website Access
Baseline Variables	N	%	N	%	p-valuea
Total	81		390
Male	8	10%	104	27%	0.0012
< 45 (n=79)	11	14%	192	49%	<.0001
Latinob (n=79, n=383)	63	80%	221	58%	0.0002
Private Insurance (n=60, n=317)	13	22%	197	62%	<.0001
< High School (n=79, n=383)	55	70%	137	36%	<.0001
English Speaking	26	32%	323	83%	<.0001
Born in US (n=79, n=384)	23	29%	250	65%	<.0001
Web Qc	4	5%	291	75%
Completed post-education Q	14	17%	320	82%	<.0001

Abbreviations: questionnaire (Q), genetic knowledge scale (GKS)

^a comparison of participants with and without website access

^b Non-Latino participants identified as Ashkenazi Jewish

^c 4 participants who completed the questionnaire electronically and completed study enrollment before the website was live.

Of the 390 participants who had access to the website, 303 (78%) accessed it a total of 1166 times. The average number of website sessions per user was 3.8 with a range of 1 to 32 times. The average total time of website sessions per user was 5.8 minutes with a range of 0-50.5 minutes (Figure 1).

Figure 1.

Histogram of total website exposure time

Of the 303 users who accessed the website, 136 (45%) accessed a video in over 617 website sessions. The average video exposure time per user was 13.5 minutes with a range of 1 second to 34 minutes for old videos and 12.5 minutes with a range of 1 second to 45 minutes for new videos (Figure 2 a,b). The combined (old and new videos) average video exposure time per user was 12 minutes with a range of 1 second to 50 minutes. For the 136 participants who accessed a video (old or new), the average number of videos accessed was 3.8 with a range of 1- 7 (Figure 3a,b). The viewing patterns of the individual new videos were assessed. Evaluation of the user viewing patterns showed that the greatest viewer drop-off occurred in the first five seconds. Users who spent more than five seconds on the video typically completed the video to the final page (data not shown).

Figure 2ab.

Histogram of video access time for the old (a) and new videos (b).

Figure 3ab.

Histogram of number of videos accessed for the old (a) and new videos (b).

The frequency of users by video exposure time were: (1) minimal (n=269), (2) moderate (n=53), and (3) maximal (n=68). There were a few differences in the demographics by exposure time; participants who were English speaking and completed the study questionnaires online were more likely to have moderate to maximum video exposure time. People who did not complete post-education questionnaire had lower video exposure time (Table 3). Video exposure times were associated with baseline 15 question GKS score (average 11.5 (range 2-15), 11.9 (8-15), 12.6 (6-15), respectively, p-value 0.003).

Table 3.

Demographics of participants who had access to the website stratified by video exposure time. N (n is noted next to the variable if data is missing), column percentage. Chi squared and Fisher exact.

			Video Exposure Time (sec.)
	<120		120-600		>600
Baseline Variables	N	%	N	%	N	%	p-valuea
Total	269		53		68
Male	73	27%	10	19%	21	31%	0.32
< 45	121	45%	35	66%	36	53%	0.02
Latinob (n=262)	107	41%	23	43%	32	47%	0.64
Private Insurance (n=217, n=47, n=59)	91	42%	15	32%	19	32%	0.23
< High School (n=262, n=43)	99	38%	17	40%	21	31%	0.48
English Speaking	205	76%	51	96%	67	99%	<.0001
Born in US (n=263)	161	61%	34	64%	55	81%	0.01
Web Q	173	64%	52	98%	66	97%	<.0001
Completed post-education GKS	207	77%	46	87%	65	96%	0.001

Abbreviations: questionnaire (Q), genetic knowledge scale (GKS)

^a comparison of the three video exposure groups

^b Non-Latino participants identified as Ashkenazi Jewish

In the unadjusted analysis there was no difference in the change from baseline to post education in GKS across video exposure groups (Figure 4). After adjusting for the confounders of baseline GKS and insurance status, there remained no association between video exposure time and change in GKS score. In the adjusted analysis, the only predictor of change in GKS score was a lower baseline GKS score (Table 4).

Figure 4.

Box plots of the three levels of video exposure times and change in the genetic knowledge scale score (GKS) from baseline to post education. Simple linear regression with video exposure time >600 seconds as the reference group. <120 sec.: average 1, range -6-11; 120-600 sec.: average 1, range -5-6; >600 sec.: average 0.8, range -2-8.

Table 4.

Multiple regression models of three level video exposure time (predictor) with >600 sec. as the reference and primary outcome of change in genetic knowledge scale (GKS) score. Adjusted for confounders association with predictor and outcomes at an alpha < 0.1.

	Estimate	SE	t value	p value	95% CI L	U
0 to <120 sec	-0.32	0.27	-1.17	0.24	-0.85	0.22
120 to <600 sec	-0.19	0.35	-0.53	0.59	-0.88	0.5
Baseline GKS	-0.52	0.05	-10.01	<.0001	-0.62	-0.42
Not Private Insurance	-0.27	0.24	-1.1	0.27	-0.74	0.2

Abbreviations: genetic knowledge scale (GKS), second (sec), standard error (SE) confidence interval (CI), lower (L), upper (U)

Overall, participants rated the educational materials positively with over 85% moderately or strongly agreeing that the website materials helped them understand the different conditions included on the genomic screen, how genomic screening can be helpful to them or their family members and improved their understanding of how genetic variants affect their risk to develop a genetic condition (Figure 5). The overall mean summed score of the website evaluation was 22 with a range of 13-57 on a scale that had a best possible score of 13 and a worst possible score of 84. Participant summed website evaluation score was not associated with video exposure time (data not shown).

Figure 5.

Discussion

This study is one of the first studies to demonstrate the feasibility and value of conducting detailed participant-specific usage analytics when developing scalable genomic education methods. The collection of log-based participant usage data enabled an exploratory analysis of participants’ characteristics and use patterns, which begins to illustrate the challenges of delivering genomic education through non-traditional, scalable platforms. The majority of participants (83%) expressed interest in learning through the website. Of the 390 participants who requested access, a minority 17% of participants accessed more than 10 minutes of videos.

Video usage was not associated with improvement in genetic knowledge as measured by knowledge questions that were part of the study baseline and post-education questionnaires. It is possible that participants accessed other resources through the website besides the videos, for which usage was not directly measured. Additionally, participants may have already had knowledge of the material included within the videos, and the non-random design of the study allowed participants to self-select which videos to view based upon their perceived baseline knowledge.

The lack of participant usage data for the prior published studies limits our ability to compare our results. As with other studies of alternative educational materials in genomics and other healthcare settings, the majority of the participants who accessed to the website endorsed the benefits of the information^8,11,13,16. Our experience differs from prior studies with regard to a change in participant genomic knowledge and other measurable participant outcomes. These differences are potentially related to a variety of factors though likely a significant factor was the study setting. Our educational tool was used at home and was self-guided, did not have any required content, and the education was not supervised in a clinical or research appointment.

Many of the other studies have administered educational intervention in a healthcare setting or in advance of a clinical appointment^6,10. We suspect that their participant usage was higher, because in those settings participants were supervised and given a task to complete and may have felt greater obligation to be compliant as opposed to an unsupervised home setting. However, to be truly scalable, education will need to be delivered in a manner that requires minimal provider supervision and is convenient for the individual. Additionally, our participants who were having genomic screening may have had lower motivation to view the videos compared to participants in other studies who were receiving focused education about a specific indication for which they were referred. Collection of participant usage data across different clinical scenarios will help to better elucidate differences in participant use and influencing factors.

Our experience supports prior concerns that web-based learning is not desirable for all participants and may exacerbate healthcare disparities ^23,24. Participants who declined access to the website were older, had lower education levels, were more likely to be immigrants, speak Spanish as their primary language and have Medicare or Medicaid than those who chose to access the study website. While it was not measured, they likely had less access to internet devices. These participants as well as participants who requested access to the website but had minimal video usage were much more likely to choose to complete the study questionnaire on paper. In this era of increasing web and electronic based studies, there are individuals who do not have web access and/or are less comfortable learning via a website or video and completing electronic forms. Furthermore, despite efforts to make the website accessible to as many people as possible with translation of content into Spanish, very few people who spoke Spanish as their primary language had meaningful video usage. Other studies have demonstrated effective video health education in minority populations and low literacy populations when the tools have been developed for these specific populations ^25-30. This suggests one single tool will not meet the needs of all, and it may be necessary to develop multiple tools to address the variable needs of different populations and ensure maximum effectiveness.

The pattern of participant usage provides important insights for future development of self-guided tools. The greatest decline in views occurred in the first five seconds of a video, and participants who watched a video in its entirety were more likely to watch other videos than those who did not. The educational videos developed for this study were visually basic with minimal animation and a voice over. Information was presented in a manner similar to a genetic counseling session; in accessible language and using analogies to explain complex information. This is likely not the most engaging manner in which information could be provided, but our strategy was limited by the research budget. Potentially platforms that immediately engage users with interesting and motivating stories and visually interesting illustrations could capture the viewers’ attention and motivate viewers to continue to watch and increase overall usage.

There are several limitations to our study. Overall, the results of this study are exploratory given the modest sample size. Future, larger, more inclusive studies are needed to confirm our findings. Our study was also restricted to participants who identified themselves as Latino or Ashkenazi Jewish and requested the receipt of genomic screening results; therefore, our findings may reflect unique perspectives of these populations and people who elect to have genomic screening results. Additionally, while the web-platform and video education were available in both English and Spanish languages, only English-speaking participants had meaningful use of the videos. The GKS was an adaptation of other scales and was not validated, and it is possible that there were educational benefits to using the website that were not captured by the GKS. The videos were modified mid-way through the study to reflect changes in the study protocol and make changes to improve the content and presentation; though duplicate analysis of exposure to old or new videos alone did not differ from the analysis of the combined video exposure.

Conclusion and Practice Implications

The development of scalable genomic education delivery methods is necessary but is more challenging than anticipated. More user testing with detailed user analytics is needed during the development stage to identify and adequately address issues of user engagement and retention. Overcoming these hurdles will require collaboration with professionals specialized in the development of interactive websites and video education in order to identify ways to deliver information in an interesting and engaging fashion. Because of the cost associated with robust development, the genomics community needs to collaboratively develop scalable and adaptable tools. Similar to the way genomic sequencing data are shared to advance collective knowledge, existing and emerging health information technologies including effective genomic education methods should be leveraged to enable tailored, layered, and large-scale communication ⁵.

More studies that include detailed user analytics are warranted to continue to explore the effectiveness of novel methods of genomic education in more diverse groups of individuals. Finally, we need to appreciate that even with scalable effective genomic education, there will be individuals for whom traditional models will be preferred and more effective based upon the characteristics of the individual and the clinical context.