Identifying Effective Approaches for Dissemination of Clinical Evidence – Correlation Analyses on Promotional Activities and Usage of a Guideline-Driven Interactive Case Simulation Tool in a Statewide HIV-HCV-STD Clinical Education Program

Abstract

Dissemination of the latest clinical evidence to community-based healthcare providers is a critical step to translate biomedical knowledge into clinical practice. We performed a study to analyze the correlations between the promotional activities and the usage of a guideline-driven interactive case simulation tool (ICST) for insomnia screening and treatment in a statewide HIV-HCV-STD clinical education program. For this purpose, we tracked users’ interactions with the ICST and the sending of promotional email newsletters during a study period of 44 weeks. Results showed that promotional activities were strongly correlated with the number of audience as well as the intensity of use of the target resource. The strength of correlation varied in specific use contexts. Strong correlations were found between the sending of email newsletters and the intensity of resource use by promotion recipients, by new users, and through the most convenient access channel associated with the promotion. Selection of approaches for resource dissemination should consider the potentials and limitations of use contexts to make them more effective.

Introduction

Effective and timely dissemination of the latest clinical evidence to community-based healthcare providers is a critical step to translate biomedical knowledge into clinical practice. Dissemination and implementation research has become a national priority of the United States to harness the investment in biomedical research with the improvement in clinical care and public health [1]. The informatics research community has accumulated a critical mass of knowledge on effective approaches to disseminating clinical evidence, including use of online platforms to host clinical and educational resources and leverage of information and telecommunication technologies to promote them [2–3]. Nevertheless, few prior studies have reported on how promotional activities can improve the actual usage of online resources, and particularly, in what specific contexts the resource usage tends to respond to promotions.

In this paper, we report a pilot study to analyze the correlations between the usage of a guideline-driven interactive case simulation tool (ICST) and resource dissemination activities [4–6]. The results from this study will: (1) identify how dissemination activities can increase the usage of online clinical and educational resources; (2) characterize the profiles of use contexts that respond to promotions; and (3) direct the future development of effective approaches to disseminating clinical evidence.

Materials and Methods

Target Resources

The target resources for dissemination in this study were from the New York State (NYS) HIV-HCV-STD Clinical Education Initiative (CEI) online training program [7]. The CEI program is sponsored by NYS Department of Health (DOH) AIDS Institute, targeting primary care clinicians such as physicians, nurse practitioners, pharmacists, dentists, physician assistants, dental hygienists, case managers, social workers, counselors, and other team members providing care for HIV-HCV-STD patients. CEI aims to increase access to quality HIV-HCV-STD healthcare, to expand the base of clinicians who can effectively manage HIV-HCV-STD patients, to disseminate the latest clinical guidelines, and to foster partnerships between community-based care providers and HIV-HCV-STD specialists. Since 2008, CEI has initiated the online training program and developed various digital resources, including 235 multimedia learning modules, 93 online CME/CNE courses, and 12 guideline-driven ICSTs [8–9]. These resources can be accessed from multiple platforms, including a main website, a mobile website, Android and iOS mobile apps, online social networks and social media, RSS feeds, and newsletters sent through CEI email listserv. Over the five-year period since its launch, the CEI website has recorded 121,150 visits and 608,487 pageviews by audiences from 171 countries around the world [6,10]. It is now consistently ranked by Google and other search engines as a top site for HIV-HCV-STD clinical education.

ICST

ICSTs are online tools for clinical decision assistance and case simulation on individual patients [4, 11]. When used by healthcare providers, they serve as vehicles to dessiminate the latest biomedical knowledge. On the back-end, ICSTs are typically based on knowledge-bases adopted from clinical practice guidelines. For example, the Insomnia Screening and Treatment ICST selected as the resource for analyses in this study was based on a quick reference guide for HIV primary care clinicians, a simplified version of the full guideline developed by the NYS DOH AIDS Institute [4, 12]. On the front-end, ICSTs support user interactions such as review of patient management processes, examination of different options for clinical decisions, and entry of case-specific data for individualized recommendations. In addition to user-defined patient cases, an ICST can include a list of predefined sample cases presenting the typical scenarios for clinical decision making as well as a text recommendation section with highlighted points adopted from the source guideline. These three sections, i.e., recommendation, sample case, and user-defined case, provide the complete user functions for an ICST [4]. The ICSTs can be accessed from the CEI website as well as mobile apps. Screenshots of the Insomnia Screening and Treatment ICST are shown in Figure 1.

Figure 1 –

Screenshots of the Insomnia Screening and Treatment ICST

CEI Resource Promotion

To ensure that its resources can reach out to the target clinicians and communities, CEI has been consistently engaging in program promotions through multiple channels. Among these, CEI newsletters sent to email listserv have been used most freqently. The CEI newsletters are regular updates of the latest CEI clinical and educational resources, such as the newly developed CME/CNE courses, multimedia learning modules, mobile tools, training events and news, etc. These newsletters contain hyperlinks pointing to the CEI website and other online platforms that host the various clinical and educational resources. For many years, we have been sending these newsletters to the CEI audience through email listserv every two to three weeks. The number of subscribers of this listserv is in the range of 2,200–2,600. When a listserv subscriber receives a CEI newsletter and finds interest in a specific item, he/she can click the hyperlink to check the details of that resource on the CEI website. He/she can also forward the email to colleagues or friends to further disseminate these resources through his/her professional and social networks [6, 13].

Study Design and Data Collection

To analyze the correlations between the ICST usage and promotional activities, we first collected the usage data from April 3, 2012 (when the Insomnia Screening and Treatment ICST was released) to February 4, 2013 (a period of 44 weeks) through tracking of user interactions with the system [4]. We focused on two usage measures: (1) episodes of use (reflecting the number of audience); and (2) frequency of visits to specific part of the ICST (reflecting the intensity of use) [4]. For this purpose, we built customized tracking functions to monitor the use of specific functions on different system user interfaces. System and user events, such as click of a button, check of a decision branch, and selection of a specific patient condition, as well as the timestamps of these events and user sessions, were stored in a tracking database for analyses. To control potential biases, usage by internal staff and for testing of app installation was excluded from the analyses [4].

To characterize the use context, we profiled user interactions by multiple dimensions:

1. Sections of ICST system function, i.e., user-defined case, sample case, recommendation, and cross-box (usage spanning over two or three of the previously listed sections);

2. New vs. returned users;

3. Access from large-screen equipment (desktops, tablets, etc.) vs. small-screen hand-held devices (smartphones, iPods, etc.);

4. Access through web browsers vs. native apps;

5. Audience from the United States vs. non-US countries;

6. Audience from the NYS vs. out of the state; and

7. Audience from healthcare organizations, government agencies, and unknown settings.

Additional technical details on tracking of user interactions, use contexts, and usage measures can be found in our previous publications [4, 13–14].

To collect the data of CEI resource dissemination through newsletters, we reviewed all the archived emails, selected those newsletters directly related to ICSTs, and recorded the date when they were sent.

Data Analyses

For data analyses, we segmented the total usage as well as the profiles of use context by the dimensions listed above as biweekly time-series. We plotted dissemination activities, i.e., sending CEI newsletters through emails, as a binary variable in a bi-weekly time-series with the same formulation. We then performed correlation analyses between each combination of usage profile/measure and the dissemination activities. We calculated Pearson correlation coefficient (r) to determine the potential correlations. We considered |r| >= 0.5 as a strong correlation, 0.5 > |r| >= 0.3 as a moderate correlation, and |r| < 0.3 as a weak or no correlation [15]. For statistical analyeses, we used the SPSS software package [16].

Results

A total of 298 episodes and 1,415 rounds of interactions (visits) with the Insomnia Screening and Treatment ICST were recorded during the study period. Meanwhile, five CEI email newsletters were sent during this period to promote the ICSTs. The detailed distributions of the total ICST usage, its use in specific contexts, and the sending of email newsletters presented as bi-weekly time-series are shown in Figure 2.

Figure 2 –

Sending of CEI Email Newsletters, Total ICST Usage, and ICST Use in Specific Contexts during the Study Period

Pearson correlation coefficients showed strong correlations between promotional activities and the following usage profiles:

1. Usage measured by episodes on total usage; and

2. Usage measured by frequency of visits on recommendation, by new users, through web browsers, by audience from the United States, and by audience from healthcare organizations and government agencies.

Meanwhile, Pearson correlation coefficients showed that there were moderate correlations between promotional activities and the following usage profiles:

1. Usage measured by episodes on recommendation, by both new and returned users, from large-screen equipment, through web browsers, by audience from the United States, by audience from both inside and outside of NYS, and by audience from all settings; and

2. Usage measured by freqency of visits from large-screen equipment, by audiences from both inside and outside of NYS, and on total usage.

Weak or no correlations were found between promotional activities and the following usage profiles:

1. Usage measured by episodes on sample case and user-defined case, from small-screen hand-held devices, through native apps, and by audience from non-US countries; and

2. Usage measured by freqency of visits on sample case and user-defined case, from returned users, from small-screen hand-held devices, through native apps, and by audience from non-US countries.

The detailed results of the correlation analyses, including r and p-value for each usage profile, are summarized in Table 1.

Table 1 –

Results of Correlation Analyses

	Episodes		Freq. of Visits
	r	p-value	r	p-value
Recommendation	0.470	0.024	0.551	0.006
Sample Case	−0.091	0.679	0.224	0.303
User-Defined Case	−0.066	0.764	0.270	0.213
Cross-Box	0.278	0.198	0.437	0.037
New User	0.492	0.017	0.538	0.008
Returned User	0.446	0.033	0.122	0.579
Large Screen	0.422	0.045	0.486	0.019
Small Screen	0.282	0.192	0.271	0.211
Web Browser	0.460	0.027	0.585	0.003
Native App	0.113	0.607	0.173	0.429
Outside of USA	0.284	0.189	0.219	0.315
Inside of USA	0.474	0.022	0.502	0.015
Outside of NYS	0.486	0.019	0.484	0.019
Inside of NYS	0.452	0.030	0.342	0.110
Healthcare Organization	0.462	0.027	0.607	0.002
Government Agency	0.420	0.046	0.548	0.007
Unknown Settings	0.463	0.026	0.325	0.131
Total Usage	0.500	0.015	0.484	0.019

Note: r: Pearson correlation coefficient; dark grey cells: strong correlation (|r| >= 0.5); light grey cells: moderate correlction (0.5 > |r| >= 0.3); white cells: weak or no correlation (|r| < 0.3).

Discussions

The results have confirmed that there were moderate (measured by intensity of use) to strong (measured by number of audience) correlations between the total usage of the Insomnia Screening and Treatment ICST and the sending of CEI email newsletters. The strength of correlation in specific use contexts was not evenly distributed. Instead, certain usage profiles demonstrated strong or moderate correlations, while others only presented weak or no correlations. This is the first time a study has characterized the profiles of use context that do and do not respond to promotions. These identified usage profiles will direct future informatics research to develop effective and targeted approaches to disseminate clinical evidence, focusing on particular users, platforms, and settings.

Regarding the specific use contexts, here are the potential explanations on strength of the correlations:

1. The recommendation section of the ICST system function was the target of the promotional email (clicking the ICST hyperlink in the email would bring a user to this section). Therefore, its use demonstrated a strong correlation with the promotional emails. In contrast, the sample case and user-defined case sections presented only weak or no correlations.

2. The new users had not yet bookmarked the ICST webpage nor downloaded the mobile apps. Therefore, they were more likely to respond to email promotions.

3. Access through web browsers was the default channel when clicking the ICST hyperlink in the promotional email. Therefore, this use context recorded more responses. In contrast, access through native apps and small-screen hand-held devices (hosting the native apps) only showed weak or no correlations.

4. Most of the CEI newsletter subscribers were from the United States. Therefore, these audiences were more likely to respond to email promotions.

5. Most of the CEI program audiences were clinicians and public health professionals. Therefore, the audiences from healthcare organizations and government agencies demonstrated strong correlations.

To plan for effective resource dissemination, the use contexts with strong correlations should be the focus. Regarding the use contexts demonstrating only weak or no correlations, resource usage is likely from a core group of loyal audiences (considering that we still recorded high level of usage in these use contexts) who do not need promotions.

There were two limitations in this study. First, we selected a single ICST as the target resource and only the email newsletter as the dissemination activity for analyses. There were many other resources developed by the CEI online training program, and these resources were promoted through multiple channels. Generalizability of the findings from this study, therefore, needs to be verified in future research. Second, this study focused only on correlations, which could be the first step to determine causality. For the use contexts with strong correlations, it may worth further investigating on the detailed pathways in which the dissemination activities can lead to the increasing use of clinical and educational resources.

Conclusion

Dissemination of clinical evidence through promotional activities is correlated with number of audience and intensity of use of online education resources. Strength of correlation depends on specific use contexts. Strong correlations were found between sending of email newsletters and intensity of resource use by promotion recipients, new users, and through the most convenient access channel associated with the promotion. Selection of approaches for resource dissemination should consider their potentials and limitations in specific use contexts to make them more effective.