Abstract
Introduction
CME providers may be interested in identifying effective marketing strategies to direct users to specific content. The use of online advertisements to recruit participants for clinical trials, public health programs, and Continuing Medical Education (CME) has been shown to be effective in some but not all studies. The purpose of this study was to compare the impact of two marketing strategies in the context of an online CME cultural competence curriculum (www.c-comp.org).
Methods
In an interrupted time-series quasi-experimental design, two marketing strategies were tested: a) wide dissemination to relevant organizations over a period of approximately four months, and b) Internet paid search using Google Ads (five consecutive eight-week periods--control 1, cultural/ CME advertisement, control 2, hypertension/ content advertisement, control 3). Outcome measures were CME credit requests, Web traffic (visits per day, page views, pages viewed per visit), and cost.
Results
Overall, the site was visited 19,156 times and 78,160 pages were viewed. During the wide dissemination phase, the proportion of visits requesting CME credit decreased between the first (5.3%) and second halves (3.3%) of this phase (p= .04). During the Internet paid search phase, the proportion of visits requesting CME credit was highest during the cultural/ CME advertisement period (control 1, 1.4%; cultural/CME ad, 4.3%; control 2, 1.5%; hypertension/content ad, 0.6%; control 3, 0.8%; p<.001). All measures of Web traffic changed during the Internet paid search phase (p<.01); however, changes were independent of the advertisement periods. The incremental cost for the cultural advertisement per CME credit requested was $0.64US.
Discussion
Internet advertisement focusing on cultural competence and CME was associated with about a three-fold increase in requests for CME credit at an incremental cost of under $1; however, Web traffic changes were independent of the advertisement strategy.
Keywords: Marketing, Continuing Medical Education, Internet, Web, Online, Cultural Competence
INTRODUCTION
Physicians are increasingly using Web-based continuing medical education (CME) to meet CME requirements. In 2008, accredited CME providers offered a total of 47,304 hours of instruction1 and data suggest that online CME comprises approximately seven to nine percent of CME utilized.2 Given that some studies suggest that Web-based CME improves knowledge, behaviors, and skills, 3, 4 CME providers may be interested in identifying effective marketing strategies to direct users to specific content.5, 6 However, to serve this function, physicians need to find relevant online CME activities. Effective marketing can be an important strategy for helping physicians locate and access online educational opportunities.5
The wide reach of the Internet makes online advertisement an attractive tool for recruiting physicians to CME websites. However, evidence for the effectiveness of online advertisement in the health care field is inconclusive. When used for recruiting participants for clinical trials7 and public health programs,8 it has been effective in some but not all studies.9 In addition, it can be costly. In a study to improve the recognition and management of intimate partner violence, promoting a free online CME activity with search engine advertisement and e-mail solicitation cost $30 to $80 per physician enrolled;10 while direct mail yielded 44% of the participants, it cost $143 per physician participant. We build on available evidence by focusing on the least expensive strategies and comparing two marketing strategies to recruit to an online CME program.
The present study is grounded on a conceptual framework—the Engel-Kollat-Blackwell (EKB) model of consumer behavior for online activities as modified by Darley et al.11 In this model, the decision making process has five main stages: problem recognition (“What do I need?”), search of alternative options (“Where can I find this information?”), alternative evaluation (“Which one should I select”), purchase (option selected), and outcomes (satisfaction). We focused on the search of alternative options aspect, where internal and external factors may influence the search behavior. In particular, personal characteristics, social influences, specific situations, and the online environment shape the nature of the decisions and the effort in searching for additional information. Importantly, several characteristics of the online environment such as Web quality and satisfaction with the site also influence search behavior. The EKB model helped frame the problem for this study. It addresses the question “What marketing strategies are effective during the search process?” More specifically, the purpose of this study was to compare the impact of two marketing strategies on recruitment to CME, Web traffic, and cost in the context of an online CME cultural competence curriculum (www.c-comp.org). The marketing strategies were 1) wide dissemination and 2) Internet paid searches.
METHODS
Overview, Setting, and Study Design
The Cultural Competence Online for Medical Practice (CCOMP) (www.c-comp.org) Website was designed and implemented after extensive formative evaluation.12 The main objective of the educational site was to teach effective cross-cultural approaches to care for African-American patients with hypertension. The site includes videos with real patient scenarios and case-based modules. Designed for practicing physicians, residents and medical students, the Website is available in the public domain. We used an interrupted time-series quasi-experimental design to explore the impact of two marketing strategies: 1) wide dissemination to relevant organizations; and, 2) Internet paid searches, consisting of consecutive periods of control (no paid searches) and paid searches (FIGURE 1).
Marketing Strategy #1: Wide Dissemination Phase
The wide dissemination phase occurred between June 3, 2008 and October 17, 2008. The purpose of this phase was to determine the impact of a wide dissemination strategy on the outcome variables.13After identifying Websites focusing on cultural competence training, minority research, cardiovascular disease and others, we asked 37 to post the CCOMP Website address with a brief description of its content. We also submitted a request to the American Association of Medical Colleges (AAMC) clearinghouse portal for peer review and posting14 and posted an entry in Wikipedia, for a total of 39 sites. To identify which Websites posted the requested link to the cultural competence Website, we reviewed each of the 39 sites at the end of the wide dissemination phase to verify that the link had been posted.
After the wide dissemination phase, we included a pilot phase to inform the length of time for the Internet paid search phase. The pilot phase consisted of a cultural ad pilot (October 18–November 18, 2008) and a no ad washout period (November 19, 2008– January 11, 2009); the length of time for the washout period was empirically determined and extended past the December holidays. The purpose of the pilot phase was to provide estimates for the number of weeks for each control and Internet paid search advertisement periods; the purpose of the washout period was to avoid any delayed contamination with the subsequent phase.
Marketing Strategy #2: Internet Paid Search Phase
The Internet paid search phase consisted of five consecutive eight-week periods between January and October, 2009; the periods were control #1 (January 12 to March 8), cultural ad period (March 9 to May 3), control #2 (May 4 to June 28), hypertension ad period (June 29 to Aug 23), and control #3 (Aug 24-Oct 18, 2009).
During the control periods, none of the ads were displayed and no other marketing strategies were in place. We did not market the site on any print venue nor make any attempts to remove the link from any of the Websites with previous link postings.
We designed and purchased Internet advertising using Google Ads to display two types of ads. The first one focused on cultural competence, the CME offering, and an authoritative sponsor (National Institutes of Health)(advertisement #1; reported here as cultural/CME ad). The second one focused on content, ie, tips to control hypertension and video (advertisement #2; reported here as hypertension/ content). Initially, a set of 26 tentative phrases were generated by a group with expertise on online CME development, with final phrases selected having face validity for cultural or hypertension issues. The first ad read: “Cultural Competency, Videos with Real Patient Stories, Online CME, NIH-sponsored www.C-Comp.org.” The second ad read: “Tips to Control Hypertension,… in your African American Patient, Videos with Real Patient Stories www.C-Comp.org.” The availability of CME credit was not prominently displayed on the main page; the viewer had to click on a link labeled “Objectives and Credit Information” to get access to that information (CME was only mentioned in the first ad).
Ads appeared on the right side of the screen after any US user typed any of the selected keywords. The keywords for the cultural ad were: cultural competence in health care. The keywords for the hypertension ad were: African Americans, African Americans hypertension, hypertension, African American hypertension treatment, African American with hypertension, hypertension in African American, hypertension in African American men, hypertension in African American women, and hypertension in the African American.
Participants
Participants were Internet users and we had no exclusion criteria. Participants interested in obtaining CME credit registered by providing name, e-mail address, and medical training level. CME credit was provided at no charge and no additional incentives were provided for participation.
Variables and Data Sources
The main dependent variables were recruitment to CME, Web traffic, and costs (TABLE 1). We defined recruitment to CME or conversion rate as the total number of CME credit requests / total number of visits (not unique); this metric is commonly used for online commercial sites.15 We defined traffic as a) frequency (total number of visits per day), b) volume (the total number of page views), and c) engagement with the site (total number of pages viewed per visit).16, 17 We calculated the incremental cost per additional CME credit obtained using a cost-effectiveness framework18 during the Internet paid search phase, incremental cost-effectiveness ratio = [Cost during advertisement periods– cost during control periods]/[# CME credits during advertisement periods - # CME credits during control periods].
TABLE 1.
Outcome | Measure |
---|---|
Recruitment to CME | Total number of CME credit requests / total number of visits |
Web traffic | Frequency: total number of visits per day |
Volume: total number of page views | |
Engagement with the site: total number of pages viewed per visit | |
Costs. | Cost of displaying the ad |
We used Urchin Software, Version 5.0 to analyze Web server log files,19 which compile data that are not linked to a specific user. We used data provided by Google Ads to determine the number of times an ad was displayed (impressions) or selected (clicks) and the average position of the ad. The average position is attributed to each keyword, “1” is the highest position on the first page of search results (keywords with an average position of 1–8 generally trigger ads on the first page of search results).
Statistical Analysis
To assess the changes in the dependent outcome variables during the wide dissemination phase, we divided this phase in two halves. We then used the Kruskall-Wallis test to compare Web traffic (number of visits, pages viewed, pages per visit per day) and the Chi square test to compare CME credit between these two halves.
We also compared the outcomes between periods in the Internet paid search phase using the Kruskall-Wallis test, the Chi square test, and run charts including control limits20 as appropriate. Using data from the pilot phase, we assumed that the mean number of pages viewed would increase from 120 (SD 120) to 190 (SD 120); at an alpha of .05, power 0.80, the number of weeks required for each period was seven weeks (47 days); we elected eight weeks. We used STATA SE, Version 10.0 (College Station, Texas, USA) for statistical analysis and used a p value of .05 for statistical significance.
The Institutional Review Board at the University of Alabama at Birmingham approved the study.
RESULTS
During the study period, the site was visited 19,156 times, 78,160 pages were viewed, and the median number of pages per visit per day was 3.7 (interquartile range, Q1–Q3, 2.2–6.8); CME credit was requested in 1.8% of the visits (339/19,156). The run charts shown in FIGURE 2 display the measures over time for frequency (# visits/day) and engagement (# of pages per visit). During the entire study period, the number of visits per day increased (Panel A) and the number and variability of engagement decreased (Panel B).
Marketing Strategy #1: Wide Dissemination Phase
The Website URL was posted on 41% of the sites we requested (16/39), including MedEdPortal21 and Wikipedia.22 The median number of visits increased between the first and second halves of the wide dissemination phase (median 5.5, Q1– Q3: 3–11; 23, Q1– Q3: 10–30; respectively, Kruskall-Wallis p<.001), suggesting that the marketing strategy had a positive impact on Web traffic (frequency). The median number of pages viewed also increased between the first and second halves of the wide dissemination phase (median 28, Q1– Q3: 9–96; 115, Q1–Q3: 58–181; respectively; Kruskall-Wallis p<.001), also suggesting that the marketing strategy had a positive impact on Web traffic (volume). However, the median number of pages per visit per day remained unchanged between the first and second halves of the wide dissemination phase (median 5.9, Q1– Q3: 1.2–11.6; 5.5, Q1– Q3: 3.4–9.8; respectively, Kruskall-Wallis p=.8), suggesting that the marketing strategy did not have an impact on engagement with the Website.
The proportion of visits resulting in requests for CME credit decreased between the first and second halves of the wide dissemination phase, from 5.3% (27/508) to 3.3% (47/1,417); respectively (Chi square p= .04), suggesting that the wide dissemination marketing strategy did not increase recruitment to CME.
Marketing Strategy #2: Internet Paid Search Phase
During the Internet paid search phase of 40 weeks, the site was visited 13,976 times (daily median 44; Q1– Q3, 28 – 70), 50,209 pages were viewed (daily median 148; Q1–Q3, 100–232), and the median number of pages per visit per day was 3.2 (Q1–Q3, 2.0–4.9).
As shown in TABLE 2, the proportion of visits resulting in requests for CME credit was highest during the cultural/CME ad period (control #1 1.4%, cultural/CME ad 4.3%, control #2 1.5%, hypertension/content ad 0.6%, and control #3 0.8%, Chi square p<.001), suggesting that the ad focusing on cultural competence and CME had the highest recruitment to CME.
TABLE 2.
Web Activity | Control 1 (Jan 12 – Mar 8) | Cultural/CME ad (Mar 9 – May 3) | Control 2 (May 4 – Jun 28) | Hypertension/ Content ad (Jun 29 – Aug 23) | Control 3 (May 4 – Jun 28) | p- value* |
---|---|---|---|---|---|---|
Recruitment to CME | ||||||
Total # CME credit/ total # of visits | 1.4% (25/1,838) | 4.3% (81/1,884) | 1.5% (51/3,387) | 0.6% (18/3,245) | 0.8% (28/3,622) | < 0.001 |
Web Traffic | ||||||
Frequency, # visits per day | 27 [19–38] | 29 [21–44] | 65 [42–74] | 50 [34–77] | 69 [47–83] | < 0.001 |
Volume, # pages viewed per day | 118 [87–210] | 175 [88–275] | 188 [131–311] | 119 [91–211] | 167 [103–250] | 0.004 |
Engagement, # pages per visit per day | 4.2 [2.7–7.2] | 5.1 [3.1–8.3] | 3.3 [2.2–4.5] | 2.3 [1.8–3.5] | 2.3 [1.9–3.6] | < 0.001 |
Values are medians [interquartile range, 25th–75th percentile, Q1–Q3] or % (counts), as noted.
Chi square test for Recruitment and Kruskall-Wallis test for Traffic.
The Web traffic measure, number of visits per day, was lowest during the first control period (median 27) and the cultural/ CME ad period (median 29), and highest during the second and third control periods (median 65 and 69 respectively) (Kruskall-Wallis p<.001, TABLE 2). The number of pages viewed per day were lowest during the first control period (median 118) and the hypertension ad period (median 119) and highest during the cultural/CME ad period (median 175) and the second control period (median 188)(Kruskall-Wallis p<.001, TABLE 2). The engagement measure, number of pages per visit per day, was lowest during the hypertension ad and third control periods (both median 2.3) and highest during the cultural/CME ad period (median 5.1)(Kruskall-Wallis p<.001, TABLE 2). In summary, we did not observe a specific and consistent increase on any measures of Web traffic during the cultural/CME or hypertension advertisement periods. The total cost of the display ads was $125. The cultural/CME ad was shown 1,460 times at an average position of 1.3; the ad was clicked 33 times (2.3%) at a total cost of $36 ($1.08 per click). The hypertension/content ad was shown 48,009 times at an average position of 5.9; the ad was clicked 89 times (0.19%) at a total cost of $89 ($1.01 per click). As compared with the first two control periods, the incremental cost for both ads per CME credit obtained was $5.43 ([$125-$0]/[99 CME credit requests-76 requests]); however; as compared with the first control period, the incremental cost for the cultural/CME ad per CME credit obtained was $0.64 ([$36-$0]/[81 CME credit requests-25 CME credit requests]).
DISCUSSION
In promoting an open access CME online cultural competence curriculum, during the wide dissemination marketing strategy phase, Web use (number of visits) increased while engagement (number of pages viewed per visit) and recruitment to CME decreased However, during the Internet paid search marketing strategy phase, Internet advertisement focusing on cultural competence, CME offering, and an authoritative sponsor (National Institutes of Health) resulted in the highest proportion of users requesting CME credit. Our paid search advertisement did not have an advertisement banner, which is one of the most expensive online advertisements. The incremental cost for the cultural/ CME ad per additional CME credit requested of under $1 seemed reasonable.
The increase in the number of visits could be explained by higher rankings of the Website over time, independent of the advertisement used. As the site was visited more often over time, and as other Websites posted a link, the proprietary algorithm used by Google probably increased the priority rankings during searches. We suspect that the decrease in engagement could be explained by market saturation. Additionally, the greater number of keywords in the hypertension/content ad may explain the difference on the number of times the ad was shown during this period (48,009 times) as compared to the cultural/ CME ad (1,460 times).
Tracking Web Traffic- Web Analytics
The Web produces vast amounts of information and methods for tracking utilization, also called Web analytics, are rapidly evolving.15, 23, 24 Web traffic statistics on frequency and volume are routinely reported;16, 17 however, how these statistics are calculated varies widely, making comparisons between studies challenging.8 Also, standard statistical techniques to track information over time may be ill-suited for analysis and more complex techniques may be more difficult to implement. Hence, the run charts we used offer another approach for CME providers to monitor process variation (such as Web traffic) over time and monitor the effects of interventions.20
Limitations
Causation cannot be concluded from this study as we used a quasi-experimental design. Because the Web site did not require participants to register, we do not know if factors such as specialty, age, gender, or frequency of computer use influenced our findings. We were unable to quantify any potential impact of confounders. Finally, the greater number of keywords in the hypertension/content ad may explain the 33-fold increase in the number of times the ad was shown during this period as compared to the cultural/ CME ad (48,009 vs.1,460 times); however, one has to keep in mind that more people may be interested in blood pressure content as over 70 million people in the US suffer from hypertension.
CONCLUSIONS
For an online cultural competence curriculum, an Internet advertisement focusing on cultural competence and CME was associated with an approximately three-fold increase in recruitment to obtain CME credit at an incremental cost of under $1. The number of visits and level of engagement seemed to be independent of the advertisement strategy.
Future studies could further explore the strategies we used to increase Web traffic in online CME activities; for example, examining whether cycles of changes in the advertisement wording or change in the number of keywords- rather than content as was done in this study - has any effect on Web traffic.
Lessons for Practice
-
◯
In an open access CME online cultural competence curriculum, Web use increased over time while engagement decreased over time.
-
◯
Internet advertisement focusing on cultural competence and CME resulted in the highest proportion of users requesting CME credit.
-
◯
The incremental cost for the cultural/ CME ad per additional CME credit obtained was under $1.
Acknowledgments
We thank Michael Schoen PhD and Trudi Horton PhD, for critical revisions of the manuscript.
Funding and Support: This study was funded by a grant from the National Heart, Lung, and Blood Institute (K07 HL081373-01) to Dr. Estrada.
Footnotes
Ethical approval: This study was approved by the institutional review boards of the University of Alabama at Birmingham.
DISCLOSURES The authors report no conflict of interest.
Other Disclosures: None
Disclaimer: The opinions expressed in this article are those of the authors alone and do not reflect the views of the Department of Veterans Affairs.
REFERENCES
- 1.ACCME [Accessed November 13, 2010];Annual Report Data 2008. 2008 http://www.accme.org/dir_docs/doc_upload/1f8dc476-246a-4e8e-91d3-d24ff2f5bfec_uploaddocument.pdf.
- 2.Harris JM, Jr., Sklar BM, Amend RW, Novalis-Marine C. The growth, characteristics, and future of online CME. J Contin Educ Health Prof. 2010;30(1):3–10. doi: 10.1002/chp.20050. [DOI] [PubMed] [Google Scholar]
- 3.Weston CM, Sciamanna CN, Nash DB. Evaluating online continuing medical education seminars: evidence for improving clinical practices. Am J Med Qual. 2008;23(6):475–483. doi: 10.1177/1062860608325266. [DOI] [PubMed] [Google Scholar]
- 4.Mansouri M, Lockyer J. A meta-analysis of continuing medical education effectiveness. J Contin Educ Health Prof. 2007 Winter;27(1):6–15. doi: 10.1002/chp.88. [DOI] [PubMed] [Google Scholar]
- 5.Vanek EP, Gard PR, Melia M, Michener W. Marketing issues in continuing medical education. J Contin Educ Health Prof. 1988;8(4):301–307. doi: 10.1002/chp.4750080408. [DOI] [PubMed] [Google Scholar]
- 6.Kennedy DW, Queeney DS. Marketing Continuing Medical Education Programs. J Contin Educ Health Prof. 1991;11(3):205–214. [Google Scholar]
- 7.Houston TK, Ford DE. A tailored Internet-delivered intervention for smoking cessation designed to encourage social support and treatment seeking: usability testing and user tracing. Inform Health Soc Care. 2008;33(1):5–19. doi: 10.1080/14639230701842240. [DOI] [PubMed] [Google Scholar]
- 8.Graham AL, Milner P, Saul JE, Pfaff L. Online advertising as a public health and recruitment tool: comparison of different media campaigns to increase demand for smoking cessation interventions. J Med Internet Res. 2008;10(5):e50. doi: 10.2196/jmir.1001. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Koo M, Skinner H. Challenges of Internet Recruitment: A Case Study with Disappointing Results. J Med Internet Res. 2005 Mar 19;7(1):e6. doi: 10.2196/jmir.7.1.e6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Harris JM, Jr., Novalis-Marine C, Amend RW, Surprenant ZJ. Promoting free online CME for intimate partner violence: what works at what cost? J Contin Educ Health Prof. 2009;29(3):135–141. doi: 10.1002/chp.20025. [DOI] [PubMed] [Google Scholar]
- 11.Darley WK, Blankson C, Luethge DJ. Toward an integrated framework for online consumer behavior and decision making process: A review. Psychol Marketing. 2010;27(2):94–116. [Google Scholar]
- 12.Crenshaw K, Shewchuk RM, Qu H, et al. What Should We Include in a Cultural Competence Curriculum? An Emerging Formative Evaluation Process to Foster Curriculum Development. Acad Med. 2011 doi: 10.1097/ACM.0b013e3182087314. in press. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Kerin RA, Hartley SW, Rudelius W. Marketing. 10th ed. McGraw-Hill/Irwin; New York, NY: 2011. Chapter 22. Pulling it all together: The strategic planning marketing process. [Google Scholar]
- 14.AAMC. Association of American Medical Colleges [Accessed November 13, 2010];MedEdPortal. Available at http://services.aamc.org/30/mededportal/servlet/segment/mededportal/information/.
- 15.Creese G, Burby J. Web Analytics Key Metrics and KPIs (Version 1.0) Web Analytics Association; Washington, DC: 2005. [Google Scholar]
- 16.Houston TK, Funkhouser E, Allison JJ, Levine DA, Williams OD, Kiefe CI. Multiple measures of provider participation in Internet delivered interventions. Stud Health Technol Inform. 2007;129(Pt 2):1401–1405. [PubMed] [Google Scholar]
- 17.Danaher BG, Boles SM, Akers L, Gordon JS, Severson HH. Defining participant exposure measures in Web-based health behavior change programs. J Med Internet Res. 2006;8(3):e15. doi: 10.2196/jmir.8.3.e15. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Hunink MGM, Glasziou PP, Siegel JE, Weeks JC. Decision Making in Health and Medicine: Integrating Evidence and Values. Cambridge University Press; Cambridge: 2001. [Google Scholar]
- 19.Urchin Software from Google [Accessed November 13, 2010];Google Analytics. http://www.google.com/urchin/index.html.
- 20.Thor J, Lundberg J, Ask J, et al. Application of statistical process control in healthcare improvement: systematic review. Qual Saf Health Care. 2007 Oct;16(5):387–399. doi: 10.1136/qshc.2006.022194. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Estrada CA. Cultural Competencies Online for Medical Practice (CCOMP): A Clinician's Guide to Reduce Cardiovascular Disparities. [Accessed November 13, 2010];MedEdPORTAL. 2009 Available from: http://services.aamc.org/30/mededportal/servlet/s/segment/mededportal/?subid=1765.
- 22.Anonymous [Accessed November 13, 2010];Cultural Competency Training. Available at http://en.wikipedia.org/wiki/Cultural_Competency_Training. [Entry Aug 21, 2008]
- 23.Upstill C, Addis MJ, Choi F, Taylor SJ, Watkins ER. S4: Data mining and fusion. University of Southampton. School of Electronics and Computer Science; United Kingdom: 2006. [Accessed November 13, 2010]. http://eprints.ecs.soton.ac.uk/12771/. [Google Scholar]
- 24.Wood FB, Benson D, LaCroix EM, Siegel ER, Fariss S. Use of Internet audience measurement data to gauge market share for online health information services. J Med Internet Res. 2005 Jul 1;7(3):e31. doi: 10.2196/jmir.7.3.e31. [DOI] [PMC free article] [PubMed] [Google Scholar]