FOAM Impact: The Influence of Open‐access Medical Education on Practice Uptake

Abstract

Objectives

The FOAM Impact study sought to examine baseline rates of intravenous (IV) lidocaine usage for the treatment of renal colic and to compare rates of use between FOAM utilizers and nonutilizers. We sought to measure the effect of FOAM resources on clinical practice by timing the release of FOAM content with publication of the LIDOKET trial.

Methods

A cross‐sectional before‐and‐after survey was conducted and disseminated on two social media platforms. The 13‐question, anonymous survey was posted for 1 week prior following the release of the LIDOKET study. Descriptive statistics, Mann‐Whitney t‐test, and chi‐square test statistics were used to describe survey respondent characteristics and Likert responses.

Results

There were a total of 472 survey respondents. A total of 321 physicians (75.7% of total respondents) provided pre‐ and postpublication survey answers. There was no significant change in the use of analgesics before and after publication of LIDOKET and concurrent REBEL EM blog post. A total of 197 (42%) survey respondents reported using lidocaine for renal colic, of which 60 respondents (13%) reported frequent or occasional use. The mean difference in perceived efficacy of lidocaine before and after publication was −0.30 (95% confidence interval [CI] = −0.80 to 0.19, p = 0.15). Being a FOAM user was not associated with changes in mean difference in perception of lidocaine efficacy (F = 0.127, p = 0.72); however, there was a significant difference in perception of lidocaine’s efficacy following LIDOKET and REBEL EM publication (F = 4.718, p = 0.03).

Conclusions

Using an online survey‐based technique, no appreciable impact of FOAM resources was immediately apparent; however, engagement with FOAM was associated with a change in perception of IV lidocaine’s efficacy. To our knowledge, this is the first study of its kind to evaluate the impact of FOAM on clinical practice. The unique method of coordinating FOAM distribution with traditional medical publication may provide future opportunities for measuring the impact of asynchronous medical education resources on medical practice.

In today’s age of advancing technology, it has become the norm to rely heavily on Internet resources to aid medical practice and continuing medical education. Online educational resources, social media, and asynchronous education increasingly dominate innovation and practice evolution for emergency physicians. With a plethora of options available, it is the physician’s responsibility to utilize reliable and effective resources. The “free open‐access medical education” (FOAM) movement has harnessed the power of global connectivity to drive narrowing of the knowledge translation window. However, critics have urged caution in using online resources and warned that peer review or quality control is lacking.1, 2, 3

REBEL EM is an online educational resource widely used throughout the FOAM community, garnering 1,000 to 30,000 views/engagements per educational post and satisfying all quality indicators that have been previously identified as markers of reliability and usability for digital scholarship.4 “EM Docs” is an online Facebook community of nearly 20,000 members restricted to practicing and trainee emergency physicians. While discussion and debate are nearly instantaneous with experience and perspective from all over the world, the open‐access nature of this resource lends it to satisfy few, if any, objective criteria of quality.

Generally, it is perceived that physicians who engage in social media educational resources have a higher baseline rate of uptake of “newer” or “cutting edge” clinical practices. While several publications have investigated reliability and quality of online resources, to our knowledge, no study to date has attempted to measure and compare the impact of social media resources on clinical practice.4, 5, 6, 7

The primary objective of the FOAM Impact study was to examine the baseline rates of use of intravenous (IV) lidocaine in the treatment of renal colic. Given the fact that this analgesic technique is rather novel and not yet standard of care, the authors suspected that this practice would be relatively unique to FOAM engagers and interesting to study. In addition to practice patterns, the authors sought to compare differences in practice patterns between REBEL EM and emDOCs users following the publication of the LIDOKET trial and its concurrent REBEL EM blog post.8

Using an online survey‐based technique, we sought to assess the impact of two different social media resources—REBEL EM and emDOCs—on clinical practice. Given advanced knowledge of the LIDOKET trial describing the utility of lidocaine in the treatment of renal colic, we coordinated release of a peer‐reviewed FOAM blog post simultaneous with the publication of the LIDOKET trial in the American Journal of Emergency Medicine (January 2019). In an effort to gauge REBEL EM’s influence on practice, we measured clinician self‐reported practices for the use of lidocaine in the treatment of renal colic both before and after the publication of the social media post and study, respectively, via an online survey.

METHODS

Study Design and Population

A cross‐sectional, internet‐based, before‐and‐after survey study was conducted from February 1, 2019, to March 1, 2019, which was approved by the Crozer‐Keystone Institutional Review Board. The survey was disseminated on two social media platforms (Facebook and Twitter). The emDOCs Facebook page and REBEL EM website were chosen as the study populations. We hypothesized that health care providers on the emDOCs Facebook page were less likely to utilize FOAM for staying up to date with advancements in the literature compared to health care providers who subscribe to REBEL EM updates. All health care providers of both groups were eligible for inclusion in this study. Participants were excluded, however, if they completed only one of the two surveys.

Survey Design

A 13‐question, anonymous survey was created and underwent multiple iterations of revisions. Survey questions included demographics (health care provider type, gender, number of years in practice) and utilization of and frequency of use of the aforementioned social media platforms. Three groups of questions focused on different modes of analgesia for renal colic (IV lidocaine, IV ketorolac, and opioids), namely, how often health care providers use each medication and the perceived effectiveness of each medication. Finally, for the group of questions on the use of IV lidocaine an additional question assessed where health care providers learned about the use of this medication for the treatment of renal colic (academic texts, formal training, FOAM, social media [Facebook or Twitter], or “I don’t use IV lidocaine”).

Survey Dissemination

The initial survey was posted on both platforms for 1 week prior to the LIDOKET study release. Following the release of the LIDOKET study and accompanying REBEL EM blog post, the survey was again posted for 1 week on both platforms.

Data Analysis

As an anonymous pretest/posttest survey without use of unique identifier codes of a large population, our analysis plan assumed independence of observations and errors. Descriptive statistics, Mann‐Whitney t‐test, and the chi‐square test statistic were used to describe survey respondent characteristics and Likert responses. The mean difference for four comparisons of perceived efficacy of ketorolac and lidocaine, pre‐ and postpublication, was assessed using the Mann‐Whitney t‐test with bootstrapped 95% confidence intervals (CIs), and depicted with a Cumming estimation plot using the “dabestr” package in the programming language R. The null hypothesis was explored with a factorial analysis of variance assessing differences in mean perception of lidocaine efficacy in renal colic while accounting for two factors, namely, when the survey was completed (before/after LIDOKET publication) and FOAM use. The variable FOAM was collapsed to a binary variable (yes/no) for this analysis. Given that there was no interaction effect between the two factors, the additive model was used for analysis. A sensitivity analysis was conducted to explore the impact of FOAM use as an ordinal variable. An alpha of 5% was used for significance. All statistical analyses were performed using R (GUI 1.70 El Capitan build).

RESULTS

Survey Respondents

There were a total of 472 total survey respondents. A total of 383 respondents provided prepublication and postpublication survey answers. A total of 321 physicians, or 75.7% of total respondents, provided pre‐ and postpublication survey answers. The remaining respondents included a combination of EMS providers, nurses, and nurse practitioners and physician assistants.

Frequency of Use and Perceived Efficacy of Different Analgesic Medications

Respondents were asked how often they incorporate IV ketorolac, lidocaine, and opioids into their analgesic pathway for patients presenting with renal colic, respectively. Results are depicted in Figure 1. There was no significant change in the use of the three analgesics before and after publication of LIDOKET and the concurrent REBEL EM blog post. A total of 346 respondents (73%) reported frequent or occasional use of opioids, with 60 (13%) reporting similar utilization of IV lidocaine and 378 (80%) frequent or occasional use of ketorolac. A total of 197 survey respondents reported using lidocaine (42%) for renal colic, of which 40% reported using it frequently or occasionally. Of the 197 who reported use of lidocaine for renal colic, 138 (70%) reported that they primarily learned about its use through FOAM or social media (Facebook or Twitter; Figure 2).

Figure 1.

Likert scale of responses for use of different analgesics for renal colic.

Figure 2.

The mean difference for two comparisons are shown in the above Cumming estimation plot. The raw data are plotted on the upper axes; each mean difference is plotted on the lower axes as a bootstrap sampling distribution. Mean differences are depicted as dots; 95% CIs are indicated by the ends of the vertical error bars. A total of 5,000 bootstrap samples were taken; the CI is bias‐corrected and accelerated. The p‐value(s) reported are the likelihood(s) of observing the effect size(s), if the null hypothesis of zero difference is true. (A) Unpaired mean difference between ketorolac pre and ketorolac post is –0.056 (95%CI = –0.40 to 0.26). The two‐sided p‐value of the Mann‐Whitney test is 0.32. The unpaired mean difference between lidocaine pre and lidocaine post is –0.301 (95% CI = –0.80 to 0.19). The two‐sided p‐value of the Mann‐Whitney test is 0.15. (B) Unpaired mean difference between ketorolac pre and lidocaine pre is –2.89 (95% CI = –3.2 to –2.6), while the unpaired mean difference between ketorolac post and lidocaine post is –3.14 (95% CI = –3.7 to –2.6). The two‐sided p‐value for both Mann‐Whitney tests is <0.001.

FOAM and Use of Lidocaine

Being a FOAM user was not associated with changes in mean difference in perception of lidocaine efficacy of renal colic (F = 0.127, p = 0.72). However, when the survey was taken—i.e., before or after the publication—was associated with a significant difference in perception of lidocaine in renal colic (F = 4.718, p = 0.03). The sensitivity analysis with FOAM as an ordinal variable did not change the significance of the model.

DISCUSSION

Identifying as a FOAM user did not influence the perception of the efficacy of lidocaine for treatment of renal colic more so than those of nonusers. There was no significant difference in perception of efficacy prior to LIDOKET and blog publication between platforms. There was a significant difference in the perception of efficacy of lidocaine for renal colic following publication of LIDOKET and the REBEL EM blog post, but no difference identified between users who identified as utilizing one platform exclusively. Both emDOCs Facebook users and FOAM users were equally impacted by the LIDOKET trial results.

Respondents, regardless of identifying as FOAM users or not, showed no change in the use of ketorolac, lidocaine, or opioids in treatment of renal colic pre‐ and postpublication. There was also no significant change in the perceived efficacy of ketorolac versus lidocaine pre‐ and postpublication, thus indicating that the LIDOKET trial results and REBEL EM blog did not significantly alter clinical practice.

The LIDOKET trial results did not significantly change clinical practice; however, it was associated with a change in perception of lidocaine’s efficacy. There was no significant difference between FOAM users and nonusers in this change of perception.

LIMITATIONS

Limitations to this study include the sample population. There was no direct way to assess the changes in practice of individuals given the distribution of the survey. The study was also limited by the self‐report of perception of frequency of the FOAM use, as well as self‐identification as a physician, a method susceptible to corruption.

CONCLUSION

Physicians have a seemingly unlimited number of resources at their disposal. This was supported by the finding that 70% of our respondents reported using FOAM or a form of social media to access information that affects their clinical practice. No appreciable impact of FOAM resources was immediately apparent among our study population for this particular practice pattern; however, continuing medical education requires informed navigation of online resources and an understanding of how these resources impact medical decision making. Admittedly the practice of using intravenous lidocaine is not yet standard of care and has potential life‐threatening consequences if not used properly. Given the risks associated with this specific practice pattern surveyed, it is not surprising that most practitioners were still reluctant to change their clinical practice, although it did increase their awareness of its efficacy. In the future, it would be interesting to study the effect of social media/FOAM on a variety of practice patterns, including those that are less controversial and risky, to see the influence that these platforms may have. This is the first study of its kind to evaluate the impact of FOAM on clinical practice. The unique method of coordinating FOAM distribution with traditional medical publications may provide future opportunities for measuring the impact of asynchronous medical education resources on medical practice.

AEM Education and Training 2020;4:239–243