Evidence-based Narratives to Improve Recall of Opioid Prescribing Guidelines: A Randomized Experiment

Abstract

Objectives

Physicians adopt evidence-based guidelines with variable consistency. Narratives, or stories, offer a novel dissemination strategy for clinical recommendations. The study objective was to compare whether evidence-based narrative versus traditional summary improved recall of opioid prescribing guidelines from the American College of Emergency Physicians (ACEP).

Methods

This was a prospective, randomized controlled experiment to compare whether narrative versus summary promoted short-term recall of six themes contained in the ACEP opioid guideline. The experiment was modeled after the free-recall test, an established technique in studies of memory. At a regional conference, emergency physicians were randomized to read either a summary of the guideline (control) or a narrative (intervention). The fictional narrative was constructed to match the summary in content and length. One hour after reading the text, participants listed all content that they could recall. Two reviewers independently scored the responses to assess recall of the six themes. The primary outcome was the total number of themes recalled per participant. Secondary outcomes included the proportion of responses in each study arm that recalled individual themes and the proportion of responses in each arm that contained falsely recalled or extraneous information.

Results

Ninety-five physicians were randomized. Eighty-two physicians completed the experiment, for a response rate of 86%. The mean of the total number of themes recalled per participant was 3.1 in the narrative arm versus 2.0 in the summary arm (difference = 1.1, 95% confidence interval [CI] = 0.6 to 1.7). For three themes, the proportion of responses that recalled the theme was significantly greater in the narrative arm compared to the summary arm, with the differences ranging from 20% to 51%. For one theme, recall was significantly greater in the summary arm. For two themes, there was no statistically significant difference in recall between the arms. In the summary arm, 54% of responses were found to contain falsely recalled or extraneous information versus 21% of responses in the narrative arm (difference = 33%, 95% CI = 14% to 53%).

Conclusions

Physicians exposed to a narrative about opioid guidelines were more likely to recall guideline content at 1 hour than those exposed to a summary of the guidelines. Future studies should examine whether the incorporation of narratives in dissemination campaigns improves guideline adoption and changes clinical practice.

Physicians do not readily adopt guidelines into their clinical practice.^1-4 Strategies to implement guidelines have been limited by social, cognitive, financial, and organizational barriers.^1,5,6 Few guideline developers have tested alternative methods to disseminate their recommendations or evaluate their adoption.^2,7,8 Most dissemination efforts use summary or probabilistic methods that synthesize data and cite statistical evidence alone.^5,8

Evidence-based narratives offer a novel method of communicating clinical recommendations.^9-11 Narratives are defined as “coherent stories with an identifiable beginning, middle, and end that provide information about scene, characters, and conflict; raise unanswered questions or unresolved conflict; and provide resolution.”¹²

Narratives are recognized as persuasive tools to change health behavior among patients.^12,13 Narratives also help patients make complex medical decisions.¹⁴ Among health care providers, narratives have been successfully incorporated into educational interventions.^15-17 To our knowledge, however, evidence-based narratives have not been compared to traditional presentations of guidelines in the dissemination of clinical evidence to health care providers.

Opioid overdose was the second leading cause of unintentional injury death in the United States in 2007, prompting the Centers for Disease Control and Prevention to label it a national epidemic.¹⁸ Nearly half of patients presenting to the emergency department (ED) have painful conditions, yet emergency patients are also considered at high risk for opioid abuse.^19,20 To deliver patient-centered care, emergency physicians (EPs) are often challenged to identify the best pain regimens for their patients.²¹ In October 2012, the American College of Emergency Physicians (ACEP) published an evidence-based clinical policy regarding the management of pain in the ED.²¹

The goal of this study was to determine whether exposure to narrative, compared to standard summary, enhanced recall of the ACEP opioid guidelines among EPs. Recall represents only an initial step toward change in clinical practice, yet lack of awareness and familiarity with guidelines are known to be major barriers to adoption.^2,22 Before narratives can be deployed as tools to drive dissemination and implementation of guidelines, the efficacy of evidence-based narratives must first be evaluated.

METHODS

Study Design

We conducted a prospective, randomized controlled experiment to compare the efficacy of narrative versus summary text in promoting recall of six themes contained in guideline recommendations regarding opioid prescribing to patients discharged from the ED. The institutional review board of the University of Pennsylvania approved the study and designated it exempt from review. All participants provided verbal informed consent.

Study Setting and Population

Participants consisted of all resident, fellow, and attending EPs who attended a single-day educational conference (October 2012). The participants represented six Accreditation Council for Graduate Medical Education (ACGME)-accredited residency training programs in the Philadelphia region. Resident physicians were required by their programs to attend the conference. Nonphysician conference attendees were excluded from participation.

Guideline and Narrative Development

We selected two excerpts from a summary of the guideline in its published format. The original guideline was developed by ACEP using an expert consensus panel to review and analyze the literature.²¹ The selected excerpts were related to the use of prescription drug monitoring programs (PDMPs) and the treatment of acute low back pain. The excerpt contained 13 sentences (273 words) and had a Flesch grade level score of 12.0. In this study, the summary excerpt served as the control.

We used content analysis to identify the core ideas in the summary.²³ Two investigators independently identified six distinct themes, each a central message of the guideline. The themes were refined with the assistance of a third investigator, and final agreement was achieved through consensus. The themes were: 1) use PDMPs; 2) attempt to use nonopioid therapies; 3) consider the risks of opioid abuse and misuse; 4) how to treat acute low back pain; 5) when prescribing opioids, use low doses and short durations; and 6) ACEP. The last theme references the author of the guideline. We also identified contextual, or background, information. Each sentence or phrase in the summary was assigned and labeled as either one of the six themes or contextual information.

From these themes, we constructed a fictional narrative that matched the summary in length and structure, with 13 sentences (273 words). Each sentence in the narrative was assigned to either one of the themes or contextual information. To avoid overemphasis of any individual theme in the narrative, the word count of each theme differed between the summary and narrative by fewer than three words. The narrative had a Flesch grade level score of 8.4. The full summary (Data Supplement S1, available as supporting information in the online version of this paper) and narrative (Data Supplement S2, available as supporting information in the online version of this paper) are available online.

Study Protocol

The basis for this protocol was the free-recall test, an established technique in the study of memory.^24,25 Free recall is defined as the ability to remember information without cues. When entering the conference, the physician attendees of the conference underwent simple randomization to receive either the summary (control) or the narrative (intervention). While seated, all participants were instructed to read the intervention or control passage individually and not to discuss the passage with others. The participants were then instructed to seal the passage in an envelope. The participants then listened to a lecture, lasting 1 hour, on an unrelated topic.

Following the lecture, all participants were instructed to list, by writing on a study form, any information that they could recall from the material without reviewing it again. Participants were instructed to not communicate with their colleagues until after completion of the study. The participants provided information regarding sex, age, level of training, training program, and self-reported prior knowledge of PDMPs. The study instrument is included in the Data Supplement S3 (available as supporting information in the online version of this paper).

The study outcomes involved recall of the six themes contained in the opioid guideline. The primary outcome was the total number of themes recalled per participant. To determine differences in the recall of individual themes between study arms, a secondary outcome was the proportion of participants who recalled each theme. Additional outcomes included the word count of written responses and the presence of falsely recalled or extraneous information in the responses. Falsely recalled or extraneous information was defined as statements that were either entirely different from or not present in the summary or narrative. The scoring criteria are included in the Data Supplement S4 (available as supporting information in the online version of this paper).

The responses were independently scored by two reviewers (ZM, AK). Using strict criteria established prior to evaluation, the presence or absence of each of the six themes was recorded as a binary score of 0 or 1 for each written response. Interrater reliability was calculated using the kappa statistic. Reviewers also recorded the presence of falsely recalled information. A third reviewer who was blind to the study hypothesis adjudicated disagreements, and final agreement was achieved through three-way consensus.

Data Analysis

We hypothesized that overall recall, measured as the mean of the total number of themes recalled, was greater in the narrative arm than the summary arm by at least one theme. Estimation of sample size for the primary outcome was based on a standard two-tailed test with an alpha level of 5% and a power of 80%. A sample size of 37 participants per study arm was required to detect a difference of one theme between the arms in the mean of the total number of themes recalled. To evaluate the total number of themes recalled per participant, we used Student’s t-test.

Results are presented as the mean of the total number of themes recalled per participant with 95% confidence intervals (CIs). To determine differences between study arms with regard to the proportion of participants who recalled each individual theme and the proportion of responses containing falsely recalled information, we used Fisher’s exact test. Results are presented as differences in the proportions between study arms with 95% CIs. To evaluate differences in response word count, Wilcoxon rank sum test was used. Results are presented as median word count with interquartile range (IQR).

We adjusted the secondary outcome of the proportion of participants who recalled each theme for possible confounding participant characteristics (sex, level of training, and prior knowledge of PDMPs). We used a generalized linear model with a log link, Gaussian error, and robust estimates of the standard errors of the model coefficients.²⁶ Results are presented as adjusted relative risks with 95% CIs. All analyses were performed using SAS statistical software (version 9.3, SAS Institute, Cary, NC); a p value of less than 0.05 was considered statistically significant.

RESULTS

The flow of participants through the study is shown in Figure 1. At the conference, 95 potential participants underwent randomization. From this group, 82 completed the study, for a response rate of 86%. There were 44 participants in the summary arm and 38 in the narrative arm. The two groups were similar with regard to baseline characteristics (Table 1).

Figure 1.

Flow diagram for randomized controlled study.

Table 1.

Characteristics of Study Participants

Characteristic	Summary Arm (n = 44), n (%)	Narrative Arm (n = 38), n (%)	p-value
Sex			0.64
Male	27 (61)	21 (55)
Female	15 (39)	16 (45)
Level of training			0.81
First year resident	14 (32)	8 (21)
Second year resident	7 (16)	7 (18)
Third year resident	9 (20)	10 (26)
Fourth year resident	9 (20)	7 (18)
Attending or fellow	5 (11)	6 (16)
Median (IQR) age (yr)	29 (28–31)	29 (28–31)	0.67
Prior knowledge of PDMPs (self-reported)			0.67
No knowledge	10 (23)	7 (18)
Some knowledge	23 (52)	18 (47)
Moderate knowledge	10 (23)	12 (32)
In-depth knowledge	1 (2)	1 (3)
Training program			0.62
Program 1	13 (30)	7 (18)
Program 2	7 (16)	9 (24)
Program 3	6 (14)	6 (16)
Program 4	5 (11)	5 (13)
Program 5	3 (7)	6 (16)
Program 6	5 (11)	2 (5)
Other	4 (9)	2 (5)

N = 82.

PDMP = prescription drug monitoring program.

For the primary outcome, the two reviewers had 96% crude agreement after initial assessment. For each theme, interrater reliability was calculated with kappa ranging from 0.76 to 1.00. All disagreements were resolved through consensus in discussion with a third independent reviewer.

The mean of the total number of themes recalled per participant was 3.1 in the narrative arm versus 2.0 in the summary arm (difference = 1.1, 95% CI = 0.6 to 1.7). The distribution of the total number of themes recalled by participants in each group is displayed in Figure 2.

Figure 2.

Distribution of the total number of themes recalled by participants in each arm.

The proportion of participants who recalled each theme was greater in the narrative arm compared to the summary arm for four individual themes (Table 2). For three themes (use PDMPs, attempt to use nonopioid therapies, and consider risks of opioid abuse and misuse), the unadjusted proportion of responses that recalled the themes ranged from 20% to 51% higher in the narrative arm, and these differences were statistically significant. In the analysis adjusted for sex, level of training, and prior knowledge of PDMPs, the relative risk of recalling those three themes ranged between 1.7 and 3.1 for the narrative arm compared to the summary arm.

Table 2.

Proportion of Participants That Recalled Each Individual Theme

Theme	Summary Arm Recall (n = 44), n (%)	Narrative Arm Recall (n = 38), n (%)	Absolute Difference, % Unadjusted (95% CI)	Relative Risk, Adjusted* (95% CI)
Use PDMP	10 (23)	28 (74)	51 (29 to 68)	3.1 (1.7 to 5.5)
Attempt to use nonopioid therapies	23 (52)	33 (87)	35 (15 to 53)	1.7 (1.2 to 2.4)
Consider the risks of opioid abuse and misuse	6 (14)	13 (34)	20 (2 to 40)	2.9 (1.1 to 7.4)
How to treat acute low back pain	27 (61)	30 (79)	18 (-3 to 37)	1.3 (0.9 to 1.7)
When prescribing opioids, use low doses and short duration	20 (46)	17 (45)	1 (-22 to 21)	1.0 (0.6 to 1.7)
ACEP	5 (11)	0 (0)	11 (1 to 25)	—†

PDMP = prescription drug monitoring program.

^*Adjusted for baseline characteristics of participants: sex, level of training, and self-reported baseline knowledge of PDMPs.

^†Relative risk not calculated due to instability of model.

The proportion of responses that recalled one theme, ACEP, was greater in the summary arm compared to the narrative arm. Although this difference was statistically significant, only a small number (n = 5) of responses recalled this theme, precluding analysis in the adjusted model.

The median word count of responses in the narrative arm was 68 (IQR = 48 to 95) versus 29 (IQR = 20 to 46.5) in the summary arm. In the summary arm, 24 (54%) responses were found to contain extraneous or falsely recalled information versus eight (21%) responses in the narrative arm (difference = 33%, 95% CI = 14% to 53%).

DISCUSSION

The purpose of this study was to compare the use of narratives with traditional summary presentations of evidence-based guidelines to promote recall of opioid prescribing recommendations. At 1 hour, readers of the narrative were able to recall, on average, one more theme than readers of the standard summary. By design, this study sought only to compare the memorability of two forms of communication, not to measure an effect on clinical practice. An increase in overall recall by one theme, therefore, demonstrated that the narrative proved more memorable than the guideline in its published format. The increased response word count and decreased presence of confabulated information in the narrative arm provided additional indications that readers were better able to remember the messages embedded in the narrative.

Falsely recalled information was significantly greater in the summary arm. One potential explanation is that low recall forced participants to confabulate items they only partially remembered. These results suggest that guidelines, in their raw form, may be misinterpreted.

Our findings fit theoretical models of cognitive processing, which hypothesize that narratives are processed differently than other forms of information.¹² The transportation-imagery model argues that engaging stories are processed like real experiences.²⁷ Several factors may increase engagement with the story, including relevance to the reader and character identification (homophily). Rather than instruct, the plot of the narrative illustrates to readers the consequences of not being able to access PDMPs, for example. We hypothesize that the narrative enhanced recall because the story invited readers to mentally rehearse the actions that they would have taken themselves.²⁷

Guided by these theoretical models, previous studies have used narratives to change patient behavior. Houston et al.¹³ deployed videos of patient-based narratives to achieve clinically significant reductions in blood pressure. Their intervention sought to maximize homophily between patients and storyteller through the use of culturally appropriate storytelling specific to the African American population of the study. In this study, the narrative portrayed a busy physician faced with a dilemma regarding opioid therapy, a situation that the participants were likely to have encountered.

Although they have not been tested specifically for guideline adoption, narratives have been incorporated into multifaceted educational interventions for providers. Sullivan et al.¹⁶ included depictions of clinical interactions in an interactive, Web-based training module regarding opioid therapy. Exposure to the training module, which also included standard summaries of evidence, increased physician knowledge and self-reported competence in the management of chronic noncancer pain. Effective approaches to knowledge dissemination may require multiple complementary tools, one of which can be narrative.⁵

LIMITATIONS

This study was not designed to detect whether exposure to narratives improved actual adoption of the guidelines. Previous studies have used guidelines to change opioid prescribing patterns but have not focused on testing different methods of implementing guidelines.²⁸ While guideline recall represents an important step toward adoption, enhanced recall alone may not be sufficient to improve uptake.^1,2,5-8 Also, this study assessed short-term recall rather than long-term learning, which may be a more robust proxy for guideline adoption.

The narrative was designed to match the thematic content of the summary. By its nature, however, the narrative did not communicate the exact content of the clinical guideline. It is unknown whether guidelines must be communicated in their original language and format to accomplish their goals of practice change. Yet this limitation could potentially be overcome by combining narrative communication with traditional content of a guideline.

Although the summary and narrative were matched with regard to content and length, they still differed across multiple factors. The order of the themes presented in the narrative was not precisely matched to the order in the summary. Also, the texts differed in Flesch grade level score, but we did not consider this difference relevant given the educational level of physicians.

Another limitation is that free recall may not simulate the typical mechanisms through which physicians learn or practice. The measurement of physician recall through multiple-choice questions, for instance, may have generated different results. In addition, physicians are able to consult references when necessary in clinical practice. However, physicians must first be able to recall that guidelines are available to consult. Furthermore, narratives may have additional benefits that were not investigated in this study, such as increasing receptiveness to controversial recommendations and highlighting scenarios in which guidelines should be used.

The inherent differences between the summary and narrative prohibited the two reviewers of the responses from being blinded to study arm. However, reviewers were blinded to each other’s scoring and resolved discrepancies through discussion with a third reviewer blinded to the study hypothesis.

The different response rates in the two arms of the study may have introduced bias into our results, as there were nine members of the narrative arm who did not complete the study as opposed to four in the summary arm. The reasons for not completing the study were not assessed. It is conceivable that readers of the narrative who did not complete the recall portion of the experiment were less engaged by the narrative and therefore would have demonstrated limited recall, decreasing the apparent difference between the intervention and control groups.

An important limitation is that the study participants primarily practice in a state without an active PDMP. The results, therefore, cannot be easily generalized to physicians in other regions. Although residents were required to attend this conference, the conference may have attracted physicians with particular clinical interests. Even if these threats to external validity were present, however, the overall low level of recall demonstrates that even motivated learners have difficulty remembering components of the opioid prescribing guidelines.

CONCLUSIONS

This investigation demonstrated that evidence-based narratives may enhance the memorability of guideline recommendations. To date, few studies have evaluated narrative as a method of disseminating clinical evidence to physicians. The optimal role of narratives in dissemination and implementation has yet to be determined, and narratives can be explored as complements to traditional communication strategies. Future studies must assess the effectiveness of narratives in improving guideline adoption in clinical settings.

Supplementary Material

Supp Data S1. Data Supplement S1.

Summary text (control) read by participants of study.

Supp Data S2. Data Supplement S2.

Narrative text (intervention) read by participants of study.

Supp Data S3. Data Supplement S3.

Study instrument used to elicit recall of narrative or summary text.

Supp Data S4. Data Supplement S4.

Scoring criteria used by investigators to evaluate written responses for the presence and absence of themes (primary outcome).