Closed-Loop eReferral From Primary Care Clinics to a State Tobacco-Cessation Quitline: Effects Using Real-World Implementation Training

Abstract

Introduction:

Patients who use tobacco are too rarely connected with tobacco use treatment during healthcare visits. Electronic health record enhancements may increase such referrals in primary care settings. This project used the Reach, Effectiveness, Adoption, Implementation, and Maintenance framework to assess implementation of a healthcare system change carried out in an externally valid manner (executed by the healthcare system).

Methods:

The healthcare system used their standard, computer-based training approach to implement the electronic health record and clinic workflow changes for eReferral in 30 primary care clinics that previously used faxed quitline referral. Electronic health record data captured rates of assessment of readiness to quit, and quitline referral 4 months pre- and 8 months (May–December 2017) post-implementation. Data, analyzed from October 2018 to June 2019, also reflected intervention reach, adoption, and maintenance.

Results:

Reach and effectiveness: From pre-implementation to post-implementation for eReferral, among adult patients who smoked, assessment of readiness to quit increased from 24.8% (2,126/8,569) to 93.2% (11,163/11,977), and quitline referrals increased from 1.7% (143/8,569) to 11.3% (1,351/11,977) and 3.6% were connected with the quitline post-implementation. Representativeness of reach: eReferral rates were especially high for women, African Americans, and Medicaid patients. Adoption: 52.6% of staff who roomed at least one patient who smoked referred to the quitline. Maintenance: eReferral rates fell by approximately 60% over 8 months but remained higher than pre-implementation rates.

Conclusions:

Real-world implementation of an electronic health record–based, eReferral system markedly increased readiness to quit assessment and quitline referral rates in primary care patients. Future research should focus on implementation methods that produce more consistent implementation and better maintenance of eReferral.

INTRODUCTION

Multiple effective smoking treatments exist.^1–3 However, it has proven difficult to meaningfully increase the rates at which primary care patients are offered and provided evidence-based smoking-cessation treatment.^4–9 Many strategies have been used to boost such rates¹⁰; perhaps the most promising and cost effective are ones that use electronic health records (EHRs) to help identify smokers, prompt cessation treatment offers, and facilitate treatment referral.^11–15

Several studies support the effectiveness of EHR-based approaches to cessation treatment offer and referral. In a demonstration project, Adsit et al.¹⁶ evaluated EHR enhancements designed to prompt smoker identification, assessment of interest in cessation treatment, and assistance with cessation treatment referral. Referral was accomplished with a secure, Health Insurance Portability and Accountability Act of 1996 (HIPAA)-compliant, EHR-based electronic referral (eReferral) order to the Wisconsin Tobacco Quit Line (WTQL). Upon receiving the referral, the WTQL attempted to contact the patient and ultimately provided secure, closed-loop feedback on the referral outcome: for example, whether the patient was contacted, accepted or declined WTQL services, and the service provided (e.g., counseling, pharmacotherapy). This feedback was automatically transmitted into the patient’s EHR.

The EHR enhancements of Adsit and colleagues¹⁶ were implemented in two primary care clinics that had been using a paper fax-to-quit method of cessation treatment referral. Prior to the EHR enhancements, about 0.3% of adult tobacco users were referred to the WTQL via fax; this rate climbed to 14% after the eReferral enhancements. Other observational studies suggest that eReferral can boost quitline referral rates in comparison with pre-existing fax referral or other methods.^17–20 One experimental study recently evaluated EHR enhancements that included eReferral to the quitline.²¹ A total of 23 primary care clinics across two healthcare systems were randomized to either eReferral resources or continued use of paper fax-to-quit referral; eReferral increased quitline referral (System A=17.9% vs 3.8%; System B=18.9% vs 5.2%) and connection rates (System A=5.4% vs 1.3%; System B=5.3% vs 2%) for eReferral and fax-to-quit, respectively.²²

The effectiveness of eReferral raises important questions about how well such enhancements can be implemented and sustained.²³ Intervention implementation in real-world settings is often extremely difficult, and poor implementation meaningfully reduces intervention effectiveness.^24–28 Moreover, EHR enhancements can pose substantial implementation challenges. In one study,¹³ even with EHR enhancements, only 54% of patients had their smoking status documented. In this same study, the majority of clinicians in the participating clinics (56%) never used a smoking-cessation treatment order set over a 9-month period.^11,13

Prior studies of EHR based smoking treatment have often used fairly intense research-based implementation systems and personnel^14,16,29,30 This, plus the inconsistent rates of smoking intervention sometimes observed with such systems,^11,22 raise questions concerning how well EHR-based smoking treatment systems will work if implemented with less intensive methods; that is, those likely to be used in real-world applications. The healthcare system, not the research team, managed implementation in the current study, enhancing the real-world relevance of this implementation analysis. This pragmatic application of the Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) framework^31,32 focused on RE-AIM constructs most relevant to health systems.

Implementation success and effects were assessed across 30 primary care clinics in a large health system (University of Wisconsin [UW] Health) in central Wisconsin to address unmet needs in tobacco use treatment within UW Health; for example, the healthcare system ranked poorly among state healthcare systems on assessment of willingness to quit. This investigation sought to examine the implementation success of a healthcare system–implemented, EHR-based, interoperable, and closed-loop eReferral mechanism for smoking treatment in primary care.

METHODS

Study Sample

Located in the Upper Midwest, UW Health is an integrated healthcare delivery system that treats >600,000 patients each year, with approximately 1,500 physicians and 16,500 other staff at six hospitals and more than 80 outpatient clinic sites. UW Health leadership, UW Center for Tobacco Research and Intervention research and outreach teams, and WTQL collaboratively designed this intervention. Over the course of 10 months (July 2016–May 2017), the intervention was conceptualized, designed, pilot tested in one large clinic, and launched in all UW Health primary and urgent care clinics. The UW School of Medicine and Public Health IRB deemed this project to be a program evaluation activity.

Prior to eReferral implementation, the UW Health policy was to determine and document every patient’s smoking status within the EHR, typically during the rooming process (“roomers,” typically medical assistants or licensed practical nurses). Primary care clinicians could then initiate a conversation with the patient about willingness to quit and provide a cessation intervention or fax a referral for the patient to the WTQL; neither action was completed and documented on a regular basis. This protocol led to high rates of screening for tobacco use (approximately 99%) but relatively low rates of documented assessment of interest in quitting or referrals to the WTQL.

The new EHR-based eReferral system was designed to increase both assessment of willingness to quit and linkage with the WTQL for treatment. In the new workflow, disseminated throughout outpatient primary and urgent care clinics (Figure 1), roomers were to ask patients about their current tobacco use status and update in the EHR as needed, assess and document readiness to quit within 30 days in all patients who used tobacco, and offer WTQL eReferral to individuals ready to quit smoking, which accords with Public Health Service Guideline recommended 5 “A”s.^1,15

Figure 1.

Wisconsin Tobacco Quit Line (WTQL) electronic referral (eReferral) workflow and feedback. Notes: This workflow documents decision points and possible outcomes of eReferral workflow, beginning with identification of a patient as currently smoking during a primary care encounter.

HER, electronic health record; HL7v2, High Level Seven International messaging version 2; ORM, object relational mapping; NRT, nicotine replacement therapy; ORU, object results mapping.

If roomers attempted to leave the vital sign assessment module in the EHR of patients who used tobacco without documenting the patient’s readiness to quit, a pop-up reminder prompted them: “The most important thing you can do to improve your health is to quit tobacco, and we can help. Are you interested in trying to cut back or quit in the next 30 days?” For adult patients who were ready to quit and had a Wisconsin address on file, an EHR alert and order set prompted the eReferral offer. The alert provided suggested language: “Could we have the Wisconsin Quit Line call you to discuss free coaching and nicotine replacement?” and three talking points to encourage enrollment in quitline services: (1) the increased likelihood of success with quitline support, (2) quit coaching is free, and (3) free nicotine-replacement medicines. This alert was suppressed for 90 days after eReferral to prevent duplicate referrals.

For patients who consented to WTQL referral, rooming staff entered the eReferral order within the EHR for clinician review. The order prompted clinicians to address smoking cessation with the patient, sign the order for quitline referral, and discuss other smoking-cessation treatments including cessation medications as appropriate. WTQL eReferral orders electronically signed by clinicians were then securely transmitted to the WTQL, where staff made up to five attempts to reach patients by phone at the preferred number listed in the EHR, starting within 3 days of eReferral. Patients who accepted a WTQL call and set a quit day within 30 days were provided quit smoking coaching and, if medically eligible according to a WTQL standing protocol, a 2-week starter kit of either nicotine patch, gum, or lozenge as selected by the patient.

The WTQL electronically returned results of the referral to the EHR within 2 weeks, both to the patient’s chart and to the referring clinician’s inbox. The EHR eReferral tools were also available during telephone, allied health, and orders-only encounters, but clinic staff had to navigate to the order set to access these tools during such encounters.

Staff received intervention training via online UW Health computer-based training (CBT), the standard practice for all new clinical practice initiatives in this system. Separate CBT resources were developed for clinicians and rooming staff. CBT components included an online video, slides, and screenshots detailing step-by-step instructions regarding the two new EHR alerts; detailed workflows; and information about WTQL services. The 12-minute video and the 11-slide set presented information on: (1) intervention goals, (2) targeted patients and involved staff, (3) specific workflow changes in consecutive steps, (4) screenshots of EHR pages illustrating specific steps and locations, (5) how to send the referral to the quitline, (6) what to convey to the patient, and (7) how to get further information (e.g., frequently asked questions) (Appendix Text). No in-person training occurred. One month before eReferral launch, clinic managers received training materials and distributed them to their rooming staff and clinicians. Clinic managers were responsible for documenting CBT completion before eReferral went live. The CBT was included in new employee training after eReferral launch. Following implementation, clinic managers received monthly feedback regarding rates of each roomer’s assessment of willingness to quit and WTQL eReferral, which they were to share with rooming staff and clinicians.

Measures

In RE-AIM, reach and effectiveness were assessed at the individual level. Reach was assessed by examining the percentages of patients who smoked who were: (1) asked if they were willing to make an aided quit attempt and (2) referred. Representativeness of reach was indexed by the reach of these intervention outcomes, including referral rates, across key population dimensions (e.g., race, gender, insurance status). Effectiveness was assessed via the percentage of patients who smoked and were connected with the WTQL (i.e., accepted WTQL treatment).

Adoption and maintenance were assessed at the setting and staff levels. Adoption was assessed by overall percentages and variance across roomers and clinicians in: (1) assessing interest in making an aided quit attempt, (2) identifying patients who smoked and were willing to make a quit attempt in the next 30 days, and (3) referring to treatment (WTQL). Finally, maintenance was examined via trends in execution of the three smoking treatment elements over the course of this study.

Statistical Analysis

The authors gathered de-identified EHR data from UW Health to evaluate the eReferral initiative. Only visits from adult patients (aged ≥18 years) who were listed as current tobacco users in the 4 months pre-implementation and 8 months post-implementation launch (May 9, 2017) were extracted. The following variables were captured for every adult tobacco user visit between January and December 2017: month of visit, patient age (truncated at 90 years), gender, race, ethnicity, insurance (coded as commercial/private, Medicare, Medicaid, or uninsured, which could overlap), clinic of encounter, tobacco products used, patient willingness to quit (yes, no, or not assessed), and WTQL referral. Administrative data on the size of clinic adult patient panels and the number of clinicians on staff at each clinic were also gathered.

Only data from adult patients who reported smoking were analyzed (versus exclusive users of other forms of nicotine such as smokeless or e-cigarettes). Data from pediatric, specialty, and urgent care clinics and from the pilot clinic were also excluded, leaving 30 clinics. The reach of quit readiness assessment and WTQL eReferral was analyzed at the patient level. The denominators reflect the total number of adult patients who smoked and were seen in a target clinic in the relevant period (i.e., rates of readiness to quit reflect all patients who smoked and made clinic visits, not just those assessed, and rates of eReferral reflect all patients who smoked and were seen, not just those ready to quit). Investigators conducted bivariate chi-square tests and t-tests to examine the representativeness of reach across demographics, insurance, tobacco use groups, and visit counts. Adoption and implementation were assessed by descriptive data examining the rates and variance in assessment of quit readiness and eReferral at the visit level; overall; and by rooming staff, clinician, and clinic. The denominator for these rates was the total number of visits by adult patients who smoked in the target period.

RESULTS

Characteristics of the adult patients who smoked and were seen in the 30 target clinics in the 4-month pre-launch and 8-month post-launch are shown in Table 1. Among those seen at participating clinics during the relevant time periods (a patient level of analysis), the rate of assessing readiness to quit increased from 24.8% pre-launch to 93.2% post-launch, quit readiness rates increased from 9.5% to 31.9%, and the WTQL referral rate increased from 1.7% (143/8,569; this estimate is not presented in Table 1) to 11.3%. The pre-launch WTQL referral rate is likely an overestimate: It included fax referrals from throughout the UW Health system, not just the target 30 clinics. Of those eReferred post-implementation, 28.7% (3.6% of all patients who smoked and were seen at a clinic) accepted WTQL services. Clinic-specific results are shown in Appendix Table 1.

Table 1.

Pre- and Post-Launch Patient-Level Descriptions by Patient Characteristics Among Patients Who Smoked

	Pre-launch (N=8,569)				Post-launch (N=11,977)
Variable or level	M (SD) or n (%)	Assessed, M (SD) or n (%)	Ready to quit, M (SD)/n (%)	M (SD) or n (%)	Assessed, M (SD) or n (%)	Ready to quit, M (SD) or n (%)	eReferred, M (SD) or n (%)
Age in years, M (SD)	48.0 (14.8)	48.1 (14.6)	46.8 (13.9)a	48.0 (14.9)	48.1 (14.8)b	47.6 (13.9)c	47.6 (13.6)
Number of clinic	1.6 (1.0)	—	—	1.9 (1.5)	1.1 (0.3)d	2.3 (1.9)e	2.4 (1.8)f
visits, M (SD)
Gender
Men	3,981 (46.5%)	1,007 (25.3%)	370 (9.3%)	5,620 (46.9%)	5,245 (93.3%)	1,751 (31.2%)	570 (10.1%)
Women	4,588 (53.3%)	1,119 (24.4%)	445 (9.7%)	6,357 (53.1%)	5,918 (93.1%)	2,072 (32.6%)	781 (12.3%)
Race
White	7,333 (86.7%)	1,805 (24.6%)	662 (9.0%)g	10,238 (86.7%)	9,576 (93.5%)g	3,204 (31.3%)g	1,100 (10.7%)g
African American	902 (10.7%)	234 (25.9%)	120 (13.3%)h	1,255 (10.6%)	1,139 (90.8%)h	461 (36.7%)h	194 (15.5%)h
Other minority group	220 (2.6%)	56 (25.5%)	21 (9.5%)g,h	318 (2.7%)	291 (91.5%)g,h	93 (29.2%)g	38 (11.9%)g
Ethnicity
Hispanic	275 (3.2%)	60 (21.8%)	30 (10.9%)	364 (3.1%)	326 (89.6%)	112 (30.8%)	34 (9.3%)
Not Hispanic	8,222 (95.9%)	2,046 (24.9%)	775 (9.4%)	11,500 (96.0%)	10,735 (93.3%)	3,667 (31.9%)	1,304 (11.3%)
Insurance type
Commercial (yes versus no)	4,752 (55.5%)	1,146 (24.1%)	461 (9.7%)	6,870 (57.4%)	6,387 (93.0%)	2,206 (32.1%)	754 (11.0%)
Medicare (yes versus no)	1,870 (21.8%)	470 (25.1%)	152 (8.1%)	2,640 (22.0%)	2,484 (94.1%)	808 (30.6%)	293 (11.1%)
Medicare (yes versus no)	1,870 (21.8%)	470 (25.1%)	152 (8.1%)	2,640 (22.0%)	2,484 (94.1%)	808 (30.6%)	293 (11.1%)
Medicaid (yes versus no)	1,534 (17.9%)	402 (26.2%)	170 (11.1%)	2,088 (17.4%)	1,956 (93.7%)	732 (35.1%)	271 (13.0%)
Uninsured (yes versus no)	789 (9.2%)	235 (29.8%)	84 (10.6%)	922 (7.7%)	874 (94.8%)	310 (33.6%)	109 (11.8%)
Other tobacco
Yes	416 (4.9%)	75 (18.0%)	16 (3.8%)	598 (5.0%)	531 (88.8%)	95 (15.9%)	22 (3.7%)
No	8,153 (95.1%)	2,051 (25.2%)	799 (9.8%)	11,379 (95.0%)	10,632 (93.4%)	3,728 (32.8%)	1,329 (11.7%)
Total	—	2,126 (24.8%)	815 (9.5%)	—	11,163 (93.2%)	3,823 (31.9%)	1,351 (11.3%)

Notes: Boldface indicates statistical significance (p<0.05) via χ² test.

^aThose ready to quit were significantly younger than those not ready to quit; mean difference=1.31, SE=0.51.

^bThose assessed were significantly older than those not assessed post-launch; mean difference=−2.29, SE=0.58.

^cThose ready to quit were significantly younger than those not ready to quit; mean difference=0.62, SE=0.28.

^dThose assessed had significantly fewer visits than those who were not assessed, Mean Difference=−0.88, SE=0.05.

^eThose ready to quit had significantly more visits than those who were not ready to quit; mean difference=−0.57, SE=0.03.

^fThose eReferred had significantly more visits than those who were not eReferred; mean difference=−0.56, SE=0.04.

^g,hRace subgroups with different superscript letters differ significantly in rate assessed, ready to quit, or eReferred. For example, for post-implementation eReferral rates, whites with a g superscript differ from African Americans with an h superscript, but not from the other minority group, which also has a g superscript.

To assess representativeness of reach, bivariate relations between available patient-level variables (demographics, insurance, and tobacco type), and rates of assessment, quit readiness, and eReferral are shown in Table 1. All three outcomes tended to vary significantly as a function of race, insurance type, and use of other (non-cigarette) tobacco products. eReferral was especially likely for women, African American patients, those receiving Medicaid, and those who smoked exclusively versus those who used multiple forms of tobacco. Age was related to whether the patient was assessed for readiness to quit and found to be ready to quit, but age differences were small. Having had more clinic visits was associated with lower rates of readiness assessment but higher rates of readiness to quit and of eReferral.

The mean number of visits with patients who smoked for the 411 roomers (who had at least one encounter) was 55.7 (SD=57.0, median=43, range=1–260). The mean percentage of patients assessed for readiness to quit smoking across roomers was 72.8% (SD=35.3%, median=89.4%, range=0%–100%), the mean percentage of patients found ready to quit across roomers was 20.2% (SD=22.0%, median=22.0%, range=0%–100%), and the mean percentage of patients eReferred to the quitline across roomers was 5.2% (SD=10.6, median=1.5%, range=0%–100%). A total of 64 of 411 (15.6%) roomers never assessed readiness to quit. These appeared to be staff who roomed infrequently, as their mean number of smoker visits was only 1.4 (SD=1.3), much lower than those who assessed readiness to quit at least once (mean=65.7, SD=56.6, t= −9.07, p<0.001). More than a quarter of roomers (114, 27.7%) never noted that a patient was ready to quit during the 8-month post-launch period. The mean number of smoker visits for such roomers was 3.0 (SD=7.8) vs 75.9 (SD=54.8; t= −14.1, p<0.001) for roomers who identified at least one patient as ready to quit. Nearly half of roomers (212, 47.4%) never eReferred a patient; the mean number of smoker visits for such roomers was 15.6 (SD=30.8) vs 88.9 (SD=51.6; t= −17.1, p<0.001) for roomers who eReferred at least one patient. Thus, roomers who saw very few patients who smoked tended to significantly underperform across all outcomes. Figure 2A shows box-and-whisker plots depicting the mean, median, quartiles, and range in roomer-level assessment, readiness to quit, and eReferral rates among roomers who had at least ten visits with patients who smoke post-implementation.

Figure 2.

Box-and-whisker plots of rates of assessing patient interest in quitting, patient readiness to quit within 30 days, and eReferral to the WTQL. Notes: Means are marked with an X, medians are marked with a horizontal black line, and quartiles (25th and 75th percentiles) are marked by the lower and upper borders of the colored boxes, respectively. Whiskers depict range within 1.5 times the interquartile range, and dots indicate outliers that are more than 1.5 times the interquartile range outside the box. Panel A shows distributions of implementation rates across roomers (MAs/licensed practical nurses) primarily responsible for implementation of assessment and eReferral. Panel B shows distributions of implementation rates across clinicians responsible for approving eReferral orders. Only roomers and clinicians who saw at least ten patients who smoked during the 8-month implementation period were included in these analyses.

eReferral, electronic referral; WTQL, Wisconsin Tobacco Quitline; MA, medical assistant; LPN, licensed practical nurse.

Examinations of Fleiss–Cuzick estimators of intraclass correlation coefficients (ICCs) for binary outcomes³³ indicated that visits occurring with individual roomers were more similar than were visits across different roomers, particularly for assessment rates (ICC=0.15) and readiness to quit (ICC=0.09). The intra-roomer correlation coefficient was lower for eReferral (ICC=0.03). These analyses were restricted to roomers with at least two visits with smokers (330 of 411 roomers, 80.3%).

Of 413 clinicians, 49 (11.9%) saw no patients identified as ready to quit and 146 (35.4%) never eReferred a patient to the quitline. Clinicians with no patients ready to quit had fewer visits with smokers (mean=4.4, SD=5.6) than did those with one or more patients ready to quit (mean=62.3, SD=54.9, t= −7.38, p<0.001). Clinicians with no eReferred patients also had fewer visits with patients who smoked than did clinicians with eReferred patients (mean=18.8, SD=24.7 vs mean=74.6, SD=55.9, t= −11.4, p<0.001). Figure 2B shows box-and-whisker plots of assessment, readiness to quit, and eReferral rates among clinicians who had at least ten visits with patients who smoke post-implementation.

Rates of visit-level assessment, readiness to quit, and WTQL referral are shown by month from launch in Figure 3 to illustrate maintenance of assessment and intervention over time. Panel A depicts dramatic increases in the rate at which readiness to quit was assessed at smoker visits after implementation. Assessment rates reached a peak in Month 1 and did not change significantly after Month 2. Rates of reported readiness to quit at visits also increased, peaking during the launch month with only small declines thereafter. Panel B shows that fax referral rates declined slightly (but not to 0) once eReferral was introduced, and that eReferral rates were consistently higher than pre-implementation fax referral rates. The percentage of all adult smoker visits that resulted in patients receiving WTQL services was consistently higher post-launch than pre-launch. Despite this, rates of eReferral declined over the post-launch period with significant declines every month through Month 5. This is unlikely to be due solely to repeat visits during the implementation period of study, as the mean number of visits per patient was two. WTQL service acceptance rates also declined significantly in the first 2 months after launch, with a more modest decline after Month 4.

Figure 3.

Plots of implementation maintenance across time. Notes: Panel A shows rates of roomer documentation of assessment of patient readiness to quit within the next 30 days (solid line) and documentation of patient reports of readiness to quit within 30 days (dashed line) in the 4 months preceding and 8 months following electronic referral (eReferral) launch. Panel B shows rates of eReferral to the Wisconsin Tobacco Quit Line (WTQL) during implementation (solid line), fax referral to the WTQL pre-implementation and during implementation (dotted line; in months −4 to −1 this includes referrals from all clinics in the healthcare system; in months 0–7, this includes fax referrals from only the 30 primary care clinics of interest in this study), and acceptance of WTQL services among all referred (regardless of referral method, dashed line).

^aThe rate for this outcome at this time point is significantly higher than the rate for the same outcome at all subsequent time points combined at p<0.05 in a χ² test.

^bThe rate for this outcome at this time point is significantly lower than the rate for the same outcome at all subsequent time points combined at p<0.05 in a χ² test.

DISCUSSION

Multiple effective smoking-cessation treatments exist,³⁴ and healthcare system changes can effectively promote their implementation and use.^1,10 However, a failure to consistently implement smoking interventions in healthcare settings is a major impediment to reducing smoking prevalence in clinic populations.^{5,6,27,35–39} In this study, implementation and maintenance strategies were designed and executed by the healthcare system, enhancing external validity. Implementation strategies included online videos, detailed instructions, and screenshots to guide medical assistants and clinicians. Research staff delivered no implementation training. Clinic managers were responsible for ensuring that staff completed training and for providing performance feedback post-implementation.

The real-world implementation/training strategy appeared to increase the reach of quitline eReferral. Among adult patients who smoked, the rate of assessing readiness to quit increased from 24.8% pre-launch to 93.2% post-launch, the rate of readiness to quit increased from 9.5% to 31.9%, and the WTQL referral rate increased from 1.7% to 11.3%. Thus, rate of assessment of smoker readiness to quit increased more than threefold while rate of referral to the quitline increased more than sixfold. There was evidence of maintenance of rooming staff documentation of the assessment of readiness to quit and patient reports of readiness. Although eReferral rates declined significantly across months, they remained at least twice as high as pre-implementation fax referral rates.

Examination of the representativeness of reach showed that the smoking-related activities were delivered fairly equitably across patient groups; some differences were significant but typically small. One meaningfully large difference was that African American patients who smoked were more likely to be eReferred to the quitline than were white patients who smoked (15.5% vs 10.7%). Also, those using other forms of tobacco in addition to cigarettes were less likely to be eReferred than were those who smoked cigarettes only (3.7% vs 11.1%). This is consistent with the finding that those who used multiple forms of tobacco were significantly less likely to express interest in quitting smoking than are those who smoke exclusively (Table 1). In sum, there was good reach across diverse smoker populations (e.g., women, those on Medicaid).

Only about one third of patients who said they were willing to quit actually accepted quitline referral, which is similar to other quitline referral results.^37–39 The cause of this drop-off is unknown; some of these patients may have wanted to quit on their own or wanted to quit with a different form of assistance, or some may have merely responded to perceived social demand. A quitline connection rate of 3.6% of smokers may seem modest, but this compares with an estimated connection rate of 0.6 at pre-launch. An estimated 10% of 600,000 patients making annual visits in this healthcare system smoked. This translates into 60,000 smokers. Thus, real-world implementation of the EHR-based system increased quitline connection from about 336 to about 2,160/year, a difference of public health significance (although future erosion of eReferral could reduce this).

The performance of intervention steps varied markedly across rooming staff and clinicians. Rates of assessing readiness to quit and quitline eReferral ranged from 0% to 100% across roomers (who were responsible for most of the workflow steps) and clinicians (who signed off on eReferral orders). However, the large majority of roomers assessed quitting interest in more than 80% of their patients, and very few fell below 60% (Figure 2A).

Variation in assessment is likely due to roomer rather than patient factors. This view is supported by ICCs that show assessment rates in particular were more similar within roomers (across visits) than across rooming staff. In this study, roomers seeing relatively few patients had particularly low assessment rates; thus, the range of implementation rates might unduly reflect such staff. Variation in rates of identifying those ready to quit and willing to be eReferred appears to be less roomer-dependent and may reflect patient motivation level.

Rates of documenting assessment of readiness to quit and smokers ready to quit were very stable across the post-implementation period. Thus, eReferral, even with low-intensity implementation, appears to result in the consistent assessment of smoker readiness over time. However, although the percentage who smoked and said they were ready to quit did not drop meaningfully in Months 2–7, eReferral of such individuals dropped significantly during this time, for unknown reasons. Roomers may have stopped using or promoting eReferral, or patients may have declined repeat referrals; low rates of multiple visits cast doubt on the latter account. The drop in eReferral rates was steep (falling by more than 50%) and warrants further research. Perhaps more intensive implementation training (e.g., extended or in-person training, or training that provides roomers more skills such as motivational intervention) might have improved maintenance.

Limitations

Limitations include a lack of follow-up data on smoking outcomes (e.g., abstinence). Also, this pragmatic⁴⁰ health system–led project comprised a limited set of implementation measures. Thus, some targeted RE-AIM constructs were not assessed (e.g., cost, adaptation⁴¹). This evaluation also lacks intensive qualitative evidence that might shed light on the sources and determinants of variation in staff and clinician performance. Finally, the extent to which the EHR-based system change altered actual intervention delivery versus reporting of delivery is uncertain.

CONCLUSIONS

Low-intensity CBT of an EHR-based system change yielded marked and persisting improvements in rates of documenting readiness to quit among adult primary care patients who smoke. It also more than doubled the rate that patients who smoked were referred to the tobacco quitline. Although eReferral rates declined over the first several months of implementation, they remained much higher than pre-implementation fax referral rates. The reach of WTQL services improved after implementation of eReferral and reach was fairly equitable across patient characteristics available for analysis. Reach was particularly high among African American patients and Medicaid recipients.

There was considerable variability in the performance of roomers. About half of roomers never eReferred a smoker to quitline treatment. Absolute eReferral and WTQL service reach rates remained low, suggesting the need to explore more intensive implementation strategies (e.g., in-person training) or motivational strategies to increase smoker willingness to accept treatment, such as incentives^42–44 and warm hand-offs (e.g., a clinician handing the patient the phone to receive quitline treatment).^44,45

This research identified factors that are associated with relatively poor eReferral implementation: (1) patients who use other forms of tobacco in addition to cigarettes, (2) staff or clinicians who do not regularly perform rooming duties or who see relatively few patients who smoke, and (3) the passage of time post implementation. Thus, this research suggests how implementation efforts may be targeted—which patients, which staff, and when.