Abstract
Smoking rates are high among low-income populations who seek care in safety-net clinics. While most safety-net clinics screen for cigarette smoking, there are substantial disparities in the delivery of smoking cessation counseling in these systems. We conducted a mixed method study between July 2016 and April 2017 to examine receipt of smoking cessation counseling and estimate recent cessation attempts among primary care patients in four safety-net clinics in San Francisco. We used the electronic health record (EHR) to examine receipt of cessation services and estimate cessation attempts, defined as transition from current to former smoking status during the 9-month study period. We conducted interviews with 10 staff and 16 patients to assess barriers to and facilitators of providing cessation services. Of the 3301 smokers identified via EHR, the majority (95.6%) received some type of cessation counseling during at least one clinical encounter, and 17.6% made a recent cessation attempt. Recent smoking cessation attempts and receipt of smoking cessation services differed significantly by clinic after adjusting for demographic factors. We identified patient and staff-level pre-disposing, reinforcing and enabling factors to increase delivery of cessation care, including increasing access to cessation medications and higher intensity counseling using a team-based approach. The EHR presents a useful tool to monitor patients' recent cessation attempts and access to cessation care. Combining EHR data with qualitative methods can help guide and streamline interventions to improve quality of cessation care and promote quit attempts among patients in safety-net settings.
Highlights
-
•
Smoking rates are high among low-income populations who seek care in safety-net clinics.
-
•
The electronic health record (EHR) can be used to estimate recent cessation attempts among patients.
-
•
EHR data and qualitative methods are useful to evaluate tobacco quality improvement initiatives.
-
•
We highlighted strategies to increase access to cessation medications and team-based counseling.
1. Introduction
Cigarette smoking remains the leading preventable cause of death in the United States (U.S. Department of Health and Human Services, 2014). Despite the decline in prevalence of cigarette smoking in the general population, smoking remains concentrated among low-income populations, those who belong to racial/ethnic minorities and those who have mental illness and substance use disorders (Jamal et al., 2018; Guydish et al., 2016; Schroeder and Morris, 2010). These populations have an array of medical, psychological, and social needs, and seek care in safety-net clinics, which are inadequately resourced to address these needs (Nguyen et al., 2016; Ein Lewin, 2000). While the majority of safety-net clinics screen patients for cigarette smoking, there are substantial disparities in providing smoking cessation care (Vijayaraghavan et al., 2017; Flocke et al., 2017; Nguyen et al., 2016). Health information technology (HIT) could fill this gap in access to cessation care.
With the use of the electronic health record (EHR) (Pillemer et al., 2016), the Center for Medicare and Medicaid Services developed the “meaningful use” criteria to incentivize health systems to use the EHR to drive practice improvements (Blumenthal and Tavenner, 2010). One of the metrics of “meaningful use” is screening and providing smoking cessation services, which are linked with incentive payments to health systems (Blumenthal and Tavenner, 2010). Thus, the EHR is a potentially useful tool for evaluating receipt of cessation services and in estimating recent cessation attempts, which are often hard to obtain on a population level given that patient-specific quitting data are impractical to collect during routine clinical encounters.
Since 2009, the San Francisco Health Network (SFHN), a network of primary care clinics, hospitals, behavioral health clinics, and other programs has implemented several QI initiatives to increase access to cessation care. These activities included training front-line staff (i.e. the medical assistant in the primary care clinics) to screen and document cigarette smoking in the EHR, and health care providers and ancillary staff (e.g. behavioral assistants who work at a health coach capacity to address psychosocial barriers to cessation) to provide more intensive counseling and document receipt of services in the EHR. SFHN's QI activities intensified in 2016, when both the medical assistants and behavioral assistants received additional training for smoking cessation counseling and clinics changed their workflows so that all smokers with a clinical encounter had the potential to receive some cessation intervention. In this study, we used a mixed methods approach to evaluate these activities among smoking patients in four primary care clinics within the SFHN from July 2016 through April 2017. We used the EHR to examine receipt of cessation services and to estimate recent cessation attempts, and used qualitative methods to assess barriers to and facilitators of providing cessation care.
2. Methods
2.1. Study sites
Four primary care clinics located in San Francisco were included because of the high prevalence of smoking among its patients: site 1, a community clinic located in downtown that serves mostly impoverished seniors (smoking prevalence 25%); site 2, an academic internal medicine practice in a public hospital that serves a diverse patient population (smoking prevalence 24%); site 3, a community clinic that serves a predominantly African American community (smoking prevalence 35%); and site 4, a community clinic that serves a predominantly homeless population (smoking prevalence 64%). All study procedures were approved by the University of California, San Francisco Committee on Human Research.
2.2. Theoretical framework
The Precede-Proceed model served as the theoretical framework for the study (Green and Kreuter, 2005). The EHR served as the primary data source for the epidemiological assessments of receipt of cessation counseling and recent smoking cessation attempts among patients. We conducted in-depth, semi-structured interviews and focus groups to identify predisposing, reinforcing, and enabling factors associated with the delivery of cessation care. The objective of these qualitative analyses was to provide complementary information to the EHR quantitative data, and to provide explanations, if any, to the observed patterns in the quantitative data. Response from qualitative data was not intended to be representative of the staff and patient populations in these clinics.
2.3. EHR data extraction & sample
We relied on i2i Tracks (http://www.i2isys.com/p/i2itracks) and Tableau (http://www.tableausoftware.com/products/desktop) to extract clinical and administrative data for all patients seen for primary or urgent care in the four clinics between July 2016 and April 2017. Search criteria included any patient listed as current smoker in the EHR at the onset of the study in July 2016 and who had at least 3 unique primary or urgent care encounters during the 9-month study period. Of the 13,000 patients seen between July 2016 and April 2017, we identified 3310 current smokers who had 3 unique encounters (9930 total encounters) for inclusion in our analytic sample.
2.4. Cessation services utilization verified using the EHR
We examined receipts of three types of smoking cessation counseling: (1) medical assistant counseling, which consisted of medical assistants screening for cigarette smoking and referring to one or more cessation resources including those available on-site and/or the California Smoker's Helpline (Zhu et al., 2002); (2) any provider counseling, including a primary care provider or other provider (e.g. urgent care provider) who provide counseling during the clinical encounter (Rizzato Lede et al., 2015); and (3) behavioral assistant counseling, counseling provided by ancillary staff at the health coach level who are trained to provide on-site smoking cessation counseling. We determined whether the patient was prescribed smoking cessation medication {e.g. nicotine replacement therapy (NRT) or non-NRT medication (Bupropion or Chantix)}.
2.5. Recent smoking cessation attempt
The primary cessation outcome measure was a dichotomous measure of recent smoking cessation attempt during the study duration, which we defined as a transition in smoking status from current smoker at visit 1 to former smoker at either visits 2 or 3. For this analysis, we did not include smoking status at visit 1 as all participants were current smokers.
2.6. Other measures
We extracted the following patient data from the EHR: demographics (age, race/ethnicity), health insurance type (Medicare, Uninsured, Medicaid [“Medi-Cal” in California], Healthy San Francisco/Healthy Workers, a county based program that provides free medical care, or “Other”] and smoking-related health comorbidities using ICD-9 codes, which were available at the time of data extraction (asthma [493.]; chronic obstructive pulmonary disease [COPD, 491.22, 496.]; depression [296.2, 296.3, 311]; diabetes [250., 249.80]; human immunodeficiency virus status [HIV, 042.]; hypertension [401.]; and ischemic vascular disease [414.9]).
2.7. Qualitative data collection
We used a purposive sampling strategy to recruit staff who were engaged in providing cessation care by contacting the clinic medical directors through email to inform them about our study, ask permission to advertise our study, and to contact staff for the interviews. Eligible staff consisted of medical assistants, behavioral assistants, nurse practitioners, counselors, and pharmacists who were able to provide informed consent. We then emailed eligible staff about our study, and for those interested, we organized a time to obtain verbal consent and conduct the interviews in a private room at the clinic site. We conducted in depth interviews with 10 staff: three medical assistants, one pharmacist, one nurse practitioner, two cessation counselors, and three behavioral assistants, for a total of one participant in site 1, four in site 2, one in site 3, and four in site 4. There were eight individual interviews and one joint interview to accommodate the schedules of two staff participants in site 4.
We recruited patients by posting flyers in the clinic lobby and waiting room, and obtained referrals from medical assistants prior to or after the clinic encounter. Patients eligible to participate were 18 years or older, current or former smokers, and able to provide informed consent. We conducted both in-depth interviews, semi-structured interviews and focus groups with patients. In two of the clinic sites (sites 1 and 4), we utilized pre-existing cessation and patient advisory groups as a forum to conduct focus groups with patients. In site 3, we conducted in-depth, semi-structured interviews of patients because there were no pre-existing groups to conduct focus groups. We did not conduct additional patient interviews in site 2 because we had reached thematic saturation.
Study staff approached eligible patients in the waiting or visit room and informed them about the study and obtained verbal consent. Those interested in participating were invited to stay and complete the interview. There were two focus groups that consisted of four patients in each group, one focus group that consisted of six patients and two individual patient interviews, for a total of six participants in site 1, two in site 3, and eight in site 4.
We assessed barriers to and facilitators of providing cessation care at the individual-, clinic-, and system-wide levels among staff, triggers to smoking, barriers to cessation, and barriers to receipt of cessation services among patients. Research team members (DW, MV) conducted interviews and/or focus groups that lasted for 30 to 45 min. Participants were provided a $10 gift card.
2.8. Quantitative data analysis (EHR data)
We conducted bivariate analyses comparing the four primary care clinics for demographics, patient co-morbidities, receipt of smoking cessation counseling, receipt of smoking cessation pharmacotherapy, and recent smoking cessation attempts, using ANOVA for continuous variables and Pearson's chi-square for categorical variables. We developed three logistic regression models to identify variables independently associated with (1) making a recent smoking cessation attempt, (2) receipt of any type of smoking cessation counseling, including medical assistant, provider, or behavioral assistant counseling, and (3) receipt of any provider counseling. We used generalized estimating equation to model these associations, clustering by participant ID and using an exchangeable correlation structure. All models adjusted for visit number, age, sex, race/ethnicity, insurance type, clinic site, and co-morbidities. For the model on recent smoking cessation attempt, we also included receipt of medical assistant counseling, any provider counseling, and prescription of NRT and non-NRT medications as additional predictors. We included data from visits 2 and 3 only (excluding visit 1) in the model for recent cessation attempts because all participants started as current smokers. We included data for all three visits for the models on receipt of smoking cessation services. Because receipt of behavioral assistant counseling was overall very low for all clinics, it was not included as an independent outcome in these models. Quantitative statistical analyses were performed using SPSS 25 (IBM Corporation, Armonk, NY, USA).
2.9. Qualitative data analysis
Audio files were transcribed verbatim and texts were redacted of any personal identification. Transcripts were coded using ATLAS.ti (Version 8). We analyzed the transcripts using a directed content analysis approach (Hsieh and Shannon, 2005). The PI and research staff developed the first iteration of a codebook based on a priori hypotheses, which was refined iteratively during the coding process. We resolved discrepancies in code description and assignment through discussion. We categorized codes into themes and subthemes, and used quotations to reflect the themes (Table 4).
Table 4.
Themes and illustrative quotes.
| Theme | Subtheme | Illustrative quote |
|---|---|---|
| Patient-level predisposing factors | Barriers to access to treatment | “I get [medications]. I've gotten it every time. I've done it through Kaiser and through San Francisco General as well, and I've always gotten it but it's been kind of like a hassle – wait for this, you gotta get your insurance for that. I mean, I would quit but- no, I'd just fall right off the wagon.” (Patient Focus Group) |
| Challenges with substance use and mental illness | “It's like – the reason why I smoke cigarettes and drink beer is because I used to be highly addicted to marijuana, you know, and I want to cut that out of my life, and until recently it was illegal, and I'd always be in trouble for havin’ a joint on me and stuff like that, and the cops would be threatening me to – jail time, and who's your dealer,– so I decided to go with cigarettes and drinkin’. And it works for me but I want to stop it.” (Patient Focus Group) | |
| Patient-level enabling factors | Access to treatment | “Interviewer: So maybe – I don't know if it is possible, but would it maybe even be helpful to have different types of groups and training offered here? Participant: I think so – we did have something like that. It was like a support group. It was like the pharmacist, and it was kind of like – they sent an invitation, − just come. Not a free-for-all, but it wasn't appointments. Where the pharmacist would talk about – this is the nicotine patch, this is the gum, and I know there's the pill like – where – to like talk about it. Demystify [it]. ‘Cause the more I know – oh, maybe I do want to try it.” (Patient Focus Group) |
| Provider level pre-dispoing factors | Competing priorities | But a lot of times, what I'm seeing is that that's kind of like the last thing that – most of our patients are really chronically ill, so the providers are in the room with the patients, tryin’ to get to the most important stuff on the agenda, and smoking is on the agenda somewhere, but it's not like the top five. (Behavioral Assistant Staff Interview) |
| Job descriptions and ownership for counseling | “I think everyone has to be involved, that comes in contact with the patient. It has to be the MEA, it has to be the nurse, and it has to be the doctor, because sometimes they come for nurse visits, and the nurse visits just want to do the wound care or they just want to do the refill. So – I think a presentation, like he was saying, to the staff, a training, This is smoking. This is how important it truly is, I know you guys are all busy, but anybody can go and buy cigarettes. That's why it's such a big problem.” (Medical Assistant Staff Interview) | |
| Provider level reinforcing factors | Training for staff to provide cessation counseling | “I think it would be [helpful] – I think for those people who don't have a real – a history – like – someone's never smoked, they really don't know what addiction really is --- but I think some type of training would be actually good – I don't know if on a consistent basis, but at least a couple of sessions, where people actually can understand what you're talking about.” (Behavioral Assistant Staff Interview) |
| Ongoing quality improvement efforts | “One of the things that we've started is, at the monthly meetings, we go over the statistics, which you might be calling metrics. We go over the data monthly together, and we talk about why there hasn't been any improvement. I think that's kind of a new conversation. Perhaps it's always been talked about intermittently, but we're doing it every month now. We're talking about what works and what doesn't. So I think making the data known to everyone is important. And we're all there, from all the clinics, so we can all kind of see what's going on.” (Behavioral Assistant Staff Interview) | |
| Provier level enabling factors | Cessation leadership | “We have a few champions among the staff, one of whom is a psych nurse-practitioner, who – we probably get at least a third of our referrals from her, because she really, really talks it up with her patients. The other [referrals] are random, actually. People have – our health worker who registers patients – when we register patients for the primary-care clinic, the nurses, as part of the regular routine, or the health worker, always asks them about smoking, and they always tell them about the group. (Nurse Practitioner Staff Interview) |
| Better communication among clients and staff | “Each BA [behavioral assistant] is assigned doctors, specific doctors, so we have huddle in the morning and afternoon, so we announce in the huddle, hey, please remember, if you have patients who are smoking and who want to speak to someone – I think that's always the key word – who want to – do the referral, do the warm hand-off.” (Behavioral Assistant Staff Interview). |
3. Results
3.1. Sample characteristics
The analytic sample included 251 smokers from site 1923 smokers from site 2, 763 smokers from site 3, and 1373 smokers from site 4 (Table 1). Patients from site 1 were on average older (67.1 ± 6.2 years old) and the majority were White (41.4%) or African American (34.7%), male (73.7%), and had the highest reporting of COPD (42.3%). Patients from site 2 were the most racially/ethnically diverse (32.0% African American, 27.5% Hispanic or Latinx, 24.3% White, and 11.2% Asian). Patients from site 3 were predominantly African American (72.2%) and had the highest percentage of women (44.8%). Patients from site 4 were majority African American (40.1%) and White (39.8%).
Table 1.
Demographic characteristics, smoking status transitions, and receipt of smoking cessation counseling and medications among current smokers by primary care clinic site.
| Variable | Site 1 (N = 251) |
Site 2 (N = 923) |
Site 3 (N = 763) |
Site 4 (N = 1373) |
Total (3310) |
χ2/F, p-value |
|---|---|---|---|---|---|---|
| Age, M ± SD | 67.1 ± 6.2 | 52.0 ± 12.7 | 51.3 ± 12.3 | 53.3 ± 9.9 | 53.5 ± 11.8 | 140.10, p < 0.001 |
| Sex, % female | 26.3% | 32.6% | 44.8% | 31.3% | 34.4% | 51.12, p < 0.001 |
| Race/ethnicitye | 550.72, p < 0.001 | |||||
| White | 41.4% | 24.3% | 6.9% | 39.8% | 28.0% | |
| African American | 34.7% | 32.0% | 72.2% | 40.1% | 44.8% | |
| Hispanic or Latinx | 8.4% | 27.5% | 12.6% | 12.4% | 16.3% | |
| Asian | 8.4% | 11.2% | 3.3% | 4.9% | 6.5% | |
| Other race/ethnicity | 4.4% | 3.1% | 3.8% | 2.2% | 3.0% | |
| Insurance type | 137.00, p < 0.001 | |||||
| Medicare | 33.5% | 20.2% | 17.8% | 19.8% | 20.5% | |
| MEDI-CAL | 61.0% | 59.7% | 67.1% | 70.9% | 66.2% | |
| Healthy worker/SF | 1.2% | 13.5% | 6.7% | 3.6% | 6.9% | |
| Other | 0.0% | 0.5% | 0.8% | 0.2% | 0.4% | |
| Uninsured | 4.4% | 6.1% | 7.6% | 5.5% | 6.1% | |
| Asthma | 3.3% | 6.2% | 11.6% | 10.0% | 8.7% | 36.37, p < 0.001 |
| COPD | 42.3% | 11.8% | 15.2% | 17.1% | 16.8% | 169.66, p < 0.001 |
| Depression | 32.2% | 30.3% | 23.6% | 35.1% | 30.7% | 39.94, p < 0.001 |
| Diabetes | 19.2% | 21.2% | 15.8% | 14.3% | 17.1% | 27.72, p < 0.001 |
| HIV | 1.6% | 0.2% | 6.8% | 13.9% | 7.1% | 229.22, p < 0.001 |
| Hypertension | 48.5% | 40.5% | 45.4% | 43.9% | 43.5% | 9.94, p = 0.02 |
| Ischemic vascular disease | 13.7% | 10.5% | 7.2% | 7.7% | 8.9% | 19.55, p < 0.001 |
| Recent smoking cessation attempta | 12.7% | 29.4% | 16.9% | 11.1% | 17.6% | 132.46, p < 0.001 |
| Medical assistant counselingb | 99.6% | 97.2% | 94.6% | 91.2% | 94.3% | 52.21, p < 0.001 |
| Any provider counselingb | 94.8% | 86.2% | 88.7% | 79.5% | 84.7% | 59.31, p < 0.001 |
| Behavioral Assistant counselingb | 21.9% | 1.7% | 5.8% | 3.9% | 5.1% | 172.58, p < 0.001 |
| Any cessation counselingc | 99.6% | 97.5% | 96.6% | 93.2% | 95.6% | 39.27, p < 0.001 |
| Prescribed NRT medicationd | 25.5% | 25.8% | 26.2% | 18.1% | 22.7% | 28.35, p < 0.001 |
| Prescribed non-NRT smoking cessation medicationd | 5.2% | 10.2% | 4.8% | 6.7% | 7.1% | 20.82, p < 0.001 |
Patients starting as current smokers who are 18 or older seen at one of 4 primary care clinics in San Francisco, CA from 7/1/2017 to 4/30/2017. Statistical comparison are between clinics. NRT = nicotine replacement therapy.
Transitioned from current smoker at the first visit to former smokers at the second or third visit during the study period.
Receipt of medical assistant (i.e. front-line staff), any provider (i.e. any primary or urgent care provider), or behavioral assistant (i.e. staff at the health coach level) cessation counseling at any of the 3 care visits during study time period.
Receipt of any cessation counseling (medical assistant, provider, or behavioral assistant) at any of the 3 care visits during the study time period.
Prescribed nicotine replacement therapy (NRT) or non-NRT cessation medication (bupropion or varenicline) at any of the 3 care visits during study time period.
Number do not add up to 100% due to 1.3% who declined to answer this question.
3.2. Recent smoking cessation attempts
Overall, the cumulative estimate of a recent cessation attempt was 17.6% (N = 584). Of the 584 patients, 33.8% (n = 198) had made a recent cessation attempt in visit 2, but relapsed to smoking by visit 3, and 66.1% (n = 386) attempted to quit at visit 3 or stayed quit between visit 2 and 3 (Table 1). Smoking patients who were misclassified as “never” smokers at visit 3 were reclassified as former smokers for the analyses (N = 209).
There was a significant difference in recent cessation attempts by sites, with all other sites having a lower proportion of patients making a recent cessation attempt compared to site 2, which had the highest proportion (Table 2). Patients who were older had higher odds of making a smoking cessation attempt (Adjusted odds ratio [AOR] = 1.01, 95% CI = 1.01,1.02 p = 0.006), Hispanic/Latinx patients had lower odds of making a smoking cessation attempt than Whites (AOR = 0.62, 95% CI = 0.47,0.81, p < 0.001), and patients who received medical assistant counseling (AOR = 1.78, 95% CI = 1.39,2.30, p < 0.001), or provider counseling (AOR = 1.45, 95% CI = 1.19,1.77, p < 0.001) had higher odds of making a recent cessation attempt.
Table 2.
Logistic regression model of factors associated with making a recent smoking cessation attempt.
| Variable | Recent smoking cessation attempta |
||
|---|---|---|---|
| AOR | 95% CI | p value | |
| Age | 1.01 | 1.00, 1.02 | 0.006 |
| Sex | |||
| Male (ref.) | – | – | – |
| Female | 1.04 | 0.85, 1.28 | 0.691 |
| Race/ethnicity | |||
| White (ref.) | – | – | – |
| African American | 0.98 | 0.76, 1.26 | 0.873 |
| Hispanic or Latinx | 0.62 | 0.47, 0.81 | <0.001 |
| Asian | 0.73 | 0.50, 1.06 | 0.100 |
| Other race/ethnicity | 0.73 | 0.43, 1.23 | 0.230 |
| Insurance type | |||
| Medicare (ref.) | – | – | – |
| MEDI-CAL | 1.24 | 0.97, 1.58 | 0.606 |
| Healthy Worker/SF | 0.94 | 0.66, 1.35 | 0.510 |
| Uninsured | 1.15 | 0.76, 1.74 | 0.748 |
| Other | 1.54 | 0.30, 7.86 | 0.092 |
| Clinic location | |||
| Site 4 (ref.) | – | – | – |
| Site 1 | 0.48 | 0.31, 0.76 | 0.002 |
| Site 2 | 0.31 | 0.24, 0.40 | <0.001 |
| Site 3 | 0.59 | 0.44, 0.78 | <0.001 |
| Primary care visit | |||
| Visit 2 (ref.) | – | – | – |
| Visit 3 | 0.84 | 0.73, 0.96 | 0.012 |
| Medical assistant counseling (ref. = no) | 1.78 | 1.39, 2.30 | <0.001 |
| Any provider counseling (ref. = no) | 1.45 | 1.19, 1.77 | <0.001 |
| Prescribed NRT (ref. = no) | 1.08 | 0.82, 1.42 | 0.818 |
| Prescribed non-NRT smoking cessation medication (ref. = no) | 0.91 | 0.56, 1.48 | 0.696 |
| Asthma (ref. = no) | 0.56 | 0.41, 0.75 | <0.001 |
| COPD (ref. = no) | 0.91 | 0.70, 1.19 | 0.507 |
| Depression (ref. = no) | 1.09 | 0.89, 1.33 | 0.431 |
| Diabetes (ref. = no) | 1.00 | 0.76, 1.30 | 0.974 |
| HIV (ref. = no) | 0.74 | 0.50, 1.08 | 0.121 |
| Hypertension (ref. = no) | 1.10 | 0.89, 1.35 | 0.396 |
| Ischemic vascular disease (ref. = no) | 0.90 | 0.65, 1.24 | 0.514 |
Patients 18 or older seen at one of 4 primary care clinics in San Francisco, CA from 7/1/2016 to 4/30/2017. NRT = nicotine replacement therapy, Ref = reference category, AOR = adjusted odds ratio, CI = confidence interval, bold indicated p < 0.05.
Recent smoking cessation attempt during the study duration, defined as a transition in smoking status from current smoker at the first visit to former smoker at the 2nd or 3rd visits during study time period.
3.3. Receipt of smoking cessation counseling and pharmacotherapy
There was an increase in the proportion of patients who received medical assistant and any provider cessation counseling over the study duration, but there were marginal increases in the already low rates for behavioral assistant counseling (Fig. 1). Overall, the cumulative proportions for receipt of any medical assistant counseling was 94.3%, for any provider cessation counseling was 84.7%, and behavioral assistant counseling was 5.1% during the study time period (Table 1). Of the smokers, 22.7% were prescribed NRT and 7.1% of patients were prescribed a non-NRT medication. There were significant differences between the clinic sites for receipt of smoking cessation counseling and pharmacotherapy.
Fig. 1.
Receipt of medical assistant, provider, and behavioral assistant smoking cessation counseling among current smokers at the first, second, and third visits (July 2017–April 2017).
3.4. Factors associated with receipt of any cessation counseling and any provider counseling
The majority of demographic characteristics were not associated with receipt of any cessation counseling or any provider cessation counseling (Table 3). Older smokers were more likely to have received any cessation counseling (AOR = 1.01,95% CI = 1.04,1.02, p = 0.004) or provider cessation counseling (AOR = 1.01,95% CI = 1.00,1.01, p = 0.046) compared to younger smokers. Compared to patients on Medicare, uninsured patients (AOR = 0.61,95% CI = 0.43,0.86, p = 0.005) were less likely to have received any smoking cessation counseling while patients on Medi-Cal (AOR = 0.82,95% CI = 0.69,0.99, p = 0.034) were less likely to have received provider cessation counseling. Compared to patients from site 4 (reference group), patients receiving care from all other sites were more likely to have received any cessation counseling and any provider counseling. Smokers with COPD, depression, diabetes, HIV, or hypertension were more likely to have received any cessation counseling and any provider counseling (Table 3).
Table 3.
Logistic regression models of factors associated with receipt of smoking cessation counseling during primary care visits.
| Variable | Any counselinga |
Any provider counselingb |
||||
|---|---|---|---|---|---|---|
| AOR | 95% CI | p value | AOR | 95% CI | p value | |
| Age | 1.01 | 1.04, 1.02 | 0.004 | 1.01 | 1.00, 1.01 | 0.046 |
| Sex | ||||||
| Male (ref.) | – | – | – | – | – | – |
| Female | 1.03 | 0.87, 1.23 | 0.708 | 1.07 | 0.93, 1.23 | 0.357 |
| Race/ethnicity | ||||||
| White (ref.) | – | – | – | – | – | – |
| African American | 1.05 | 0.86, 1.28 | 0.662 | 0.92 | 0.78, 1.09 | 0.343 |
| Hispanic or Latinx | 0.87 | 0.68, 1.11 | 0.254 | 0.86 | 0.70, 1.06 | 0.167 |
| Asian | 1.04 | 0.73, 1.49 | 0.820 | 0.84 | 0.63, 1.13 | 0.254 |
| Other race/ethnicity | 0.96 | 0.59, 1.58 | 0.881 | 0.95 | 0.62, 1.45 | 0.814 |
| Insurance type | ||||||
| Medicare (ref.) | – | – | – | – | – | – |
| MEDI-CAL | 0.90 | 0.72, 1.11 | 0.320 | 0.82 | 0.69, 0.99 | 0.034 |
| Healthy worker/SF | 0.75 | 0.53, 1.06 | 0.105 | 0.75 | 0.56, 1.02 | 0.064 |
| Uninsured | 0.61 | 0.43, 0.86 | 0.005 | 0.74 | 0.54, 1.02 | 0.065 |
| Other | 0.55 | 0.19, 1.55 | 0.255 | 0.50 | 0.21, 1.23 | 0.131 |
| Clinic location | ||||||
| Site 4 (ref.) | – | – | – | – | – | – |
| Site 1 | 5.82 | 3.56, 9.53 | <0.001 | 2.26 | 1.68, 3.03 | <0.001 |
| Site 2 | 3.16 | 2.58, 3.89 | <0.001 | 2.27 | 1.91, 2.70 | <0.001 |
| Site 3 | 1.73 | 1.40, 2.13 | <0.001 | 1.59 | 1.33, 1.90 | <0.001 |
| Primary care visit | ||||||
| Visit 1 (ref.) | – | – | – | – | – | – |
| Visit 2 | 2.41 | 2.24, 2.60 | <0.001 | 2.04 | 1.93, 2.16 | <0.001 |
| Visit 3 | 10.72 | 9.11, 12.61 | <0.001 | 6.70 | 6.09, 7.38 | <0.001 |
| Asthma (ref. = no) | 1.18 | 0.89, 1.56 | 0.272 | 1.17 | 0.92, 1.50 | 0.209 |
| COPD (ref. = no) | 1.32 | 1.04, 1.67 | 0.025 | 1.22 | 1.01, 1.47 | 0.041 |
| Depression (ref. = no) | 1.26 | 1.06, 1.49 | 0.009 | 1.21 | 1.05, 1.40 | 0.010 |
| Diabetes (ref. = no) | 1.52 | 1.21., 1.93 | <0.001 | 1.46 | 1.22., 1.77 | <0.001 |
| HIV (ref. = no) | 2.16 | 1.59, 2.94 | <0.001 | 2.03 | 1.55, 2.66 | <0.001 |
| Hypertension (ref. = no) | 1.40 | 1.17, 1.67 | <0.001 | 1.26 | 1.09, 1.47 | 0.002 |
| Ischemic vascular disease (ref. = no) | 1.47 | 1.06, 2.04 | 0.020 | 0.99 | 0.78, 1.26 | 0.940 |
Patients 18 or older seen at one of 4 primary care clinics in San Francisco, CA from 7/1/2016 to 4/30/2017. Ref = reference category, AOR = adjusted odds ratio, CI = confidence interval, bold indicated p < 0.05.
Receipt of any smoking cessation counseling: medical assistant (i.e. front-line staff), any provider (i.e. any primary or urgent care provider), or behavioral assistant (i.e. staff at the health coach level) cessation counseling during study time period.
Receipt of any provider (i.e. any primary or urgent care provider) smoking cessation counseling during study time period.
3.5. Qualitative findings
Interviews focused on identifying predisposing, enabling, and reinforcing factors associated with receipt of or delivery of cessation services among patients and staff, respectively.
3.5.1. Patient-level predisposing factors
3.5.1.1. Barriers to access to treatment
Patients reported several barriers to access to treatment, including challenges with obtaining medications for cessation and lack of access to on-site cessation counseling (Table 4). Patients reported that being unstably housed, having physical disabilities, or being unable to obtain transportation posed barriers to attending counseling sessions in sites that were not co-located in their primary clinics. The consensus was that if services were offered on-site, including the provision of NRT, it would substantially reduce barriers to receiving smoking cessation care.
3.5.1.2. Challenges with substance use and mental illness
Substance use and mental illness were barriers to successful smoking cessation; lack of concurrent treatment for substance use and nicotine dependence made it harder to address nicotine addiction. Several patients described experiences with forced quit attempts during extended hospital stays or incarceration. Patients reported smoking in order to cope with anxiety or the stressors of homelessness, and some substituted cigarettes for other illicit substances as a method of harm reduction.
3.5.2. Patient-level enabling factors
3.5.2.1. Access to treatment
Patients preferred on-site access to pharmacist-delivered cessation medications in conjunction with a smoking cessation group. This model of cessation treatment, the primary model in site 4, was perceived to be successful because it eliminated barriers to access to treatment. However, this model was not feasible in other clinics because they did not have pharmacists on site who were available to prescribe NRT, explaining the overall low rates of prescribing of cessation medications. Patients also preferred counseling that took place on an on-going basis instead of a structured class for a finite time.
3.5.3. Provider-level predisposing factors
3.5.3.1. Competing priorities
Clinics used the “warm hand-off” model where medical assistants and providers referred patients to behavioral assistants for smoking cessation counseling during the clinical encounter. However, behavioral assistants were often occupied with other tasks during the hand-off, resulting in the rescheduling of most of the counseling sessions with few patients actually following up. This, in part, explained the low rates of utilization of behavioral assistants counseling. Behavioral assistants also reported that in a busy clinic with many patients with mental health needs where cigarette smoking was also a comorbidity, counseling all smokers using the warm hand-off model was not feasible – “If everybody that smoked got referred, there is no way that I could see everybody”. Behavioral assistants described the need to risk-stratify patients to determine who would benefit most from on-site counseling versus referrals to outside sources.
3.5.3.2. Job descriptions and ownership of counseling
Most behavioral assistants and medical assistants reported that they facilitated cessation by providing referrals – “I'll also offer a referral to resources like 1-800-nobutts [because] I am limited in what I can provide in terms of cessation” – and believed that the essential components of smoking cessation counseling should be initiated by the provider. Staff reported that in settings where there were multiple resources for cessation such as group counseling versus one-on-one counseling by behavioral assistants, there was no coordination around the use of these resources. For instance, one behavioral assistant asked why patients should have to see her for smoking cessation “when they could go upstairs to attend the group [counseling]”.
3.5.4. Provider-level reinforcing factors
3.5.4.1. Trainings for staff to provide cessation counseling
Several staff highlighted that they were inadequately trained to provide cessation counseling and described a need for annual trainings to refresh their knowledge on providing smoking cessation counseling. Medical assistants reported that the lack of a standardized script to screen for and document cigarette smoking status in the EHR was confusing for staff, as some staff offered more counseling than others, and some incorrectly documented smoking status in the EHR.
3.5.5. Provider-level enabling factors
3.5.5.1. Cessation leadership
Most staff reported that having clinic-level leadership support and having an on-site smoking cessation champion increased awareness of cessation resources among patients and support for staff to provide cessation counseling. For instance, site 4 had a registered nurse who was the local cessation champion and who led a well-attended weekly cessation group for interested patients.
3.5.5.2. Better communication among clients and staff
There was consensus among staff that there was inadequate signage around smoking cessation in clinics and/or information around smoking cessation resources for patients. One of the behavioral assistants from site 1 reported using a huddle model, where staff and providers met before clinic to discuss announcements, updates to practice, and tasks for scheduled patients for the day, including discussing the options for referring to resources for cessation. This practice was not common in the other three sites. A behavioral assistant also mentioned that their clinic had regular quality improvement meetings that offered a forum to discuss updates, problem-solve, and brainstorm new ideas around smoking cessation.
4. Discussion
In this study, we examined receipt of cessation services and recent smoking cessation attempts among 3301 smokers engaged in primary care in four SFHN clinics. We found that 17.6% of patients attempted a recent cessation attempt, with a 37% lower cessation attempt rate among patients seen in the clinic that served predominantly homeless patients. Over 95% of patients received some form of smoking cessation counseling from either a medical assistant, behavioral assistant, or provider, and less than one-fourth received medications for cessation. Patients with medical and psychiatric comorbidities were more likely to have received smoking cessation counseling. Receipt of any cessation counseling or provider counseling were associated with greater odds of patients making a recent smoking cessation attempt, supporting prior studies that showed that counseling is associated with an increase in cessation attempts (Fiore et al., 2008). The recent smoking cessation attempt rate observed in our study is higher than the population smoking cessation attempt rate among persons quitting without assistance (~4%) (Pierce et al., 2012), but lower than patients enrolled in clinical trials of behavioral counseling and pharmacotherapy (~30%–40%) (Fiore and Baker, 2011). Through the triangulation of the EHR and qualitative data, we identified several potential avenues for intervention including increasing access to cessation medications and higher intensity counseling by behavioral assistants and providers.
We found significant differences in recent smoking cessation attempts and receipt of cessation services among the four primary care clinics, which had notable differences in the sociodemographic characteristics of the patient populations. The clinics that served predominantly African American patients (site 3), homeless patients (site 4), and low-income elderly patients (site 1) had fewer individuals attempting to quit smoking compared to patients seen in the academic safety-net clinic (site 2). These disparities in recent smoking cessation attempts existed despite the fact that site 1, which served an older population, was more likely to systematically address cigarette smoking among patients at all encounters (95.2% for any cessation counseling and 10.0% for behavioral assistant counseling). Elderly individuals (Messer et al., 2008), those experiencing homelessness (Vijayaraghavan et al., 2014; Vijayaraghavan et al., 2016), and African American patient populations (Trinidad et al., 2011; MMWR, 2011) have among the lowest rates of successful quitting despite making quit attempts at the same rate as the general population. Academic clinic practices, have ongoing training of health care providers using a QI framework, which may provide an avenue to continuously improve metrics for receipt of smoking cessation counseling services (Kruse et al., 2012; Kruse et al., 2013). Findings suggest that certain low-income populations within our safety-net clinics may benefit from targeted intensive cessation efforts, as highlighted in the qualitative interviews where participants preferred on-site and ongoing cessation counseling groups over remote counseling options. QI activities could be used to drive improvements to reduce disparities through the addition of targeted interventions among certain disparity populations (e.g. African Americans) (Webb Hooper et al., 2018) or ensuring higher intensity services (e.g. on-site groups) in clinics that serve these patient populations.
Hispanic/Latinx patients were less likely to make a smoking cessation attempt during the study period than non-Hispanic White patients. Hispanic/Latinx smokers tend to smoke fewer cigarettes per day and are more likely to be non-daily smokers compared to non-Hispanic White smokers (Trinidad et al., 2009). Despite being lighter smokers in general, our data suggests that additional measures are needed to promote smoking cessation among this group.
Although a team-based approach to cessation counseling is a strength of our health system and builds upon previous models of smoking cessation service delivery (Kruse et al., 2012), we identified potential pitfalls in its implementation. Our clinic system had invested substantial resources to train behavioral assistants to provide on-site smoking cessation counseling using a warm hand-off model (Rigotti et al., 2011), yet this resource was substantially under-utilized during the study time period, for an overall rate of use of only 5.1%. According to the behavioral assistants interviewed, the warm hand-off model yielded low engagement in practice because behavioral assistants were often occupied with other tasks during the hand-off, resulting in the rescheduling of smoking cessation counseling and few patients actually following up. However, our findings also highlighted potential opportunities to improve utilization of this counseling, including workforce development activities that would support the behavioral assistants to become local smoking cessation champions, educating other team members on the role of behavioral assistants in providing team-based cessation care, and using clinic huddles to increase awareness about on-site smoking cessation services.
There are several limitations to our study. Recent cessation attempts are not an indicator of successful quitting because the vast majority of smokers who attempt to quit subsequently relapse to smoking (Pierce et al., 1998). However, this data provides a cross-sectional view of quitting behavior and receipt of cessation services, which are useful to assess the immediate impact of newly-implemented interventions. EHR data relied on patient self-report, and smoking status was not biochemically verified, leading to a potential for misclassification bias. By excluding people with missing data in the EHR analysis, we may have introduced some bias (Weber et al., 2017) and for this reason our sample may not be representative of all patients seen in these clinics. Collecting more detailed information on age of smoking initiation, quit date, or number of cigarettes smoker per day may allow for better classification and validation of smoking status. Updating and documenting smoking status consistently in the EHR, along with maintaining a quality improvement assessment team to evaluate this process can help to improve the quality of EHR smoking-related data (Spencer et al., 1999). Qualitative data was not intended to be generalizable to patients and staff in the participating clinics; however, its triangulation with the EHR data provided explanations for some of the observed patterns.
In conclusion, the EHR can be used for rapid cycle evaluations of cessation QI activities. While our study demonstrated one application of the EHR to provide a snapshot of recent cessation attempts among patients engaged in clinical care, there are other potential uses of EHR data to provide longitudinal, patient-specific estimates of successful quitting for clinic populations as well as trends in smoking prevalence and cessation over time (Atkinson et al., 2017; Polubriaginof et al., 2017; Taggar et al., 2012). This model of evaluation is important to improve reach and efficacy of smoking cessation services to patient populations with a high smoking prevalence and burden of tobacco related disease.
Role of funding source
This work was supported by San Francisco Cancer Initiative (SFCAN 16-19347) and by the California Tobacco Related-Disease Research Program (TRDRP 25CP-0002). The preparation of this manuscript was supported by the NIH fellowship F32 DA 042554. These funding sources had no role in the analysis or interpretation of the data, writing of the manuscript, or the decision to submit the paper for publication. The content is solely the responsibility of the authors and does not necessarily represent the official views of any funding source.
References
- Atkinson M.D., Kennedy J.I., John A., Lewis K.E., Lyons R.A., Brophy S.T., Group, D. R Development of an algorithm for determining smoking status and behaviour over the life course from UK electronic primary care records. BMC Med Inform Decis Mak. 2017;17(1):2. doi: 10.1186/s12911-016-0400-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Blumenthal D., Tavenner M. The "meaningful use" regulation for electronic health records. N. Engl. J. Med. 2010;363(6):501–504. doi: 10.1056/NEJMp1006114. [DOI] [PubMed] [Google Scholar]
- (2000). In M. Ein Lewin & S. Altman (Eds.), Americas's Health Care Safety Net: Intact but Endangered. Washington (DC). [PubMed]
- Fiore M.C., Baker T.B. Clinical practice. Treating smokers in the health care setting. N. Engl. J. Med. 2011;365(13):1222–1231. doi: 10.1056/NEJMcp1101512. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Fiore M.C., Jaen C.R., Baker T.B., Bailey W.C., Benowitz N.L., Curry S.J., Staff, P. G. U Treating tobacco use and dependence: 2008 update US public health service clinical practice guideline executive summary. Respir. Care. 2008;53(9):1217–1222. [PubMed] [Google Scholar]
- Flocke S.A., Hoffman R., Eberth J.M., Park H., Birkby G., Trapl E., Zeliadt S. The prevalence of tobacco use at federally qualified health centers in the United States, 2013. Preventing Chronic Disease. 2017;14 doi: 10.5888/pcd14.160510. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Green L.W., Kreuter M.W. Health Program Planning: An Educational and Ecological Approach New York: McGraw-Hill. 2005:2005. [Google Scholar]
- Guydish J., Tajima B., Pramod S., Le T., Gubner N.R., Campbell B., Roman P. Use of multiple tobacco products in a national sample of persons enrolled in addiction treatment. Drug Alcohol Depend. 2016;166:93–99. doi: 10.1016/j.drugalcdep.2016.06.035. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Hsieh H.-F., Shannon S.E. Three approaches to qualitative content analysis. Qual. Health Res. 2005;15(9):1277–1288. doi: 10.1177/1049732305276687. [DOI] [PubMed] [Google Scholar]
- Jamal A., P.E., Gentzke A.S. Current cigarette smoking among adults—United States, 2016. MMWR Morb. Mortal. Wkly Rep. 2018;67:53–59. doi: 10.15585/mmwr.mm6702a1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Kruse G.R., Kelley J.H., Linder J.A., Park E.R., Rigotti N.A. Implementation of an electronic health record-based care management system to improve tobacco treatment. J. Gen. Intern. Med. 2012;27(12):1690–1696. doi: 10.1007/s11606-012-2174-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Kruse G.R., Chang Y., Kelley J.H., Linder J.A., Einbinder J.S., Rigotti N.A. Healthcare system effects of pay-for-performance for smoking status documentation. Am. J. Manag. Care. 2013;19(7):554–561. [PMC free article] [PubMed] [Google Scholar]
- Messer K., Trinidad D.R., Al-Delaimy W.K., Pierce J.P. Smoking cessation rates in the United States: a comparison of young adult and older smokers. Am. J. Public Health. 2008;98(2):317–322. doi: 10.2105/AJPH.2007.112060. [DOI] [PMC free article] [PubMed] [Google Scholar]
- MMWR Quitting smoking among adults—United States, 2001–2010. Centers for Disease Control and Prevention (CDC) 2011;60(44):1513–1519. [PubMed] [Google Scholar]
- Nguyen O.K., Makam A.N., Halm E.A. National use of safety-net clinics for primary care among adults with non-Medicaid Insurance in the United States. PLoS One. 2016;11(3) doi: 10.1371/journal.pone.0151610. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Pierce J.P., Farkas A.J., Gilpin E.A. Beyond stages of change: the quitting continuum measures progress towards successful smoking cessation. Addiction. 1998;93(2):277–286. doi: 10.1046/j.1360-0443.1998.93227711.x. [DOI] [PubMed] [Google Scholar]
- Pierce J.P., Cummins S.E., White M.M., Humphrey A., Messer K. 2012. Quitlines and Nicotine Replacement for Smoking Cessation: Do we Need to Change Policy? Annual Review of Public Health, Vol 33, 33, 341−+ [DOI] [PubMed] [Google Scholar]
- Pillemer F., Price R.A., Paone S., Martich G.D., Albert S., Haidari L.…Mehrotra A. Direct release of test results to patients increases patient engagement and utilization of care. PLoS One. 2016;11(6) doi: 10.1371/journal.pone.0154743. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Polubriaginof F., Salmasian H., Albert D.A., Vawdrey D.K. Challenges with collecting smoking status in electronic health records. AMIA Annu Symp Proc. 2017;2017:1392–1400. [PMC free article] [PubMed] [Google Scholar]
- Rigotti N.A., Bitton A., Kelley J.K., Hoeppner B.B., Levy D.E., Mort E. Offering population-based tobacco treatment in a healthcare setting: a randomized controlled trial. Am J Prev Med. 2011;41(5):498–503. doi: 10.1016/j.amepre.2011.07.022. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Rizzato Lede D.A., Benitez S.E., Mayan J.C., 3rd, Smith M.I., Baum A.J., Luna D.R., Bernaldo de Quiros F.G. Patient safety at transitions of care: use of a compulsory electronic reconciliation tool in an academic hospital. Stud Health Technol Inform. 2015;216:232–236. [PubMed] [Google Scholar]
- Schroeder S.A., Morris C.D. Confronting a neglected epidemic: tobacco cessation for persons with mental illnesses and substance abuse problems. Annu. Rev. Public Health. 2010;31(31):297–314. doi: 10.1146/annurev.publhealth.012809.103701. [DOI] [PubMed] [Google Scholar]
- Spencer E., Swanson T., Hueston W.J., Edberg D.L. Tools to improve documentation of smoking status. Continuous quality improvement and electronic medical records. Arch. Fam. Med. 1999;8(1):18–22. doi: 10.1001/archfami.8.1.18. [DOI] [PubMed] [Google Scholar]
- Taggar J.S., Coleman T., Lewis S., Szatkowski L. The impact of the quality and outcomes framework (QOF) on the recording of smoking targets in primary care medical records: Cross-sectional analyses from the health improvement network (THIN) database. Bmc Public Health. 2012;12 doi: 10.1186/1471-2458-12-329. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Trinidad D.R., Perez-Stable E.J., Emery S.L., White M.M., Grana R.A., Messer K.S. Intermittent and light daily smoking across racial/ethnic groups in the United States. Nicotine Tob. Res. 2009;11(2):203–210. doi: 10.1093/ntr/ntn018. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Trinidad D.R., Perez-Stable E.J., White M.M., Emery S.L., Messer K. A Nationwide analysis of US racial/ethnic disparities in smoking behaviors, smoking cessation, and cessation-related factors. Am. J. Public Health. 2011;101(4):699–706. doi: 10.2105/AJPH.2010.191668. [DOI] [PMC free article] [PubMed] [Google Scholar]
- U.S. Department of Health and Human Services . Atlanta (GA) 2014. The health consequences of Smoking-50 years of Progress: a report of the surgeon general. [Google Scholar]
- Vijayaraghavan M., Penko J., Vittinghoff E., Bangsberg D.R., Miaskowski C., Kushel M.B. Smoking behaviors in a community-based cohort of HIV-infected indigent adults. AIDS Behav. 2014;18(3):535–543. doi: 10.1007/s10461-013-0576-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Vijayaraghavan M., Tieu L., Ponath C., Guzman D., Kushel M. Tobacco cessation behaviors among older homeless adults: results from the HOPE HOME study. Nicotine Tob. Res. 2016;18(8):1733–1739. doi: 10.1093/ntr/ntw040. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Vijayaraghavan M., Yuan P., Gregorich S., Lum P., Appelle N., Napoles A.M.…Satterfield J. Disparities in receipt of 5As for smoking cessation in diverse primary care and HIV clinics. Prev. Med. Rep. 2017;6:80–87. doi: 10.1016/j.pmedr.2017.02.012. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Webb Hooper M., Carpenter K., Payne M., Resnicow K. Effects of a culturally specific tobacco cessation intervention among African American Quitline enrollees: a randomized controlled trial. BMC Public Health. 2018;18(1):123. doi: 10.1186/s12889-017-5015-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Weber G.M., Adams W.G., Bernstam E.V., Bickel J.P., Fox K.P., Marsolo K., Mandl K.D. Biases introduced by filtering electronic health records for patients with “complete data”. J. Am. Med. Inform. Assoc. 2017;24(6):1134–1141. doi: 10.1093/jamia/ocx071. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Zhu, S. H., Anderson, C. M., Tedeschi, G. J., Rosbrook, B., Johnson, C. E., Byrd, M., & Gutierrez-Terrell, E. (2002). Evidence of real-world effectiveness of a telephone quitline for smokers. New England Journal of Medicine, 347(14), 1087–1093. doi:DOI 10.1056/NEJMsa020660. [DOI] [PubMed]

