Skip to main content
BMJ Open logoLink to BMJ Open
. 2020 Jul 14;10(7):e034983. doi: 10.1136/bmjopen-2019-034983

Primary care physicians’ attitudes and perceptions towards antibiotic resistance and outpatient antibiotic stewardship in the USA: a qualitative study

Rachel M Zetts 1,, Andrea Stoesz 1, Andrea M Garcia 2, Jason N Doctor 3, Jeffrey S Gerber 4, Jeffrey A Linder 5, David Y Hyun 1
PMCID: PMC7365421  PMID: 32665343

Abstract

Objectives

At least 30% of outpatient antibiotic prescriptions are unnecessary. Outpatient antibiotic stewardship is needed to improve prescribing and address the threat of antibiotic resistance. A better understanding of primary care physicians (PCPs) attitudes towards antibiotic prescribing and outpatient antibiotic stewardship is needed to identify barriers to stewardship implementation and help tailor stewardship strategies. The aim of this study was to assess PCPs current attitudes towards antibiotic resistance, inappropriate antibiotic prescribing and the feasibility of outpatient stewardship efforts.

Design

Eight focus groups with PCPs were conducted by an independent moderator using a moderator guide. Focus groups were audio recorded, transcribed and coded for major themes using deductive and inductive content analysis methods.

Setting

Focus groups were conducted in four US cities: Philadelphia, Birmingham, Chicago and Los Angeles.

Participants

Two focus groups were conducted in each city—one with family medicine and internal medicine physicians and one with paediatricians. A total of 26 family medicine/internal medicine physicians and 26 paediatricians participated.

Results

Participants acknowledged that resistance is an important public health issue, but not as important as other pressing problems (eg, obesity, opioids). Many considered resistance to be more of a hospital issue. While participants recognised inappropriate prescribing as a problem in outpatient settings, many felt that the key drivers were non-primary care settings (eg, urgent care clinics, retail clinics) and patient demand. Participants reacted positively to stewardship efforts aimed at educating patients and clinicians. They questioned the validity of antibiotic prescribing metrics. This scepticism was due to a number of factors, including the feasibility of capturing prescribing quality, a belief that physicians will ‘game the system’ to improve their measures, and dissatisfaction and distrust of quality measurement in general.

Conclusions

Stakeholders will need to consider physician attitudes and beliefs about antibiotic stewardship when implementing interventions aimed at improving prescribing.

Keywords: quality in health care, infectious diseases, primary care, public health, qualitative research


Strengths and limitations of this study.

  • This study presents new data on US-based primary care physicians attitudes towards antibiotic resistance, inappropriate antibiotic prescribing and outpatient antibiotic stewardship approaches.

  • Eight focus groups with internal medicine physicians, family medicine physicians and paediatricians were held in four geographically dispersed US cities, which allowed for a wide-range of viewpoints to be represented in the dataset.

  • The focus groups did not include some types of clinicians that provide primary care in the USA (eg, nurse practitioners, physician assistants).

  • Although physicians from across the USA were included in this study, the small sample size limits the generalisability of these findings.

Introduction

Antibiotic resistance poses a growing threat to public health and antibiotic use is a primary driver of the development of resistant bacteria. In the USA, the majority of antibiotics used in humans are prescribed in outpatient healthcare settings.1 Considering the volume of antibiotics prescribed and data from other countries, ambulatory antibiotic prescribing likely accounts for 80%–90% of all antibiotic prescribing.2 3

There were 270.2 million outpatient antibiotic prescriptions dispensed in 2016.4 While this represents a 5% decrease since 2011, prescribing rates have been relatively stable from 2014 to 2016.4 Previous studies have found that a significant proportion of outpatient antibiotic prescriptions are inappropriate.5–9 Many of these inappropriate prescriptions were for acute respiratory conditions that often do not require antibiotics.5 7 8

In order to improve antibiotic prescribing in primary care offices and other outpatient healthcare settings, the Centers for Disease Control and Prevention (CDC) published core elements of outpatient antibiotic stewardship, highlighting steps that stakeholders can take in support of stewardship efforts.10 However, additional work is needed to ensure outpatient stewardship efforts are expanded nationwide. A better understanding of physicians attitudes towards antibiotic prescribing and their perceptions on the feasibility and impact of stewardship interventions would identify barriers to stewardship implementation in US ambulatory settings and would allow stakeholders to better tailor strategies to improve prescribing.

Previous research has evaluated what drives inappropriate antibiotic prescribing in healthcare settings. These studies, although evaluating physician populations in differing countries, have found some consistent themes. Many studies, including those from the USA and Europe, have shown that physicians consider patient demand and prescribing of other physicians to be primary drivers of inappropriate antibiotic prescribing.11–25 In order to counteract these drivers of inappropriate prescribing, physicians have highlighted a need for improved public education around antibiotic resistance and the need for appropriate prescribing.20–23 26 Additionally, studies from USA and European countries have shown that physician time constraints, fear of undertreating patients due to diagnostic uncertainty, and certain clinical factors are also seen as drivers of inappropriate prescribing.11–13 15–17 19–21 23 24 27 28

Studies have also found that physicians do not necessarily see the impact of antibiotic resistance in their daily practice. Two interview-based studies of primary care clinicians in the UK and Europe showed a general recognition that antibiotic resistance is an important issue, but many were less concerned about resistance in their daily practice.29 30 A systematic review of studies from different countries found a similar dynamic.26 However, a US-based qualitative study did find that while physicians did not commonly mention antibiotic resistance as a factor when making prescribing decisions, some did express concerns about the availability of antibiotics in the future.11

Expanding on this research to gain a better understanding of current attitudes about antibiotic prescribing and the perceived impact of different antibiotic stewardship approaches among US outpatient physicians is needed. This is especially true for primary care physicians (PCPs) given that they account for the largest proportion of outpatient antibiotic prescriptions (38%) in the USA.31

In the USA, primary care services are often provided by family medicine physicians, internists and paediatricians.32 The provision of these services can be fragmented. Many patients do not receive extended primary care services and after-hours care from their usual primary care offices.33 Additionally, PCPs in the USA receive payment for their services from a range of commercial and public payers,32 all of which frequently measure the quality of care to determine reimbursement levels. All of these factors have the potential to influence U.S. PCPs views on antibiotic prescribing and approaches to improving antibiotic use in outpatient settings.

In order to assess these attitudes and perceptions and inform strategies for antibiotic stewardship tailored to US outpatient settings, we conducted a series of semistructured focus groups among PCPs in the USA.

Methods

Study design

We conducted eight focus groups in November and December of 2017 with PCPs in four US cities—Philadelphia, Pennsylvania; Birmingham, Alabama; Chicago, Illinois and Los Angeles, California. Focus groups were chosen for this study to allow for open discussion among participants and to allow for different opinions and debate. This allows both for the identification of areas where there is dissension and broad consensus during the analysis process, and adds further complexity to the themes.

The four cities were selected to represent each of the four US Census regions in order to account for any potential differences in attitudes based on geographical region. Research has shown a clear difference in overall outpatient antibiotic prescribing rates by geographic region in the USA, with individuals in Southern states prescribed antibiotics at higher rates than those in any other part of the country.4 31 For example, in 2017, the antibiotic prescribing rate in West Virginia (the state with the highest rate) was more than double that of Alaska (the state with the lowest rate).31 Additionally, many PCPs in the USA specialise in family medicine, internal medicine or paediatrics.32 As such, two focus groups were conducted in each city—one with family medicine and internal medicine physicians and one with paediatricians.

A screening questionnaire was developed to recruit participants. Inclusion criteria included self-report of board certification in paediatrics, family medicine or internal medicine; being a full-time physician primarily practising in an outpatient office setting; spending >50% of medical practice time in direct patient care; and fluency in English. Participants were excluded if they reported being >65 years old; board certified in a subspecialty outside of primary care; or an employee or paid consultant of any of the following organisations: a pharmaceutical, medical device or biotechnology company, an advertising or healthcare marketing company or a governmental or regulatory agency.

Study participants were recruited by M3 Global Research, a medical market research firm. Participants were initially recruited from a panel of healthcare professionals maintained by M3. For three cities—Chicago, Birmingham and Los Angeles—additional participants were recruited from physician panels maintained by local partners to ensure adequate participation. Individuals located within a 30 mile radius of each focus group facility were contacted by telephone or online and screened for participation in this study. Any participant recruited online received a follow-up call from M3 to confirm their eligibility.

Each focus group lasted between 1.5 and 2 hours and was moderated by the same independent moderator with experience in qualitative research. Prior to each focus group, participants received an informed consent form to review and sign. All participants signed the informed consent form. Each participant received US$400 to compensate for their time.

The Standards for Reporting Qualitative Research reporting guidelines were used in the reporting of study findings (see online supplementary file 1).34

Supplementary data

bmjopen-2019-034983supp001.pdf (32.3KB, pdf)

Data collection and analysis

The study team and the external moderator developed a semistructured moderator guide (see online supplementary file 2). This guide aimed to draw out issues identified based on previous research—such as perceptions of antibiotic resistance and drivers of inappropriate prescribing, including patient demand—as well as explore new areas, such as the perceived impact of different stewardship strategies. The guide began by asking participants to rank a number of public health issues in terms of importance in their daily practice. These issues included excess body weight and obesity, antibiotic resistance, misinformation about childhood vaccines (paediatricians only), opioid abuse, diabetes, patient non-compliance with drug regimens and smoking/tobacco use.

Supplementary data

bmjopen-2019-034983supp002.pdf (162.8KB, pdf)

The guide then asked questions aimed at understanding the physicians’ attitudes and perceptions around antibiotic use and stewardship, including factors that influence their antibiotic prescribing decisions and if/how they communicate with patients about these decisions. They were also given handouts that defined and provided examples of the CDC’s Core Elements of Outpatient Antibiotic Stewardship.10 These handouts were used to gauge perceptions on the feasibility and impact of the core elements and associated activities.

Finally, participants were asked for feedback on activities that encourage antibiotic stewardship implementation and resource availability to do so. Respondents provided opinions on the feasibility and effectiveness of example policies and activities that could be implemented by healthcare stakeholders to encourage stewardship implementation. To assess resource availability, participants were asked to provide feedback on current access to certain tools to support antibiotic stewardship efforts, such as feedback reports on antibiotic prescribing practices or access to patient education materials and, if not, how much of a burden it would be to obtain access.

All focus groups were audio and video recorded, transcribed (using the audio recording) and the transcripts were coded for major themes in NVivo V.11 (QSR International). Common themes were identified by three study authors (RMZ, AS and DYH), using both deductive and inductive content analysis methods.35 36 We applied the following steps for analysing the transcripts. First, researchers (RMZ, DYH) familiarised themselves with the data by observing all eight focus groups. Next, an initial list of themes was developed based on (1) a review of past studies on the topic of antibiotic resistance and stewardship in outpatient settings11–18 37 38 and (2) the data familiarisation process. These themes were independently applied to the transcripts and coded by two authors (RMZ, AS) and reviewed by another author (DYH). During this process, new themes were identified through further review of the transcripts and some of the initial themes were modified. Any disagreement in coding was discussed until consensus was met. Coding was considered complete once theoretical saturation was reached and no additional themes could be identified.36

Patient and public involvement

This research was done without patient involvement. Patients were not invited to comment on the study design and were not consulted to develop patient relevant outcomes or interpret the results. Patients were not invited to contribute to the writing or editing of this document for readability or accuracy.

Results

A total of 52 PCPs—26 family medicine and internal medicine physicians and 26 paediatricians—accepted the invitation and participated in the focus groups. No demographic information was collected for these participants.

A number of common themes were identified across these focus groups that illustrated attitudes on the following topics: (1) antibiotic resistance as a public health issue, (2) drivers of antibiotic prescribing, (3) the acceptability of antibiotic stewardship interventions—patient and physician education, and (4) acceptability of performance reporting. Themes within each of these areas are highlighted below, along with areas of disagreement among participants where appropriate.

Antibiotic resistance as a public health issue

The initial discussions within each focus group centred on what participants thought about antibiotic resistance as a public health issue. Two themes were seen across focus groups—antibiotic resistance was seen as less of a priority than other public health issues faced by participants and antibiotic resistance was considered an issue but not for their patient population (table 1).

Table 1.

Themes and quotations from primary care physicians regarding antibiotic resistance as a public health issue

Themes Quotations
Antibiotic resistance seen as less important than other public health issues faced by primary care physicians
  1. ‘We are seeing some MRSA (methicillin-resistant Staphylococcus aureus). Everybody does. It is just so low on the totem pole compared with the other things that we are seeing’. – Birmingham, family medicine/internal medicine physician

  2. ‘It’s important, but in everyday practice I thought that other things were more important.’ – Chicago, paediatrician

Antibiotic resistance is an issue, but not for my patient population
  1. ‘I thought about antibiotic resistance as more of a problem, not in my practice that much, but in a hospital with a very sick person where they can’t find something because somebody’s resistant.” – Chicago, paediatrician

  2. ‘It’s not like I’m seeing my patients having an issue on a regular basis like these other things are. There’s this threat of this crazy super bug that will take over the world and kill us all, but I’ve never – it doesn’t seem like reality’. – Philadelphia, family medicine/internal medicine physician

  3. ‘We’re starting to see it in the community. I think if you had a table full of infectious disease doctors working in intensive care units, you would have different priorities. But in the outpatient, we probably see it less(…)It is a matter of time before we see it more. Who knows, a year, two, three from now, these numbers might be different.’ – Philadelphia, family medicine/internal medicine physician

Antibiotic resistance seen as less important than other public health issues

A common theme among focus group participants was the perception of antibiotic resistance being less important in their daily practice when compared with other public health issues they commonly faced, such as obesity, diabetes and opioid misuse.

Antibiotic resistance is an issue, but not for my patient population

While many participants acknowledged that antibiotic resistance is a concern, many did not see it as an issue that impacted their patients or their daily practice. Instead, most participants considered antibiotic resistance as something affecting sicker, hospitalised patients. In contrast, some participants acknowledged that they have seen an increase in resistant infections in their patients with urinary tract infections or skin infections. However, these participants still classified resistance as an issue largely impacting inpatient medicine.

Drivers of antibiotic prescribing

Participants in all focus groups also discussed what they thought was driving outpatient antibiotic prescribing. Two themes emerged: (1) participants argued that other physicians were the ones driving inappropriate prescribing and (2) patient demand for antibiotics continues to be an issue in primary care (table 2).

Table 2.

Themes and quotations from primary care physicians regarding drivers antibiotic prescribing

Themes Quotations
Attribution of inappropriate antibiotic prescribing to others
  1. ‘I think those of us who have our own practice and control of things probably(…)‘get it’ more than the hourly non-vested person in your walk-in clinics who are just basically drawing an hourly salary and their whole interest is in just getting rid of somebody.’ – Birmingham, family medicine/internal medicine physician

  2. ‘We’re always practicing evidence-based medicine, so it becomes incredibly challenging. With adult medicine, they’ll give out antibiotics over the phone, antibiotics without doing swabs and chest X-rays, things like that, or even seeing the patient.’ – Chicago, paediatrician

  3. ‘A lot of us don’t like to prescribe antibiotics, but they go to urgent cares and they go to(…)1 min clinics and they get prescribed antibiotics.’ – Los Angeles, family medicine/internal medicine physician

Patient demand as a driving factor
  1. ‘We’re under pressure all day. You don’t want to get written up, potentially, for being insensitive, or not taking care of them, or physician ratings.’ – Birmingham, paediatrician

  2. ‘They come in and it’s a boxing match. You are fighting in that corner with the misconception, preconceived notion and you’re trying to tell them that 2+2 = 4 and they are saying’, ‘No, it’s purple’. – Birmingham, family medicine/internal medicine physician

  3. ‘Sometimes you just like, you know what, I’m beaten down; so, here’s your Z-Pak. See you. Next patient. I’m not going to sit here and argue with somebody for 5 min over why they don’t need it.’ – Philadelphia, family medicine/internal medicine physician

Attribution of inappropriate antibiotic prescribing to others

Participants indicated that they believed inappropriate outpatient antibiotic prescribing is largely driven by clinicians other than themselves, namely those practising in urgent care offices and retail clinics. This contributed to the feeling that resisting patient demand for antibiotics is futile, as patients can simply see another clinician and get what they want. Participants also said that patients’ past experiences of receiving antibiotics from another clinician reinforced patient expectations for antibiotics for the same complaint. This, in their mind, strengthens patient resolve to demand antibiotics.

Patient demand as a driving factor

When discussing drivers of antibiotic prescribing habits, a common theme was the pressure participants said they experience from patients who the prescribers perceive to expect antibiotics even when not medically indicated. Participants often returned to this theme throughout the focus group discussions. Participants contended that patients often feel entitled to leave a visit with a material treatment—often an antibiotic—after spending time and money at a doctor’s office. Participants argued that patient pressure is compounded by the use of patient satisfaction scores when grading physician performance. They expressed concern that if they refused to prescribe an antibiotic for a patient who expected one, that the patient might write a negative review and/or score the physician poorly.

However, it is important to note that some participants indicated that the impact of patient expectations for antibiotics on their prescribing decisions can vary. For example, some participants indicated that they may be more willing to push back against prescribing an antibiotic if they have a long-standing relationship with a patient. This was more common among paediatricians as many of them indicated they have many opportunities to interact with patients and their parents during well child visits, making it easier for them to discuss why an antibiotic is or is not needed with parents.

Acceptability of antibiotic stewardship interventions: patient and physician education

On the topic of antibiotic stewardship efforts focused on patient and physician education, participants primarily indicated support for these activities (table 3).

Table 3.

Themes and quotations from primary care physicians regarding patient and physician education as antibiotic stewardship activities

Themes Quotations
Need for patient education
  1. ‘It will not work unless you educate the population. You cannot attack the doctors and curtail what they are doing until you educate patients that your doctor is doing the right thing.’ – Birmingham, family medicine/internal medicine physician

  2. ‘I think it’s more education. I think you could probably do more with a commercial than you can with anything else.’ – Chicago, paediatrician

Acceptability of physician education
  1. ‘Parents are going to ask. They don’t know what’s right or wrong. They’re not medically trained. It’s the physicians that need more education about not prescribing.’ – Chicago, paediatrician

  2. ‘I think the best education strategy we could get and maybe there could be a study done is how, what is the best way to communicate to patients that antibiotic overprescribing and resistance is a problem and that rings true to them, that we can tell them this and they’re going to understand that and accept the fact that it didn’t lead to antibiotics.’ – Los Angeles, family medicine/internal medicine physician

Need for patient education

Consistent with the perception of patient demand for antibiotics generating concern, participants emphasised that, in order for them to be able to effectively do their jobs, their patients need to be educated about when antibiotics are and are not appropriate and why judicious antibiotic use is critical to combating antibiotic resistance. Participants suggested several approaches for educating the public, including written education materials in different languages, educational videos for waiting rooms and direct-to-consumer advertisements. Finally, many physicians emphasised the need to provide education in advance of a patient visit. By the time a patient is at a doctor’s office for an illness, many felt it was too late to change patient expectations.

Acceptability of physician education

Many participants indicated that physician education would also be a welcome approach for outpatient antibiotic stewardship. Participants described educational efforts as more helpful for physicians compared with other interventions, such as providing feedback on prescribing practices, which was viewed as more critical of physicians. For example, participants indicated that training in how to communicate antibiotic prescribing decisions to patients would be helpful. One area of disagreement emerged around whether this education should be mandatory or voluntary. A few participants mentioned that requiring outpatient physicians to complete continuing medical education (CME) on antibiotic use—similar to requirements for CME around opioid prescribing—may be helpful. However, other participants indicated that they would prefer voluntary rather than mandatory CME.

Acceptability of performance reporting

When presented with examples of stewardship efforts aimed at measuring and providing feedback on antibiotic prescribing practices, physicians were less supportive compared with educational efforts. A number of themes emerged in this area—both themes specific to antibiotic prescribing measurement, as well as themes regarding quality measurement efforts more broadly (table 4).

Table 4.

Themes and quotations from primary care physicians regarding the acceptability of performance reporting

Feasibility of measuring antibiotic prescribing
  1. ‘Like I said, you’ll get patients who were seen within hours by two different people, and one gives the antibiotic and the other one doesn’t. It’s not necessarily that the person who doesn’t give it is always right, and the other one’s always wrong. It’s too subjective.’ – Chicago, paediatrician

  2. ‘There’s more thought process into the physician having to, there’s a reason basically why a physician chooses or not chooses to, the management specifically. So, until they actually come and look at our, the history, the physical, and overall clinical management, they really will not know why we prescribed the way we did it.’ – Los Angeles, family medicine/internal medicine physician

Belief that physicians will ‘game the system’
  1. ‘As soon as you start having measurements like that, you’re going to have a lot more diagnoses of walking pneumonia or pneumonia.’ – Los Angeles, family medicine/internal medicine physician

  2. ‘People don’t put down accurate diagnoses, and then when you have something like this, then everyone is going to start gaming the system. ‘I’m not going to put down diagnosis of bronchitis. No, I’m going to put sinusitis.’ Even through it’s bronchitis, I can give you the antibiotic and not get dinged for it.’ – Philadelphia, family medicine/internal medicine physician

Dissatisfaction with the quality measurement system
  1. ‘These days we’re all getting measured on everything. Every time we click a button on the EMR (electronic medical record) whether it’s diabetes, cholesterol, blood pressure, antibiotic prescribing, no matter what it is someone’s measuring it. Someone’s telling us what we should be doing. I think, I’ll speak for myself; physicians are starting to get tired of being told what to do.’ – Philadelphia, family medicine/internal medicine physician

  2. ‘That’s going to fall into a P for P programme. A payment for performance which is the insurance company’s way of paying doctors less money.’ – Los Angeles, paediatrician

  3. ‘We’ve discovered that they don’t work very well, and then, almost always if there’s an incentive for doing something, there’s going to be a punishment for not doing it. There’s never just the incentive.’ – Birmingham, paediatrician

Distrust of tracking and reporting systems
  1. ‘For example, I vaccinate every kid that comes to see me with Menactra(…)(Insurance company) recently said that I did not get 23 kids, but when I go to the state registry, every single one of those kids got their Menactra, before the age of 13. Their data collection practices are questionable and manipulable, and I don’t trust it.’ – Birmingham, paediatrician

  2. ‘The quality of the data seems always so poor(…)I have patients that I’ve never seen that are on my list, I had a patient that was dead for 2 years that was on my list. So the quality of the data collection and how you’re going to do that is so important.’ – Los Angeles, family medicine/internal medicine physician

Feasibility of measuring antibiotic prescribing

Participants expressed concerns about antibiotic stewardship activities focused on measuring inappropriate antibiotic use, questioning the feasibility of assessing prescribing quality while accounting for different patient populations. Some participants indicated that developing antibiotic use reports would likely require significant financial and time investments. Many participants argued that antibiotic use measures are unlikely to capture all of the clinical elements from an office visit to provide the full context behind an antibiotic prescription, and that setting standards for the quality of antibiotic use would be difficult. Some questioned who would be qualified to set these standards and how that might impact the accuracy and fairness of antibiotic use measures.

Belief that physicians will ‘game the system’

Participants also argued that, if antibiotic use measures were developed and implemented, other physicians would simply use the rules put in place to manipulate the desired outcome (ie, ‘game the system’) to improve their antibiotic prescribing scores. Some participants indicated that physicians could easily identify what diagnosis codes they were being measured on and shift coding practices to more antibiotic-appropriate conditions.

Dissatisfaction with the quality measurement system

Many participants also expressed dissatisfaction and general distrust of quality measurement systems and reporting processes. Participants expressed a sense of feeling overmeasured and being blamed for things beyond their control. Participants argued that quality measures assume that medicine is black and white and do not account for their need to use clinical judgement when treating patients. Some expressed concern that any new measure could eventually be turned around and used against them. Examples mentioned in different focus groups included using quality measures as a way to reimburse physicians at a lower level or a reason to fire a physician.

Distrust of tracking and reporting systems

Finally, participants described issues that they have experienced with the inaccuracy of tracking and reporting systems. For example, participants indicated that they often receive feedback reports that include patients that they have not seen in years, or feedback reports with clear coding errors. These inaccuracies lead them to generally dismiss the utility of these reports.

Discussion

We conducted focus groups with PCPs to assess their knowledge and attitudes towards antibiotic resistance, inappropriate antibiotic prescribing and outpatient antibiotic stewardship approaches. While participants recognised the public health importance of antibiotic resistance and antibiotic use, they felt these issues were less important compared with other public health priorities in outpatient care. This finding echoes past research,26 29 30 and adds further insight by placing antibiotic resistance within the broader context of public health issues encountered in primary care. Our participants consistently identified antibiotic resistance as a lower priority for their practice compared with other health concerns. This is consistent with what has been shown in hospital-based studies, with one Swedish study identifying a lack of recognition of antibiotic resistance as an imminent threat as a barrier to stewardship.39

Additionally, most study participants felt they were already good antibiotic stewards, but that their efforts were hindered by patient demand for antibiotics and the prescribing practices of other physicians. These findings are consistent with previous research on perceptions of drivers of outpatient antibiotic prescribing.11–25 In particular, the perception of patient demand as a driving force behind inappropriate prescribing practices continues to be a consistent finding across studies both in the USA and other countries. Similar perceptions have also been documented among inpatient physicians who believe that prescribers outside of hospitals are primarily responsible for antibiotic overuse and antibiotic resistance, underscoring the value for individual feedback on prescribing patterns to help these physicians recognise the need for antibiotic stewardship in their practice.40 This indicates a continued need for stewardship efforts to address these concerns moving forward.

Along those lines, participants in our study reacted positively to education-focused stewardship activities—particularly those activities aimed at educating the general public. Participants were also supportive of education targeting physicians, such as trainings on how to best communicate antibiotic prescribing decisions with patients. Previous studies from both Europe and the USA have shown physicians are receptive to educational efforts—ranging from national media campaigns to educating children in schools about antibiotics.15 16 21 26 Additionally, findings from inpatient studies also suggest treatment guidelines can be an impactful educational tool for changing prescribing behaviours.41 42

In contrast to educational efforts, our study participants remained unconvinced about the utility of antibiotic use tracking and reporting as a stewardship strategy. Participants’ negative attitudes regarding the feasibility of measuring the quality of antibiotic use in an accurate or fair manner, and their distrust of the quality measurement system in the USA in general factored into participants’ perceptions on the impact of antibiotic use measurement. A previous US study evaluating paediatrician perceptions of an intervention that included audit and feedback of antibiotic prescribing practices found high scepticism among physicians about the quality and accuracy of the feedback reports.12 However, while physicians remain hesitant about the approach of tracking and reporting antibiotic use, studies from the USA and UK have shown that this stewardship approach is effective at reducing inappropriate prescribing.43–47

Additional studies have evaluated physician perceptions of broader quality measurement systems. One study of US physicians in three states found that 71% felt that paediatric quality reports were effective at improving paediatric care.48 However, in interviews with providers in two of these states, authors found that physicians were frustrated with certain aspects of the quality reports, such as the inclusion of measures that they felt were outside of their control.48 A 2009 survey of US physicians on perceptions of Medicare’s Physician Quality Reporting Initiative (PQRI) found that 50% of physicians participating in PQRI programmes believed it had no impact on quality of care.49 By comparison, qualitative studies in the UK have shown that the national pay-for-performance programme has been mostly well received by the general practitioners.50 51 However, some concerns remain over reduced clinical autonomy, perceptions of micromanagement and scepticism of the validity of certain quality indicators. While these studies demonstrate that negative perceptions remain around financial incentive mechanisms, another study did find that how antibiotic use incentives are framed could be impactful—specifically framing these efforts as addressing harms to patients.52

Many of our findings are consistent with research on self-enhancement bias—that people take full credit for their success but are quick to dismiss failures as caused by external factors.53 54 Self-enhancement is adaptive because it protects against being discouraged or down on one’s self, preserves a person’s self-image and keeps them motivated to work and thrive in their life. This may explain findings from our focus groups, including: (1) the physicians’ belief that patients’ antibiotic knowledge deficits and other clinicians’ behaviours were key drivers of overprescribing and (2) their defensive responses when confronted with the potential for reports of their own prescribing by questioning the validity of the measurement enterprise. These perceptions present a challenge when addressing inappropriate antibiotic prescribing. Some approaches that have been successful in reducing antibiotic overprescribing invert the problem of self-enhancement by using it as a way to encourage or reward appropriate prescribing. These interventions engender reputational concerns when antibiotics are used or make explicit social comparisons of performance with others to encourage pursuit of a positive self-image (eg, the prospect of becoming a ‘top performer’) through lower prescribing.55 56

This study provides important insight into current attitudes of PCPs in the USA on issues related to antibiotic resistance, inappropriate antibiotic prescribing and antibiotic stewardship implementation. Enhanced understanding of perceptions among these providers is critical as healthcare stakeholders work to expand stewardship activities into primary care settings. Two of the four core elements for outpatient antibiotic stewardship are tracking and reporting antibiotic prescribing practices, and the provision of education and access to expertise on antibiotic prescribing.10 While our study shows support from PCPs for educational initiatives, more work will be needed to address PCP concerns related to tracking and reporting antibiotic use. Healthcare stakeholders will need to work to build trust among the PCP community around antibiotic prescribing feedback, and will need to identify ways to make the feedback provided to physicians actionable to help ensure impact.

Another important barrier to expanded stewardship implementation identified by this study is a continued lack of prioritisation of this issue among PCPs. Healthcare stakeholders—including public health authorities, health plans and health systems—will need to identify ways to incentivise stewardship uptake in primary care practices in the USA.

By conducting focus groups in cities across the USA and with internists, family medicine physicians and paediatricians, we were able to gather data from a wide-range of perspectives. The consistent themes identified throughout the focus groups suggest that barriers to stewardship implementation may be similar across the USA.

This study also has limitations. Because this is a qualitative study with a small sample size, these findings cannot be generalised to the broader US PCP population or to general practitioners in other countries. Participants were drawn from physician databases maintained for research purposes. Physicians who were recruited and who participated in these focus groups may have different or stronger opinions than those who did not. Additionally, as we recruited physicians from cities in each of the four US Census regions, it is possible that views and perceptions of PCPs from rural practices were underrepresented in this study. Finally, this study evaluated physicians who specialised in family medicine, internal medicine or paediatrics. We did not include other primary care clinicians, such as nurse practitioners or physician assistants. Additional research will be needed to assess whether these findings are applicable to the broader primary care clinician community.

In conclusion, the findings from these focus groups show that more work is needed to elevate the issue of antibiotic resistance and the need for improved prescribing among PCPs in the USA. Additionally, current scepticism among PCPs about the feasibility and accuracy of antibiotic use measurement may create concern around interventions that rely solely on tracking and reporting prescribing. It will be important to address these perceptions when designing interventions aimed at decreasing inappropriate antibiotic prescribing in outpatient settings.

Supplementary Material

Reviewer comments
Author's manuscript

Footnotes

Twitter: @jeffreylinder

Contributors: RMZ led the development of the study concept and design, participated in the analysis and interpretation of data, and drafted the manuscript. AS participated in the analysis and interpretation of data and provided critical revision of the manuscript. AMG provided input in the development of the study concept and design, participated in the interpretation of data, and provided critical revision of the manuscript. JND provided input in the development of the study concept and design, participated in the interpretation of data, and provided critical revision of the manuscript. JSG provided input in the development of the study concept and design, participated in the interpretation of data, and provided critical revision of the manuscript. JAL provided input in the development of the study concept and design, participated in the interpretation of data, and provided critical revision of the manuscript. DYH provided supervision in the development of the study concept and design, participated in the analysis and interpretation of data, and provided critical revision of the manuscript.

Funding: The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

Competing interests: JAL, JSG and JND all received honoraria for time dedicated to this research project. JND has received consultant fees from Precision Health Economics and University of Pennsylvania Health System. JND was supported by grants from the National Institutes of on Aging (R21AG057395, R21AG057400, R21AG057396, R21AG057383, P30AG024968, R33AG057395), the National Institute on Drug Abuse (R01 DA046226), the National Institute of Arthirist and Musculoskeletal Conditions (R01 AR073486), the Agency for Healthcare Research and Quality (R01 HS026506) and the Patient-CenteredCentred Outcome Research Institute (CDRN-1306–04864). JAL was supported by grants from the National Institute on AgingAgeing (R21AG057400, R21AG057396, R21AG057383, R21AG057395), Agency for Healthcare Research and Quality (R01HS024930, R01HS026506), The Peterson Center on Healthcare, and a contract from the Agency for Healthcare Research and Quality (HHSP2332015000201).

Patient consent for publication: Not required.

Ethics approval: The study protocol was reviewed for ethical considerations and deemed exempt by the Chesapeake IRB (now known as Advarra).

Provenance and peer review: Not commissioned; externally peer reviewed.

Data availability statement: All data relevant to the study are included in the article or uploaded as online supplementary information. No additional data beyond what is presented in the manuscript is available.

References

  • 1.Suda KJ, Hicks LA, Roberts RM, et al. . Antibiotic expenditures by medication, class, and healthcare setting in the United States, 2010-2015. Clin Infect Dis 2018;66:185–90. 10.1093/cid/cix773 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2.Public Health England English surveillance programme for antimicrobial utilisation and resistance (ESPAUR): report, 2014. Available: https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/362374/ESPAUR_Report_2014__3_.pdf [Accessed 18 April 2019].
  • 3.Public Health Agency of Sweden, National Veterinary Institute Consumption of antibiotics and occurrence of antibiotic resistance in Sweden. Swedres-Svarm 2014. Report No.: ISSN 1650-6332. Available: https://www.folkhalsomyndigheten.se/pagefiles/20281/Swedres-Svarm-2014-14027.pdf [Accessed 18 April 2019].
  • 4.King LM, Bartoces M, Fleming-Dutra KE, et al. . Changes in US outpatient antibiotic prescriptions from 2011-2016. Clin Infect Dis 2020;70:370–7. 10.1093/cid/ciz225 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5.Fleming-Dutra KE, Hersh AL, Shapiro DJ, et al. . Prevalence of inappropriate antibiotic prescriptions among US ambulatory care visits, 2010-2011. JAMA 2016;315:1864–73. 10.1001/jama.2016.4151 [DOI] [PubMed] [Google Scholar]
  • 6.Hersh AL, Fleming-Dutra KE, Shapiro DJ, et al. . Frequency of first-line antibiotic selection among US ambulatory care visits for otitis media, sinusitis, and pharyngitis. JAMA Intern Med 2016;176:1870–2. 10.1001/jamainternmed.2016.6625 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7.Chua K-P, Fischer MA, Linder JA. Appropriateness of outpatient antibiotic prescribing among privately insured us patients: ICD-10-CM based cross sectional study. BMJ 2019;364:k5092. 10.1136/bmj.k5092 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.Palms DL, Hicks LA, Bartoces M, et al. . Comparison of antibiotic prescribing in retail clinics, urgent care centers, emergency departments, and traditional ambulatory care settings in the United States. JAMA Intern Med 2018;178:1267–9. 10.1001/jamainternmed.2018.1632 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9.Fischer MA, Mahersi M, Lii J, et al. . Invisible threats to improving antibiotic use: non-infection-related and non-visit-based antibiotic prescribing for Medicaid patients. Health Aff 2019. [DOI] [PubMed] [Google Scholar]
  • 10.Centers for Disease Control and Prevention The core elements of outpatient antibiotic stewardship, 2016. Available: https://www.cdc.gov/antibiotic-use/community/improving-prescribing/core-elements/core-outpatient-stewardship.html [Accessed 11 Nov 2018].
  • 11.Sanchez GV, Roberts RM, Albert AP, et al. . Effects of knowledge, attitudes, and practices of primary care providers on antibiotic selection, United States. Emerg Infect Dis 2014;20:2041–7. 10.3201/eid2012.140331 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12.Szymczak JE, Feemster KA, Zaoutis TE, et al. . Pediatrician perceptions of an outpatient antimicrobial stewardship intervention. Infect Control Hosp Epidemiol 2014;35 Suppl 3:S69–78. 10.1086/677826 [DOI] [PubMed] [Google Scholar]
  • 13.Petursson P. GPs' reasons for "non-pharmacological" prescribing of antibiotics. A phenomenological study. Scand J Prim Health Care 2005;23:120–5. 10.1080/02813430510018491 [DOI] [PubMed] [Google Scholar]
  • 14.Evans CT, Rogers TJ, Weaver FM, et al. . Providers' beliefs and behaviors regarding antibiotic prescribing and antibiotic resistance in persons with spinal cord injury or disorder. J Spinal Cord Med 2011;34:16–21. 10.1179/107902610X12886261091794 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15.Dempsey PP, Businger AC, Whaley LE, et al. . Primary care clinicians' perceptions about antibiotic prescribing for acute bronchitis: a qualitative study. BMC Fam Pract 2014;15:194. 10.1186/s12875-014-0194-5 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.Butler CC, Rollnick S, Pill R, et al. . Understanding the culture of prescribing: qualitative study of general practitioners' and patients' perceptions of antibiotics for sore throats. BMJ 1998;317:637–42. 10.1136/bmj.317.7159.637 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.May L, Gudger G, Armstrong P, et al. . Multisite exploration of clinical decision making for antibiotic use by emergency medicine providers using quantitative and qualitative methods. Infect Control Hosp Epidemiol 2014;35:1114–25. 10.1086/677637 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18.Bauchner H, Pelton SI, Klein JO. Parents, physicians, and antibiotic use. Pediatrics 1999;103:395–401. 10.1542/peds.103.2.395 [DOI] [PubMed] [Google Scholar]
  • 19.Yates TD, Davis ME, Taylor YJ, et al. . Not a magic pill: a qualitative exploration of provider perspectives on antibiotic prescribing in the outpatient setting. BMC Fam Pract 2018;19:96. 10.1186/s12875-018-0788-4 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.van der Zande MM, Dembinsky M, Aresi G, et al. . General practitioners' accounts of negotiating antibiotic prescribing decisions with patients: a qualitative study on what influences antibiotic prescribing in low, medium and high prescribing practices. BMC Fam Pract 2019;20:172. 10.1186/s12875-019-1065-x [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21.O'Doherty J, Leader LFW, O'Regan A, et al. . Over prescribing of antibiotics for acute respiratory tract infections; a qualitative study to explore Irish general practitioners' perspectives. BMC Fam Pract 2019;20:27. 10.1186/s12875-019-0917-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22.Fletcher-Lartey S, Yee M, Gaarslev C, et al. . Why do general practitioners prescribe antibiotics for upper respiratory tract infections to meet patient expectations: a mixed methods study. BMJ Open 2016;6:e012244. 10.1136/bmjopen-2016-012244 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23.Rose J, Crosbie M, Stewart A. A qualitative literature review exploring the drivers influencing antibiotic over-prescribing by GPs in primary care and recommendations to reduce unnecessary prescribing. Perspect Public Health 2019:175791391987918. 10.1177/1757913919879183 [DOI] [PubMed] [Google Scholar]
  • 24.Sunde M, Nygaard MM, Høye S. General practitioners' attitudes toward municipal initiatives to improve antibiotic Prescribing-A mixed-methods study. Antibiotics 2019;8:120. 10.3390/antibiotics8030120 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 25.McKay R, Mah A, Law MR, et al. . Systematic review of factors associated with antibiotic prescribing for respiratory tract infections. Antimicrob Agents Chemother 2016;60:4106–18. 10.1128/AAC.00209-16 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 26.McCullough AR, Rathbone J, Parekh S, et al. . Not in my backyard: a systematic review of clinicians' knowledge and beliefs about antibiotic resistance. J Antimicrob Chemother 2015;70:2465–73. 10.1093/jac/dkv164 [DOI] [PubMed] [Google Scholar]
  • 27.Brookes-Howell L, Hood K, Cooper L, et al. . Clinical influences on antibiotic prescribing decisions for lower respiratory tract infection: a nine country qualitative study of variation in care. BMJ Open 2012;2:e000795. 10.1136/bmjopen-2011-000795 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 28.Poss-Doering R, Kamradt M, Stuermlinger A, et al. . The complex phenomenon of dysrational antibiotics prescribing decisions in German primary healthcare: a qualitative interview study using dual process theory. Antimicrob Resist Infect Control 2020;9:6. 10.1186/s13756-019-0664-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 29.Wood F, Phillips C, Brookes-Howell L, et al. . Primary care clinicians' perceptions of antibiotic resistance: a multi-country qualitative interview study. J Antimicrob Chemother 2013;68:237–43. 10.1093/jac/dks338 [DOI] [PubMed] [Google Scholar]
  • 30.Simpson SA, Wood F, Butler CC. General practitioners' perceptions of antimicrobial resistance: a qualitative study. J Antimicrob Chemother 2007;59:292–6. 10.1093/jac/dkl467 [DOI] [PubMed] [Google Scholar]
  • 31.Centers for Disease Control and Prevention Outpatient antibiotic prescriptions – United States, 2017. Available: https://www.cdc.gov/antibiotic-use/community/programs-measurement/state-local-activities/outpatient-antibiotic-prescriptions-US-2017.html [Accessed 10 January 2020].
  • 32.Petterson S, McNellis R, Klink K, et al. . The state of primary care in the United States: a Chartbook of facts and statistics, 2018. Available: http://graham-center.org/content/dam/rgc/documents/publications-reports/reports/PrimaryCareChartbook.pdf [Accessed 10 Jan 2020].
  • 33.Doty MM, Tikkanen R, Shah A, et al. . Primary Care Physicians’ Role In Coordinating Medical And Health-Related Social Needs In Eleven Countries. Health Aff 2020;39:115–23. 10.1377/hlthaff.2019.01088 [DOI] [PubMed] [Google Scholar]
  • 34.O'Brien BC, Harris IB, Beckman TJ, et al. . Standards for reporting qualitative research: a synthesis of recommendations. Acad Med 2014;89:1245–51. 10.1097/ACM.0000000000000388 [DOI] [PubMed] [Google Scholar]
  • 35.Elo S, Kyngäs H. The qualitative content analysis process. J Adv Nurs 2008;62:107–15. 10.1111/j.1365-2648.2007.04569.x [DOI] [PubMed] [Google Scholar]
  • 36.Bradley EH, Curry LA, Devers KJ. Qualitative data analysis for health services research: developing taxonomy, themes, and theory. Health Serv Res 2007;42:1758–72. 10.1111/j.1475-6773.2006.00684.x [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 37.de Bont EGPM, Peetoom KKB, Moser A, et al. . Childhood fever: a qualitative study on GPs’ experiences during out-of-hours care. Fam Pract 2015;32:cmv029–55. 10.1093/fampra/cmv029 [DOI] [PubMed] [Google Scholar]
  • 38.Teixeira Rodrigues A, Roque F, Falcão A, et al. . Understanding physician antibiotic prescribing behaviour: a systematic review of qualitative studies. Int J Antimicrob Agents 2013;41:203–12. 10.1016/j.ijantimicag.2012.09.003 [DOI] [PubMed] [Google Scholar]
  • 39.Björkman I, Berg J, Röing M, et al. . Perceptions among Swedish hospital physicians on prescribing of antibiotics and antibiotic resistance. Qual Saf Health Care 2010;19:e8. 10.1136/qshc.2008.029199 [DOI] [PubMed] [Google Scholar]
  • 40.Krockow EM, Colman AM, Chattoe-Brown E, et al. . Balancing the risks to individual and society: a systematic review and synthesis of qualitative research on antibiotic prescribing behaviour in hospitals. J Hosp Infect 2019;101:428–39. 10.1016/j.jhin.2018.08.007 [DOI] [PubMed] [Google Scholar]
  • 41.Feiring E, Walter AB. Antimicrobial stewardship: a qualitative study of the development of national guidelines for antibiotic use in hospitals. BMC Health Serv Res 2017;17:747. 10.1186/s12913-017-2683-4 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 42.Skodvin B, Aase K, Charani E, et al. . An antimicrobial stewardship program initiative: a qualitative study on prescribing practices among hospital doctors. Antimicrob Resist Infect Control 2015;4:24. 10.1186/s13756-015-0065-4 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 43.Ratajczak M, Gold N, Hailstone S, et al. . The effectiveness of repeating a social norm feedback intervention to high prescribers of antibiotics in general practice: a national regression discontinuity design. J Antimicrob Chemother 2019;74:3603–10. 10.1093/jac/dkz392 [DOI] [PubMed] [Google Scholar]
  • 44.Bradley DT, Allen SE, Quinn H, et al. . Social norm feedback reduces primary care antibiotic prescribing in a regression discontinuity study. J Antimicrob Chemother 2019;74:2797–802. 10.1093/jac/dkz222 [DOI] [PubMed] [Google Scholar]
  • 45.Gulliford MC, Prevost AT, Charlton J, et al. . Effectiveness and safety of electronically delivered prescribing feedback and decision support on antibiotic use for respiratory illness in primary care: reduce cluster randomised trial. BMJ 2019;1:l236 10.1136/bmj.l236 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 46.Hallsworth M, Chadborn T, Sallis A, et al. . Provision of social norm feedback to high prescribers of antibiotics in general practice: a pragmatic national randomised controlled trial. Lancet 2016;387:1743–52. 10.1016/S0140-6736(16)00215-4 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 47.Gerber JS, Prasad PA, Fiks AG, et al. . Durability of benefits of an outpatient antimicrobial stewardship intervention after discontinuation of audit and feedback. JAMA 2014;312:2569–70. 10.1001/jama.2014.14042 [DOI] [PubMed] [Google Scholar]
  • 48.Zickafoose JS, Ireys HT, Swinburn A, et al. . Primary care physicians' experiences with and attitudes toward pediatric quality reporting. Acad Pediatr 2016;16:750–9. 10.1016/j.acap.2016.07.005 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 49.Federman AD, Keyhani S. Physicians’ participation in the Physicians’ Quality Reporting Initiative and their perceptions of its impact on quality of care. Health Policy 2011;102:229–34. 10.1016/j.healthpol.2011.05.003 [DOI] [PubMed] [Google Scholar]
  • 50.Lester H, Matharu T, Mohammed MA, et al. . Implementation of pay for performance in primary care: a qualitative study 8 years after introduction. Br J Gen Pract 2013;63:e408–15. 10.3399/bjgp13X668203 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 51.Maisey S, Steel N, Marsh R, et al. . Effects of payment for performance in primary care: qualitative interview study. J Health Serv Res Policy 2008;13:133–9. 10.1258/jhsrp.2008.007118 [DOI] [PubMed] [Google Scholar]
  • 52.Liao JM, Schapira MS, Navathe AS, et al. . The effect of emphasizing patient, societal, and institutional harms of inappropriate antibiotic prescribing on physician support of financial penalties: a randomized trial. Ann Intern Med 2017;167:215–6. 10.7326/L17-0102 [DOI] [PubMed] [Google Scholar]
  • 53.Brown JD. Evaluations of self and others: Self-Enhancement biases in social judgments. Soc Cogn 1986;4:353–76. 10.1521/soco.1986.4.4.353 [DOI] [Google Scholar]
  • 54.John OP, Robins RW. Accuracy and bias in self-perception: individual differences in self-enhancement and the role of narcissism. J Pers Soc Psychol 1994;66:206–19. 10.1037/0022-3514.66.1.206 [DOI] [PubMed] [Google Scholar]
  • 55.Meeker D, Linder JA, Fox CR, et al. . Effect of behavioral interventions on inappropriate antibiotic prescribing among primary care practices: a randomized clinical trial. JAMA 2016;315:562–70. 10.1001/jama.2016.0275 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 56.Linder JA, Meeker D, Fox CR, et al. . Effects of behavioral interventions on inappropriate antibiotic prescribing in primary care 12 months after stopping interventions. JAMA 2017;318:1391–2. 10.1001/jama.2017.11152 [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplementary data

bmjopen-2019-034983supp001.pdf (32.3KB, pdf)

Supplementary data

bmjopen-2019-034983supp002.pdf (162.8KB, pdf)

Reviewer comments
Author's manuscript

Articles from BMJ Open are provided here courtesy of BMJ Publishing Group

RESOURCES