Skip to main content
Proceedings of the National Academy of Sciences of the United States of America logoLink to Proceedings of the National Academy of Sciences of the United States of America
. 2018 Dec 17;115(51):E11988–E11995. doi: 10.1073/pnas.1810840115

Estimating the proportion of bystander selection for antibiotic resistance among potentially pathogenic bacterial flora

Christine Tedijanto a,1, Scott W Olesen b, Yonatan H Grad b,c, Marc Lipsitch a,b
PMCID: PMC6304942  PMID: 30559213

Significance

Bystander selection, defined as the inadvertent pressure imposed by antibiotic treatment on microbes other than the targeted pathogen, is hypothesized to be a major factor in the propagation of antibiotic resistance, but its extent has not been characterized. We estimate the proportion of bystander exposures across a range of antibiotics and potential pathogens commonly found in the normal flora and describe factors driving variability of these proportions. Impact estimates for antibiotic resistance interventions, including vaccination, are often limited to effects on a target pathogen. However, the reduction of antibiotic treatment for conditions caused by one pathogen may have the broader potential to decrease bystander selection pressures for resistance on many other organisms.

Keywords: antibiotic resistance, vaccines, microbiome

Abstract

Bystander selection—the selective pressure for resistance exerted by antibiotics on microbes that are not the target pathogen of treatment—is critical to understanding the total impact of broad-spectrum antibiotic use on pathogenic bacterial species that are often carried asymptomatically. However, to our knowledge, this effect has never been quantified. We quantify bystander selection for resistance for a range of clinically relevant antibiotic–species pairs as the proportion of all antibiotic exposures received by a species for conditions in which that species was not the causative pathogen (“proportion of bystander exposures”). Data sources include the 2010–2011 National Ambulatory Medical Care Survey and National Hospital Ambulatory Medical Care Survey, the Human Microbiome Project, and additional carriage and etiological data from existing literature. For outpatient prescribing in the United States, we find that this proportion over all included antibiotic classes is over 80% for eight of nine organisms of interest. Low proportions of bystander exposure are often associated with infrequent bacterial carriage or concentrated prescribing of a particular antibiotic for conditions caused by the species of interest. Applying our results, we roughly estimate that pneumococcal conjugate vaccination programs result in nearly the same proportional reduction in total antibiotic exposures of Streptococcus pneumoniae, Staphylococcus aureus, and Escherichia coli, despite the latter two organisms not being targeted by the vaccine. These results underscore the importance of considering antibiotic exposures of bystanders, in addition to the target pathogen, in measuring the impact of antibiotic resistance interventions.


Antibiotic use creates selective pressures favoring resistant microbes. While designed to control the pathogenic bacteria causing an infection (we use the term “target pathogen”), currently available antibiotics often have antimicrobial activity against many bacterial species and disseminate widely throughout the body (1). Thus, the bacteria that comprise the human microbiome are subject to the selective pressures applied by most antibiotic consumption (24). These selective pressures for resistance experienced by microbial flora due to antibiotic exposures for a condition not caused by that species can be called “bystander selection.” While nonpathogenic commensals residing in the microbiome are always bystanders, opportunistic and obligate pathogens, which are often carried asymptomatically, lie at the critical intersection where resistance is clinically relevant and the extent of bystander selection among them is unknown. Although commensals may be an important reservoir for resistance elements, our ability to measure horizontal transmission between organisms in the microbiome is limited. Therefore, we focus on bystander selection for resistance due to antibiotic exposures experienced directly by potential pathogens.

Quantifying bystander selection, as defined above, is important for evaluating the impact of antibiotic resistance control interventions. For example, vaccination and infection control strategies are designed to reduce the need for appropriate antibiotic treatments and thus decrease selective pressure for resistance on a target pathogen: vaccination by reducing the incidence of disease from, say, Streptococcus pneumoniae and thus the need for antibiotic treatment (5, 6), and infection control by reducing the incidence of hospital-acquired infections that will require treatment. Often overlooked is the impact that averted treatment may have beyond the target pathogens, because each treatment averted would have exerted selection on bystanders as well. For stewardship interventions, which aim to avert inappropriate treatment of conditions that are never or seldom caused by bacteria, the primary goal of the intervention is to avert bystander selection of the patient’s normal flora. Mathematical transmission models that aim to simulate the dynamics of antibiotic resistance and to project the impact of interventions on pathogenic bacteria with an asymptomatic carriage state often assume that treatment incidence is independent of colonization with the bacterium of interest, implying that bystander selection is the rule rather than the exception (79). Prior to this study, there has not been sufficient evidence to support this claim.

This work aims to estimate the extent of bystander selection for resistance due to outpatient prescribing in the United States for a range of clinically relevant species and antibiotic combinations. Prescriptions are used as a measured proxy for exposures and, ultimately, for selection. We use existing data, including the National Ambulatory Medical Care Survey and National Hospital Ambulatory Medical Care Survey (NAMCS/NHAMCS) to estimate prescription volume and associated diagnoses and the Human Microbiome Project (HMP) and other studies of bacterial carriage to estimate the microbial communities subject to selection. We quantify bystander selection as the proportion of total exposures of an antibiotic experienced by a species when that species was not the target pathogen of treatment and will refer to this measure as the “proportion of bystander exposures.”

Understanding the contribution of bystander exposures to the landscape of selective pressures for antibiotic resistance at the population level will help to inform interventions including vaccines and antibiotic stewardship. Given the special attention of the current issue of PNAS to vaccines and antimicrobial resistance, we spell out how such measures can contribute to estimating the impact of vaccines, in particular pneumococcal conjugate vaccines (PCVs), whose impact on antimicrobial resistance has received arguably the most attention of any vaccine (10, 11).

Results

Data Source Characteristics.

After exclusion of visits resulting in hospital or observation unit admission, the NAMCS and NHAMCS from 2010 to 2011 with nationally representative sampling weights were used to estimate outpatient diagnosis and prescription volume in the United States (Materials and Methods); 94.4% of total sampled visits were included in the analysis, with 14.7% of these visits resulting in at least one antibiotic prescription. Visits with one or more of the conditions explicitly included in the bystander analysis (x axis of Fig. 1A) accounted for 55.8% of unweighted prescriptions of our antibiotic classes of interest. Of included visits with one of these conditions, 6% were clinical encounters with patients less than 1 y old and 19% with patients between the ages of 1 and 5 y. The HMP data include isolates from healthy individuals between the ages of 18 and 40 y sampled at 15–18 locations across five major body sites: the nasal passages, oral cavity, skin, gastrointestinal tract, and urogenital tract. This analysis was agnostic to body site; an individual was considered to have positive carriage status of a particular species if that species was identified at any body site.

Fig. 1.

Fig. 1.

Inputs and overall results of bystander analysis. (A) Heat map shading represents the proportion of visits (after weighting to be nationally representative) with a diagnosis of the specified condition, given that the visit resulted in a prescription of the specified antibiotic class. Results for TMP/SMX and nitrofurantoin are for the individual drug instead of an antibiotic class. Rows are not required to sum to 100%, as only a subset of conditions are shown, and each visit may be associated with more than one condition. Antibiotics included in each class are based on the Multum Lexicon classification system. Macr./Linc., Macrolides/lincosamides; TMP/SMX, trimethoprim–sulfamethoxazole. Diagonal lines indicate cells with value of 0. (B) Heat map shading represents the estimated etiology of each condition by species. If etiological data were available for multiple age groups, the weighted mean based on the relative frequency of visits (after weighting to be nationally representative) for that condition is shown. Diagonal lines indicate cells with value of 0. (C) Bars indicate mean carriage prevalence of each species across age groups, weighted by relative frequency of visits (after weighting to be nationally representative). (D) Bars indicate proportion of bystander exposures by antibiotic class and species (Bas) with 95% confidence intervals. “Overall” estimates reflect exposures to antibiotic in any of the classes shown in A.

Estimating the Proportion of Bystander Selection for Resistance by Species and Antibiotic or Class.

The three inputs required to calculate the proportion of bystander exposures are antibiotic prescriptions by condition, condition etiologies, and carriage prevalence of each species. In Fig. 1A, we depict antibiotic use as the proportion of weighted visits at which the specified condition was diagnosed, given that the visit resulted in a prescription of the specified antibiotic class; this value is a function of both antibiotic use by condition and volume of visits for the given condition. For example, this proportion is relatively high for use of penicillins for suppurative otitis media and macrolide/lincosamide use for bronchitis, common conditions leading to frequent prescriptions of the respective antibiotic class. While a high proportion of pneumonia cases also result in antibiotic prescriptions, low incidence in the outpatient setting leads to low proportions of antibiotic use associated with this condition. Nitrofurantoin presents an extreme case where use is targeted toward a single, common condition, urinary tract infections (UTIs). While many of the conditions that we consider do not have bacterial etiologies, those that do are infrequently caused by our bacteria of interest; conditions which are primarily caused by a single organism are streptococcal sore throat (strep), which we assume is always caused by Streptococcus pyogenes, and UTIs, commonly caused by Escherichia coli (Fig. 1B). Carriage prevalence varies dramatically from nearly 75% for Haemophilus influenzae and E. coli to well below 5% for S. pyogenes, Streptococcus agalactiae, and Pseudomonas aeruginosa (Fig. 1C).

The connections between antibiotic use, etiology, and carriage prevalence may be observed more clearly when considering bystander selection for a single antibiotic or class. We discuss two examples: E. coli and quinolones and S. pyogenes and penicillins. Quinolones, such as ciprofloxacin, are frequently used to treat UTIs (Fig. 1A), which, as previously mentioned, are commonly caused by E. coli (Fig. 1B). This alignment between antibiotic use and etiology results in a lower proportion of bystander exposures to quinolones for E. coli, compared with other species (SI Appendix, Fig. S1). This relationship is accentuated for species with low carriage prevalence. For example, a moderate proportion of penicillin use is directed toward strep throat (Fig. 1A), for which we assume S. pyogenes is the sole cause (Fig. 1B). Due to this extreme etiology and low carriage prevalence, exposures of S. pyogenes to all included antibiotics, especially penicillins, occur more frequently when S. pyogenes is a target pathogen and not a bystander. This factor also contributes to the low bystander proportion of P. aeruginosa for antibiotics used to treat UTIs. The bystander proportion for P. aeruginosa is often comparable to that of E. coli, even though it is a far less common cause of UTIs; compare this to Klebsiella pneumoniae, which causes more UTIs than P. aeruginosa but is more prevalent in carriage and thus experiences more bystander exposures. Thus, low carriage prevalence is also a driver of low bystander selection.

Overall, the proportion of bystander exposures exceeded 80% for eight out of nine organisms (all except S. pyogenes) when considering exposures to any of our antibiotic classes of interest (Fig. 1D) and was above 80% for 133 out of 153 (86.9%) antibiotic–species pairs (SI Appendix, Fig. S4). These results indicate that for the majority of antibiotic and species combinations, fewer than 20% of the exposures of that species to the antibiotic in question occur in the context of treating a disease caused by that species. Of particular clinical interest, 83.9% (95% CI: 80.9%, 86.6%) of S. pneumoniae exposures to penicillins and 93% (95% CI: 90.7%, 94.5%) of exposures to macrolides occurred when S. pneumoniae was not the target pathogen of disease. For E. coli, the proportion of bystander exposures was 81.3% (95% CI: 79.1%, 84%) for quinolones and 93.2% (95% CI: 91.4%, 94.3%) for cephalosporins, both of which are often used to treat pathogenic E. coli. The proportion of bystander exposures was similarly high for Staphylococcus aureus and penicillins at 91% (95% CI: 86%, 94.6%). S. pyogenes, which is rarely carried asymptomatically, only experienced 35.2% (95% CI: 30.6%, 55.9%) of its exposures to penicillins as a bystander.

Antibiotic resistance in Neisseria gonorrhoeae is of urgent concern, and recent ecological (12) and individual-level (13) studies have implicated bystander selection as a potential driver of macrolide resistance. Due to the low incidence of gonorrhea in the general population, limited data were available from NAMCS/NHAMCS. We used additional data from the Gonococcal Isolate Surveillance Project (GISP) (14) with slightly modified methods (Materials and Methods) to estimate the proportion of bystander exposures for N. gonorrhoeae. From 2010 to 2011, the proportion of bystander exposures for N. gonorrhoeae was 97.7% for ciprofloxacin and 4.8% for ceftriaxone. At the antibiotic class level, the proportion of bystander exposures for N. gonorrhoeae was 97.5% for quinolones and 14.6% for cephalosporins. GISP data on macrolide and tetracycline use were unavailable for 2010. For 2011, GISP reports combine prescription data for azithromycin and erythromycin and for doxycycline and tetracycline. Assuming that 75–95% of the azithromycin–erythromycin volume can be attributed to azithromycin, we estimate that the proportion of bystander exposures for N. gonorrhoeae ranges from 22.8 to 18.9%, respectively. Similarly, assuming that 75–95% of reported doxycycline–tetracycline use is doxycycline results in estimates for the proportion of doxycycline bystander exposures for N. gonorrhoeae of 29.7–25%. At the antibiotic class level, the proportion of bystander exposures for N. gonorrhoeae was 22.8% for macrolides/lincosamides, including all azithromycin-erythromycin use, and 28.6% for tetracyclines, accounting for all doxycycline–tetracycline use.

Application to Vaccine Impact on Antimicrobial Exposures in Bystanders.

We provide here preliminary estimates of the bystander impact of vaccines to illustrate how such calculations might be performed. Because the requisite quantities have not all been estimated in the same population, we combine estimates from different populations for the purposes of illustration, but we note the need for additional data to improve the level of confidence in such calculations by comparing quantities within a single population. We take the example of the PCV, which reduces bacteremia, meningitis, pneumonia, and otitis media caused by 7, 10, or 13 serotypes of pneumococci, depending on the formulation.

Considering only immediate direct effects, we made a conservative estimation of the effect of PCV on antimicrobial exposures of bystander organisms (Materials and Methods). A randomized controlled trial of the seven-valent PCV found a 7.8% reduction in otitis media among vaccinated vs. control children and a 5.4% in all-cause antimicrobial prescribing, mainly attributed to reduced otitis media (5). Restricting attention to the 0- to 1- and 1- to 5-y-old age groups in our study, this would translate to a 5.2% reduction in exposure of S. aureus to antibiotics and a 5.4% reduction in exposure of E. coli to antibiotics in these age groups.

Much larger estimates of impact on otitis media are obtained in studies that account for herd immunity effects and for the possibility that PCVs can indirectly prevent some nonpneumococcal otitis media (15). An Israeli study found a 57–71% reduction in all-cause otitis media associated with the rollout of PCV13 in various age groups up to the third birthday (16), while a study in the United Kingdom found a 36% reduction in otitis media among children under 10 y old comparing the post-PCV13 period to the pre-PCV7 period and a 29% reduction in otitis media-associated antimicrobial prescribing for the same comparison (6). Impact on total antimicrobial prescribing was not reported in the United Kingdom (6). If we assume that the ratio of 0.69 percentage points reduction in total prescribing per percentage point reduction in otitis media can be extrapolated from the randomized controlled trial in California (5), this 36% reduction in otitis media would correspond to a 25% reduction in all-cause antibiotic prescribing. Using our estimates of prevalence and bystander proportion, this would yield a 24.99% and 24.2% reduction in outpatient exposure of bystanders E. coli and S. aureus to antibiotics. While these calculations require a number of assumptions, they underscore the potentially substantial impact of vaccines on bystander selection and the need for improved data on the impact of vaccination on use of specific antimicrobials in specific populations.

Discussion

For most bacterial species, the majority of their antibiotic exposures were the result of treatment for a condition that they did not cause. This held true across a range of different organisms and antibiotics. Carriage prevalence was the key predictive factor of the differences in proportion of bystander exposures between organisms, with species that were commonly carried asymptomatically (SI Appendix, Table S2), such as E. coli, H. influenzae, and S. pneumoniae, having consistently high bystander proportions, and more rarely carried species such as S. pyogenes and N. gonorrhoeae, which are frequently associated with antibiotic-treated disease, having lower ones. Among drugs/drug classes, nitrofurantoin, used almost exclusively for UTIs, had low bystander proportions for common urinary tract pathogens, which frequently are the cause of nitrofurantoin treatment. In contrast, broad-spectrum drug classes such as beta-lactams, cephalosporins, and quinolones typically have high bystander proportions for most or all species considered, because they are used for a wide variety of conditions caused by a wide variety of species, as well as for treatment of conditions that are often nonbacterial.

Quantifying bystander selection for resistance for different antibiotic–species combinations has several potential applications. As previously discussed, mathematical transmission models of antibiotic prescribing and resistance commonly assume that bystander selection is the rule rather than the exception, and these findings confirm this has been a sensible assumption, at least for outpatient antibiotic use. For policy discussions, the high bystander proportions obtained here suggest that interventions to reduce antimicrobial use may have broad effects in reducing the strength of selection across a number of bacterial species, not only the ones involved in the pathogenesis of the disease targeted by such efforts. For example, improved adherence to guidelines on unnecessary antimicrobial prescribing might mainly affect prescribing for respiratory infections yet might reduce selection for resistance on potential pathogens that reside on the skin (e.g., S. aureus) or in the gut (e.g., E. coli and Klebsiella species), as well as on respiratory bacteria. In the area of antimicrobial stewardship, these findings suggest that each reduction in inappropriate antibiotic prescribing for a particular indication may have broad impacts across many species but may not dramatically reduce the exposures to antibiotics of any one species, as long as prescribing for other indications remains unchanged.

As discussed, another example of an intervention that can reduce antimicrobial prescribing is vaccination. Vaccines can reduce the incidence of resistant infections directly (by preventing disease from their target pathogens) and indirectly (by preventing the need for antibiotic prescribing, thereby protecting bystander bacteria from exposure to antibiotics that can promote resistance). High bystander proportions are seen here for broad-spectrum antibiotic classes that are frequently prescribed for respiratory infections, and respiratory infections (including otitis media) account for a large fraction of total antimicrobial use. These considerations suggest that vaccines against pathogens that cause respiratory infections, such as Bordetella pertussis, S. pneumoniae, influenza virus, and respiratory syncytial virus, may substantially reduce the exposure of a broad range of pathogenic bacterial species to antibiotics, via prevention of bystander selection. Notably, this includes vaccines that prevent viral respiratory infections, which are often inappropriately treated with antibiotics (17), and perhaps prevent bacterial secondary infections that might be appropriately treated if they occurred (18). We have described an approach for using estimates of bystander exposures to estimate how vaccines could reduce exposure across various nontarget pathogens. However, as noted by Sevilla et al. (19) in PNAS, quantifying the impact of vaccines on antimicrobial resistance is a complex task, and many components of such calculations will depend on the population, vaccine, and timescale considered, among other variables.

It is informative to consider the antimicrobial agents not included in our analysis. Most antimycobacterial agents have little effect on other bacterial species, while most broad-spectrum antibacterial classes are of little use against Mycobacterium tuberculosis. Quinolones are an exception to both rules—bystander selection has been documented both in treatment of what was thought to be bacterial pneumonia but was actually tuberculosis (20) and in treatment with quinolones in a tuberculosis ward promoting the spread of quinolone-resistant S. pneumoniae (21). Other exceptions include rifamycins, also commonly used for treatment of tuberculosis, and macrolides, which may be prescribed for Mycobacterium avium complex disease. With these exceptions, bystander selection by antimycobacterial drugs is expected to be limited, and bystander selection on M. tuberculosis is also expected to be limited. This is reflected in an appropriate focus for tuberculosis resistance management in ensuring adequate treatment to prevent emergence of resistance and prevent transmission, rather than on bystander-focused interventions. Additionally, antiviral agents, such as the neuraminidase inhibitor oseltamivir for influenza, have no substantial known activity against other components of the (bacterial) microbiome, so the rationale for prudent use of oseltamivir would include avoiding side effects and costs, but not avoiding selection for resistance. Beyond antiinfectives, a recent in vitro study found that 24% of 835 therapeutic compounds with molecular targets in human cells inhibited the growth of at least one bacterial species commonly found in the human gut microbiome (22). This work suggests that our focus on antimicrobials underestimates bystander selection for resistance, but further research is needed to elucidate which drug–species combinations may be prone to such effects.

This analysis has several limitations. First, all necessary inputs—incidence and etiology of bacterial infections, antibiotic prescribing practices, and composition of the microbial flora—are derived from different data sources and are highly heterogeneous, varying over time and by age, gender, and geographic location. Antibiotic prescribing additionally depends upon safety and toxicity profiles in certain populations. Microbiome diversity varies between and within individuals, depending on demographic characteristics, diet, and disease. For simplicity, we only consider age-group stratification to calculate population-level, average estimates. Carriage prevalences and etiologies are also applied uniformly across visits. This may bias our estimates depending on the extent of microbial ecological or etiological relationships. For example, if abundance (or lack) of organism A in the microbiome contributes to the pathogenicity of organism B, organism A may be more (or less) prone to bystander exposure of antibiotics used to treat the condition caused by organism B than we calculate. Additionally, while we estimate the impact on the organism at the species level due to data constraints, selection pressures may be more relevant at the strain level; for example, true bystander selection may be lower for infrequently carried, more pathogenic strains compared with our overall, species-level estimate.

Second, the limitations of the datasets used in our analysis also extend to our results. For example, the HMP was conducted in a restricted, healthy study population and prevalence estimates may not be generalizable to the US population. To our knowledge, a more extensive and nationally representative source of microbiome data is unavailable, and little is known about how microbiome composition among individuals with common outpatient conditions may differ from that of healthy individuals. Additionally, HMP samples a limited number of body sites and may exclude the most relevant colonization sites for some species of interest. For example, measurement of E. coli in the stool as a proxy for the large intestine likely contributes to its low carriage prevalence in HMP (66.3%), leading to underestimation of colonization and thus bystander proportion. Similarly, unavailability of samples from the anterior nares for some individuals may have led to underestimation of the carriage prevalence of S. aureus in HMP data (12.4%). Etiologic studies are burdensome and thus often conducted among very small populations; small sample sizes may miss infrequent causative agents of disease, but this is unlikely to have a substantial effect on our point estimates. GISP, used for the analysis of N. gonorrhoeae, is based on convenience sampling of male patients at sexually transmitted disease clinics. In 2016, the CDC estimated that 90.8% of patients in sexually transmitted disease clinics received the recommended azithromycin/ceftriaxone regimen compared with 79.8% of patients in other provider settings; the same relationship, though much weaker, was observed when comparing men to women (23). Since these differences are fairly small, any underestimation of bystander selection for N. gonorrhoeae due to the recommended antibiotics of azithromycin and ceftriaxone is likely to be minor. Additionally, although NAMCS/NHAMCS are unique in providing a large sample of outpatient visits with corresponding diagnoses and prescriptions, a direct link between diagnosis and prescription is unavailable—therefore, exposures may be incorrectly counted as “bystander” when the prescription was in fact written for a second diagnosis caused by the species of interest. In this analysis, we make no assumptions about linkage between prescriptions and diagnoses; instead, we attribute any antibiotic prescription to all of the diagnoses recorded in the same visit. This may bias the proportion of bystander exposures in either direction, depending on the antibiotic and species pair in question; use of an alternative tiered method for assigning diagnoses to visits (24) resulted in similar findings (98% of antibiotic class and species pairs were within 5% of the values reported here).

Finally, we use prescriptions as a proxy for exposure, which is itself a proxy for selective pressure. NAMCS/NHAMCS do not contain information on whether or not the prescriptions were filled; even after being filled, we have no information on compliance to the listed medications. Furthermore, little is known about how exposures of a particular antibiotic correspond to selection pressures. This may differ widely by antibiotic, regimen, organism, and body site. Strength of selective effects at different body sites will further vary based on pharmacodynamics, pharmacokinetics, and context (e.g., microbiome composition) (25). Most of the antibiotic classes considered here may be assumed to exert some selective pressure for resistance throughout the body. Direct evidence of selection on the gut normal flora by oral antibiotics has been reported (26). Likewise, selection on the flora of the upper respiratory tract is likely the rule for many of these classes, including macrolides, penicillins, cephalosporins, and trimethoprim/sulfamethoxazole because they are routinely prescribed for upper respiratory infections and are documented to affect bacterial carriage at in the nasopharynx (2729). Antibiotics in major classes including penicillins and cephalosporins (30), macrolides (31), and quinolones (32) have been detected in sweat, indicating that they can exert selection on skin flora. One exception to this general rule is nitrofurantoin, which tends to concentrate in the urinary tract and is accordingly used to treat UTIs. The relationship between drug concentration and selective pressure is also not straightforward, with subinhibitory concentrations likely playing an important role in selection for resistance (33). In general, the metric used in the present study, counting prescriptions as exposures, could be refined in studies of individual drugs, classes, microbes, or body sites to incorporate more biological detail, and deviations from the estimates shown here will be specific to the antibiotic and organism of interest.

The bystander proportions quantified in this analysis are a step toward better characterizing the dynamics of antibiotic resistance and should be considered in the development and prioritization of interventions. In this paper, we specifically address selective pressures for resistance among potential pathogens in the microbial flora due to outpatient antibiotic prescribing. Outpatient prescribing constitutes ∼90% of total antimicrobial volume for human health in developed countries (34, 35), but certainly further work is needed to consider the inpatient context as it affects nosocomial pathogens. In addition, off-target antibiotic exposures also contribute to resistance dynamics in other ways not captured in this analysis, including selection for resistance elements among nonpathogenic bacteria that may be horizontally transferred to pathogens and depletion of beneficial bacteria which play active roles in metabolism, pathogen resistance, and immune responses (36, 37). Research on the broader effects of antibiotic use on the microbiome is greatly needed to further understand the implications of bystander exposures on the spread of antibiotic resistance.

Materials and Methods

Data Sources.

All estimates except for N. gonorrhoeae were based on three main data sources: the NAMCS/NHAMCS collected by the National Center for Health Statistics, the HMP and other studies of carriage prevalence, and etiological studies.

The NAMCS and NHAMCS are cross-sectional national surveys designed to collect data on ambulatory care services provided at office-based physician practices, emergency, and hospital outpatient departments throughout the United States. At each sampled visit, patient characteristics (e.g., age), visit characteristics (e.g., reason for visit, diagnosis, prescriptions), and physician characteristics are recorded, including up to three diagnoses and up to eight prescribed medications. Sampling is based on a multistage probability sampling scheme. The most recent 2 y of fully available data for both NAMCS and NHAMCS (2010–2011) were used for this analysis. As the focus of our analysis was outpatient antibiotic use, visits that resulted in hospital or observation unit admission were excluded. Antibiotics were grouped into classes based on the Multum Lexicon system used in NAMCS/NHAMCS.

The first phase of the HMP consisted of collecting microbiome samples from 300 healthy individuals between the ages of 18 and 40 y at multiple timepoints across five major body sites: the nasal passages, oral cavity, skin, gastrointestinal tract, and urogenital tract. Microbial composition was characterized using MetaPhlAn2 (38), a taxonomic profiling method for whole-metagenomic shotgun samples. Prevalence estimates from HMP data were based on presence of the species at any body site. For children under 5 y old, carriage prevalences were compiled from primary sources in the literature (SI Appendix, Table S1). As individual carriage studies tended to collect samples from only one body site, carriage prevalences at each body site were estimated as an average across studies weighted by sample size, and overall prevalence was calculated assuming independence at each body site. This process was also used to estimate carriage prevalences of S. pyogenes and S. pneumoniae in the >5-y age group, as MetaPhlAn2 did not distinguish between these and closely related species (e.g., Streptococcus mitis and Streptococcus oralis). Etiologies for conditions of interest were based on etiologic studies cited in the medical resource UpToDate (SI Appendix, Table S3).

Calculations of Bystander Proportions.

A bystander exposure was defined as a prescription of antibiotic (or antibiotic class) a received by an individual carrying species s for a diagnosis of condition c that was not caused by s. Exposures were estimated on average at the population level. Let Bas be the proportion of bystander exposures of antibiotic a received by species s, equivalent to one minus the ratio of Nas, the number of exposures of antibiotic a received by species s for a case of some condition c that was caused by species s, and Tas, the total number of exposures of antibiotic a received by species s. Additionally, let dacg be the number of prescriptions of antibiotic a written for condition c in age group g, let pscg be the proportion of cases of condition c that are colonized with species s in age group g, and let escg be the proportion of cases of condition c caused by species s in age group g. The proportion pscg was calculated under the assumption that all individuals with condition c not caused by species s were colonized with species s at the group-specific prevalence estimated from HMP or other studies (denoted psg), while individuals with condition c caused by species s were colonized with species s with probability 1: pscg = escg + (1 − escg)psg. Since the inputs dacg, escg, and psg may be highly variable by age, estimates were summed over three age strata g (<1 y old, 1–5 y old, and over 5 y old). The proportion of bystander exposures for antibiotic a, species s, and condition c was calculated as follows:

Bas=1NasTas=1g=1Gc=1Cdacg×escgg=1Gc=1Cdacg×pscg+g=1Gdac¯g×psg.

Conditions were based on diagnostic categories delineated by Fleming-Dutra et al. (24) with the following exceptions: (i) “Other bacterial infections” includes all codes listed under “miscellaneous bacterial infections” plus other intestinal infectious diseases (ICD-9CM codes 001–008) but excludes a subset of infectious diseases (040–041, 130–139), mastoiditis (383), and peritonsillar abscess (475) and (ii) we include only cellulitis (681–682) from the category “Skin, cutaneous and mucosal infections.” The set of conditions C includes conditions for which antibiotic use was relatively high (>2% of weighted prescriptions; viral upper respiratory tract infection contributed the most, at 10.8% of weighted prescriptions) and reasonable estimates of escg were available. When diagnoses were excluded, this was most often due to one of these two limitations. Influenza was also included due to clear etiology and vaccination-related interest, and acne was included due to high tetracycline use for this indication. All cases with ICD-9CM code indicating the causative agent [e.g., 480: viral pneumonia, 481: streptococcal pneumonia (S. pneumoniae pneumonia)] were attributed to that agent.

The second term in the denominator of the proportion of bystander exposures accounts for exposures of antibiotic a that were not associated with any of the conditions in C, where dac¯g represents prescriptions of antibiotic a that occur at visits not associated with any condition in C. The use of psg in this term implies that our species of interest are rarely, if ever, causative agents for conditions that are not included in our analysis. Violations of this assumption will lead to overestimation of the proportion of bystander exposures. For comparison, we performed the analysis excluding this term (SI Appendix, Fig. S5) as lower-bound estimates of the proportion of bystander exposures for all species–antibiotic class pairs.

Confidence intervals were estimated by simulation. The proportion of bystander exposures was calculated for 1,000 random draws of dacg, psg, and escg based on empirically estimated distributions. Draws for dacg were based on the normal distribution, using variances calculated by the survey package in R (39). For HMP prevalence estimates, with A presences and B absences, random draws were simulated from a beta distribution with parameters (A+0.5, B+0.5), the posterior distribution using Jeffreys prior. Resampling was done similarly for etiological fractions. The 2.5th and 97.5th percentiles were utilized as the bounds of the 95% confidence interval. Table 1 provides a summary of notation used in this analysis.

Table 1.

Summary of notation with definitions, sources, and relevant figures and tables

Variable Definition Source Figure/table
dacg Number of prescriptions (using nationally representative weights) of antibiotic a associated with condition c in age group g NAMCS/NHAMCS 2010–2011 SI Appendix, Fig. S2 (unweighted)
dac¯g Number of prescriptions (using nationally representative weights) of antibiotic a associated with all conditions not explicitly included in analysis (c¯) in age group g NAMCS/NHAMCS 2010–2011 N/A
psg Carriage prevalence of species s in age group g HMP and carriage studies Fig. 1 and SI Appendix, Table S2
pscg Carriage prevalence of species s among individuals with condition c in age group g escg+(1escg)psg N/A
escg Proportion of cases of condition c in age group g caused by species s Etiologic studies Fig. 1 and SI Appendix, Table S3
Nas Number of exposures of antibiotic a experienced by species s for cases of conditions in C caused by species s g=1Gc=1Cdacg×escg N/A
Tas Total number of exposures of antibiotic a experienced by species s over all conditions g=1Gc=1Cdacg×pscg+g=1Gdac¯g×psg N/A
Bas Proportion of bystander exposures for antibiotic a and species s 1Nas/Tas Fig. 1 and SI Appendix, Figs. S1 and S4

N/A, not applicable.

Calculations of Bystander Proportion for N. gonorrhoeae.

Separate analyses were conducted for N. gonorrhoeae using additional data from the GISP (14). The same formula for the proportion of bystander exposures was applied. When GISP data were reported by clinic site, the weighted average by sample size over all clinics was used. Age groups were further stratified into those used by GISP (<20, 20–24, 25–29, 30–34, 35–39, 40–44, and >45 y old) for this analysis.

N. gonorrhoeae was assumed to be the causative agent for all gonorrhea cases only (escg = 1 for gonorrhea and 0 for all other conditions of interest). The number of prescriptions of antibiotic a written for all conditions except gonorrhea was calculated from NAMCS/NHAMCS as in the previous analyses. We include only prescriptions for patients >5 y old, as we expect gonococcal infection and carriage to be very rare among children age 5 y and below. For gonorrhea, the number of prescriptions of antibiotic a written in 2010–2011 was estimated by multiplying the total number of reported gonorrhea cases [309,341 in 2010 (40) and 321,849 in 2011 (41)] by the proportion of GISP participants treated with a. Prescriptions were counted equally regardless of dosage. Data on quinolone and cephalosporin prescribing only were available for 2010, while 2011 reports also included macrolides and tetracyclines. Therefore, bystander proportions for azithromycin and doxycycline are only available for 2011.

N. gonorrhoeae is identified in HMP, but the measured prevalence of 37%, with 95% of these identified in oral isolates, indicates that these data may include false positives. Miller et al. (42) report that prevalence of N. gonorrhoeae in urine samples from a nationally representative sample of young adults (National Longitudinal Study of Adolescent Health) was 0.64% among participants 20–21 y old, 0.47% among participants 22–23 y old, and 0.24% among participants 24–25 y old. Using the target population weights reported in table 1 of ref. 42, we estimated the carriage prevalence of N. gonorrhoeae to be 0.56% among participants 20–23 y old and 0.48% among participants 20–25 y old. We applied the mean of these two groups, 0.52%, as the carriage prevalence of N. gonorrhoeae in the 20–24 y age category used by GISP. For all other age groups designated by GISP, this prevalence was inflated by the relative proportion of GISP isolates from that age group. For example, in 2010, 31.4% of GISP isolates were sampled from individuals aged 20–24 y old, while 21.2% were from individuals aged 25–29 y old. The carriage prevalence among 20- to 24-y-olds was multiplied by 21.2%/31.4% to estimate carriage prevalence in the 25–29 y age category.

Impact of Vaccine.

To approximate the impact of a vaccine in reducing antimicrobial exposure of nontargeted species (e.g., E. coli for a pneumococcal vaccine), we initially assume as an input the observed reduction r in all-cause antimicrobial use in a particular age group, such as the 5.4% reduction in all-cause antibiotic use in a randomized PCV trial in 0- to 2-y-olds (5). From our analysis, we approximate values for 0- to 2-y-olds as the average of results from the 0–1 y and 1–5 y age groups. We reason as follows.

Table 2 shows the possible combinations of presence/absence of E. coli in a treated patient, and E. coli as cause or not cause of the treatment. One cell (absent, but causal) is empty because by assumption the species must be present to cause treatment. Let A, B, and D represent proportions of all treatments so A+B+D=1.

Table 2.

Classifying all-cause antibiotic treatments with respect to a potential bystander species, E. coli, as present or not, and cause of treatment or not

E. coli present in treated patient
E. coli is the cause of treatment No Yes
No A B
Yes D

—, not available.

In our example, the total treatment reduction, r, is 5.4% of all treatments. However, this reduction is unequally apportioned. All of the reduction occurs in categories A and B, because we assume that PCV would have no effect on the rate of treatment for a disease that was caused by E. coli. Define pEc as the prevalence of E. coli in the microbiome data for the relevant age group. Then, by our modeling assumptions, pEc=B/(A+B). Thus, the amount of treatment reduction in category B is rB/(A+B)=rpEc.

We seek the proportional reduction in B+D, the exposure of E. coli to treatment. D is unchanged, so the reduction is rpEc/(B+D)=rpEc/(1A). Defining the proportion of bystander exposures for E. coli to all antibiotics as Ball,Ec=B/(B+D)=B/(1A), some algebra yields the quantity we seek, the reduction in E. coli’s total (causal plus bystander) exposure to antibiotics attributable to a reduction r in all-cause antibiotic treatment from a vaccine that prevents no disease caused by E. coli:

rpEc1A=r(pEc+Ball,EcpEcBall,Ec).

Analogous calculations can be made for any other bacterial species for which disease is not reduced by the vaccine. For pathogens (e.g., H. influenzae) in which vaccination may cause a reduction in the amount of disease they cause [e.g., through indirectly preventing nonpneumococcal otitis media (15, 16)], this estimate would be a lower bound.

Data from the National Ambulatory Medical Care Survey and National Hospital Ambulatory Medical Care Survey (NAMCS/NHAMCS) are publicly available from the CDC (https://www.cdc.gov/nchs/ahcd/datasets_documentation_related.htm). Analysis code is available on GitHub (https://github.com/ctedijanto/Bystander-selection-outpatient).

Supplementary Material

Supplementary File

Acknowledgments

We thank Curtis Huttenhower, Eric Franzosa, and Jason Lloyd-Price for helpful comments and assistance with MetaPhlAn2 data. This work was supported by Grant U54GM088558 (Models of Infectious Disease Agent Study, Center for Communicable Disease Dynamics) from the National Institute of General Medical Sciences, Grant R01 AI132606 from the National Institute of Allergy and Infectious Diseases (to Y.H.G.), Grant CK000538-01 from the Centers for Disease Control and Prevention, and the Doris Duke Charitable Foundation. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institute of General Medical Sciences, National Institutes of Health, National Institute of Allergy and Infectious Diseases, CDC, Department of Health and Human Services, or the Doris Duke Charitable Foundation.

Footnotes

Conflict of interest statement: M.L. has received consulting income from Affinivax, Antigen Discovery, Merck, and Pfizer and research grants through Harvard School of Public Health from Pfizer and PATH.

This article is a PNAS Direct Submission.

See companion Introduction to the Antimicrobial Resistance and the Role of Vaccines Special Feature on page 12868.

Data deposition: All code is available on GitHub (https://github.com/ctedijanto/Bystander-selection-outpatient).

This article contains supporting information online at www.pnas.org/lookup/suppl/doi:10.1073/pnas.1810840115/-/DCSupplemental.

References

  • 1.Sullivan A, Edlund C, Nord CE. Effect of antimicrobial agents on the ecological balance of human microflora. Lancet Infect Dis. 2001;1:101–114. doi: 10.1016/S1473-3099(01)00066-4. [DOI] [PubMed] [Google Scholar]
  • 2.Gustafsson I, et al. Bacteria with increased mutation frequency and antibiotic resistance are enriched in the commensal flora of patients with high antibiotic usage. J Antimicrob Chemother. 2003;52:645–650. doi: 10.1093/jac/dkg427. [DOI] [PubMed] [Google Scholar]
  • 3.Lindgren M, Löfmark S, Edlund C, Huovinen P, Jalava J. Prolonged impact of a one-week course of clindamycin on Enterococcus spp. in human normal microbiota. Scand J Infect Dis. 2009;41:215–219. doi: 10.1080/00365540802651897. [DOI] [PubMed] [Google Scholar]
  • 4.Nyberg SD, et al. Long-term antimicrobial resistance in Escherichia coli from human intestinal microbiota after administration of clindamycin. Scand J Infect Dis. 2007;39:514–520. doi: 10.1080/00365540701199790. [DOI] [PubMed] [Google Scholar]
  • 5.Fireman B, et al. Impact of the pneumococcal conjugate vaccine on otitis media. Pediatr Infect Dis J. 2003;22:10–16. doi: 10.1097/00006454-200301000-00006. [DOI] [PubMed] [Google Scholar]
  • 6.Lau WCY, et al. Impact of pneumococcal conjugate vaccines on childhood otitis media in the United Kingdom. Vaccine. 2015;33:5072–5079. doi: 10.1016/j.vaccine.2015.08.022. [DOI] [PubMed] [Google Scholar]
  • 7.Cobey S, et al. Host population structure and treatment frequency maintain balancing selection on drug resistance. J R Soc Interface. 2017;14:20170295. doi: 10.1098/rsif.2017.0295. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.Bonhoeffer S, Lipsitch M, Levin BR. Evaluating treatment protocols to prevent antibiotic resistance. Proc Natl Acad Sci USA. 1997;94:12106–12111. doi: 10.1073/pnas.94.22.12106. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9.Davies NG, Flasche S, Jit M, Atkins KE. November 10, 2017. Within-host dynamics explain the coexistence of antibiotic-sensitive and resistant bacteria. bioRxiv:10.1101/217232.
  • 10.Atkins KE, et al. Use of mathematical modelling to assess the impact of vaccines on antibiotic resistance. Lancet Infect Dis. 2018;18:e204–e213. doi: 10.1016/S1473-3099(17)30478-4. [DOI] [PubMed] [Google Scholar]
  • 11.Lipsitch M, Siber GR. How can vaccines contribute to solving the antimicrobial resistance problem? MBio. 2016;7:e00428-16. doi: 10.1128/mBio.00428-16. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12.Olesen SW, et al. Azithromycin susceptibility in Neisseria gonorrhoeae and seasonal macrolide use. J Infect Dis. September 15, 2018 doi: 10.1093/infdis/jiy551. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13.Wind CM, et al. Decreased azithromycin susceptibility of Neisseria gonorrhoeae isolates in patients recently treated with azithromycin. Clin Infect Dis. 2017;65:37–45. doi: 10.1093/cid/cix249. [DOI] [PubMed] [Google Scholar]
  • 14.CDC 2018 Sexually transmitted diseases (STDs). Gonococcal Isolation Surveillance Project (GISP). Available at https://www.cdc.gov/std/gisp/default.htm. Accessed June 12, 2018.
  • 15.Dagan R, Pelton S, Bakaletz L, Cohen R. Prevention of early episodes of otitis media by pneumococcal vaccines might reduce progression to complex disease. Lancet Infect Dis. 2016;16:480–492. doi: 10.1016/S1473-3099(15)00549-6. [DOI] [PubMed] [Google Scholar]
  • 16.Ben-Shimol S, et al. Impact of widespread introduction of pneumococcal conjugate vaccines on pneumococcal and nonpneumococcal otitis media. Clin Infect Dis. 2016;63:611–618. doi: 10.1093/cid/ciw347. [DOI] [PubMed] [Google Scholar]
  • 17.Goyal MK, et al. Pediatric Care Applied Research Network (PECARN) Racial and ethnic differences in antibiotic use for viral illness in emergency departments. Pediatrics. 2017;140:e20170203. doi: 10.1542/peds.2017-0203. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18.Loeb M. Community-acquired pneumonia. BMJ Clin Evid. 2010;2010:1503. [PMC free article] [PubMed] [Google Scholar]
  • 19.Sevilla JP, Bloom DE, Cadarette D, Jit M, Lipsitch M. Toward economic evaluation of the value of vaccines and other health technologies in addressing AMR. Proc Natl Acad Sci USA. 2018;115:12911–12919. doi: 10.1073/pnas.1717161115. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.Shen G-H, et al. Does empirical treatment of community-acquired pneumonia with fluoroquinolones delay tuberculosis treatment and result in fluoroquinolone resistance in Mycobacterium tuberculosis? Controversies and solutions. Int J Antimicrob Agents. 2012;39:201–205. doi: 10.1016/j.ijantimicag.2011.11.014. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21.von Gottberg A, et al. Group for Enteric, Respiratory and Meningeal Disease Surveillance in South Africa (GERMS-SA) Emergence of levofloxacin-non-susceptible Streptococcus pneumoniae and treatment for multidrug-resistant tuberculosis in children in South Africa: A cohort observational surveillance study. Lancet. 2008;371:1108–1113. doi: 10.1016/S0140-6736(08)60350-5. [DOI] [PubMed] [Google Scholar]
  • 22.Maier L, et al. Extensive impact of non-antibiotic drugs on human gut bacteria. Nature. 2018;555:623–628. doi: 10.1038/nature25979. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23.Weston EJ, Workowski K, Torrone E, Weinstock H, Stenger MR. Adherence to CDC recommendations for the treatment of uncomplicated gonorrhea–STD surveillance network, United States, 2016. MMWR Morb Mortal Wkly Rep. 2018;67:473–476. doi: 10.15585/mmwr.mm6716a4. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24.Fleming-Dutra KE, et al. Prevalence of inappropriate antibiotic prescriptions among US ambulatory care visits, 2010-2011. JAMA. 2016;315:1864–1873. doi: 10.1001/jama.2016.4151. [DOI] [PubMed] [Google Scholar]
  • 25.Ferrer M, Méndez-García C, Rojo D, Barbas C, Moya A. Antibiotic use and microbiome function. Biochem Pharmacol. 2017;134:114–126. doi: 10.1016/j.bcp.2016.09.007. [DOI] [PubMed] [Google Scholar]
  • 26.Lange K, Buerger M, Stallmach A, Bruns T. Effects of antibiotics on gut microbiota. Dig Dis. 2016;34:260–268. doi: 10.1159/000443360. [DOI] [PubMed] [Google Scholar]
  • 27.Dagan R, Barkai G, Leibovitz E, Dreifuss E, Greenberg D. Will reduction of antibiotic use reduce antibiotic resistance?: The pneumococcus paradigm. Pediatr Infect Dis J. 2006;25:981–986. doi: 10.1097/01.inf.0000239266.20642.26. [DOI] [PubMed] [Google Scholar]
  • 28.Samore MH, et al. Mechanisms by which antibiotics promote dissemination of resistant pneumococci in human populations. Am J Epidemiol. 2006;163:160–170. doi: 10.1093/aje/kwj021. [DOI] [PubMed] [Google Scholar]
  • 29.Feikin DR, et al. Increased carriage of trimethoprim/sulfamethoxazole-resistant Streptococcus pneumoniae in Malawian children after treatment for malaria with sulfadoxine/pyrimethamine. J Infect Dis. 2000;181:1501–1505. doi: 10.1086/315382. [DOI] [PubMed] [Google Scholar]
  • 30.Høiby N, Pers C, Johansen HK, Hansen H. The Copenhagen Study Group on Antibiotics in Sweat Excretion of β-lactam antibiotics in sweat–A neglected mechanism for development of antibiotic resistance? Antimicrob Agents Chemother. 2000;44:2855–2857. doi: 10.1128/aac.44.10.2855-2857.2000. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 31.Strausbaugh LJ, Bolton WK, Dilworth JA, Guerrant RL, Sande MA. Comparative pharmacology of josamycin and erythromycin stearate. Antimicrob Agents Chemother. 1976;10:450–456. doi: 10.1128/aac.10.3.450. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 32.Høiby N, et al. Excretion of ciprofloxacin in sweat and multiresistant Staphylococcus epidermidis. Lancet. 1997;349:167–169. doi: 10.1016/S0140-6736(96)09229-X. [DOI] [PubMed] [Google Scholar]
  • 33.Andersson DI, Hughes D. Microbiological effects of sublethal levels of antibiotics. Nat Rev Microbiol. 2014;12:465–478. doi: 10.1038/nrmicro3270. [DOI] [PubMed] [Google Scholar]
  • 34.SWEDRES/SVARM . Consumption of antibiotics and occurrence of antibiotic resistance in Sweden. Public Health Agency of Sweden and National Veterinary Institute; Solna Municipality, Sweden: 2014. [Google Scholar]
  • 35.Public Health Agency of Canada . Canadian antimicrobial resistance surveillance system 2017 report. Public Health Agency of Canada; Ottawa, ON, Canada: 2017. [Google Scholar]
  • 36.Dethlefsen L, Relman DA. Incomplete recovery and individualized responses of the human distal gut microbiota to repeated antibiotic perturbation. Proc Natl Acad Sci USA. 2011;108:4554–4561. doi: 10.1073/pnas.1000087107. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 37.Dethlefsen L, Huse S, Sogin ML, Relman DA. The pervasive effects of an antibiotic on the human gut microbiota, as revealed by deep 16S rRNA sequencing. PLoS Biol. 2008;6:e280. doi: 10.1371/journal.pbio.0060280. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 38.Truong DT, et al. MetaPhlAn2 for enhanced metagenomic taxonomic profiling. Nat Methods. 2015;12:902–903. doi: 10.1038/nmeth.3589. [DOI] [PubMed] [Google Scholar]
  • 39.Lumley T. 2016 Survey: Analysis of complex survey samples. Available at https://rdrr.io/rforge/survey/. Accessed October 7, 2017.
  • 40.CDC 2011 2010 sexually transmitted diseases surveillance: Gonorrhea. Available at https://www.cdc.gov/std/stats10/gonorrhea.htm. Accessed June 12, 2018.
  • 41.CDC 2012 2011 sexually transmitted diseases surveillance: Gonorrhea. Available at https://www.cdc.gov/std/stats11/gonorrhea.htm. Accessed June 12, 2018.
  • 42.Miller WC, et al. Prevalence of chlamydial and gonococcal infections among young adults in the United States. JAMA. 2004;291:2229–2236. doi: 10.1001/jama.291.18.2229. [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplementary File

Articles from Proceedings of the National Academy of Sciences of the United States of America are provided here courtesy of National Academy of Sciences

RESOURCES