Rethinking Antimicrobial Resistance Surveillance: A Role for Lot Quality Assurance Sampling

Abstract

Global surveillance of antimicrobial resistance (AMR) is a key component of the 68th World Health Assembly Global Action Plan on AMR. Laboratory-based surveillance is inherently biased and lacks local relevance due to aggregation of data. We assessed the feasibility, sensitivity, and affordability of a population-based AMR survey using lot quality assurance sampling (LQAS), which classifies a population as having a high or low prevalence of AMR based on a priori defined criteria. Three studies were carried out in Medan and Bandung, Indonesia, between April 2014 and June 2017. LQAS classifications for 15 antibiotics were compared with AMR estimates from a conventional population-based survey, with an assessment of the cost of a single LQAS classification using microcosting methodology, among patients suspected of urinary tract infection at 11 sites in Indonesia. The sensitivity of LQAS was above 98%. The approach detected local variation in the prevalence of AMR across sites. Time to reach LQAS results ranged from 47 to 138 days. The average cost of an LQAS classification in a single facility was US$466. The findings indicate that LQAS-based AMR survey is a feasible, sensitive, and affordable strategy for population-based AMR surveys, providing essential data to inform local empirical treatment guidelines and antimicrobial stewardship efforts.

The global threat of antimicrobial resistance is underlined by the 68th World Health Assembly’s adoption of the Global Action Plan on Antimicrobial Resistance (AMR) and is of particular concern in low- and middle-income countries (1, 2). One of the pillars of the plan is to support national strategies through enhanced global surveillance. The Global AMR Surveillance System proposes laboratory-based surveillance in which results of routine culture and susceptibility testing of diagnostic clinical samples are used to estimate the prevalence of AMR, as an initial step toward global AMR surveillance (3). This strategy assumes that samples are routinely submitted for microbiological diagnostics for all patients suspected of an infection in a given setting. Unfortunately, this is rarely the case, particularly in low- and middle-income countries where access to quality microbiology diagnostics is poor (4), resulting in selection bias in AMR estimates (5, 6). In addition, the recommended aggregation of national data is typically not representative of local care settings, further limiting its relevance to inform local empirical treatment choices.

Population-based surveillance is a preferred strategy, and its place in the “understanding of the effect of antimicrobial resistance on human health” (7, p 241) has been acknowledged in an assessment of the early phase of the Global AMR Surveillance System implementation. However, population-based surveillance is time-, labor-, and cost-intensive. There is a need for rapid, feasible, and affordable surveillance strategies that can reliably inform local and national empirical treatment guidelines (8).

While conventional population-based surveillance yields a prevalence estimate with a certain precision, an approach that yields a classification of the AMR prevalence as “high” or “low” might be sufficient to guide empirical treatment decisions. Such a classification approach in general needs a far lower sample size than a conventional prevalence survey, thereby increasing speed, feasibility, and affordability, and fits clinical practice and guidelines (9). One classification approach is lot quality assurance sampling (LQAS), a methodology developed in the manufacturing industry to assess the quality of a batch (lot). The classification of a batch as acceptable or unacceptable is based on the assessment of a small number of goods from the batch, the frequency of faulty items, and an upper threshold of the frequency above which the batch is deemed of unacceptable quality (decision rule) (10–13). Key to LQAS is the definition of thresholds and allowable probability of misclassification, both of which dictate the sample size required for a classification. In the context of AMR, the upper threshold is the prevalence of resistance against an antibiotic above which empirical use of this drug is no longer justified. The lower threshold delimits the range of the unknown AMR prevalence in which misclassification can occur. Misclassification refers to classifying a population with a low prevalence of AMR as having a high prevalence and vice versa. With a lot being a specific health facility or a district, LQAS-based AMR surveillance could provide locally relevant data that cannot be obtained with a conventional prevalence survey without a large enough sample size for each specific setting.

We tested and applied an LQAS-based approach to assess the prevalence of AMR in inpatients and outpatients suspected of urinary tract infection (UTI) in Indonesia, with the aims to: 1) estimate test characteristics for identifying populations with a high prevalence of AMR in UTI pathogens; 2) provide an LQAS classification for 15 antibiotics in 11 different settings; and 3) estimate the cost of obtaining an LQAS classification in a single health facility.

The findings of the study are relevant beyond the initial geographical setting and the syndrome under investigation. Being a sampling and analytical approach, LQAS-based AMR surveys can be implemented in any setting that requires local information on the prevalence of AMR for any given population or clinical syndrome.

METHODS

Assessment of LQAS test characteristics

We used 6 LQAS definitions that differed in their lower and upper thresholds but had identical allowable probability of misclassification (misclassifying high resistance: 5%; misclassifying low resistance: 10%). The definitions used a single sampling plan, leading to a static 2-way LQAS design. Each definition was projected on the data from a conventional AMR survey conducted in an outpatient setting between April 2014 and May 2015 in Medan and Bandung, Indonesia, as described elsewhere (14). The data contain the prevalence of AMR in Escherichia coli and Klebsiella pneumoniae isolates from urine specimens for 13 antibiotics. We used these data as 13 different “lots” of equal size, with a known true prevalence of AMR, from which we drew the required sample size, and we classified the draw based on the decision rule for each of the LQAS definitions. We repeated this exercise 1,000 times with replacement, providing 1,000 LQAS classifications for each of the 13 “lots,” after which we calculated the sensitivity and specificity of each LQAS definition (15). Sensitivity was defined as percentage of draws accurately classified as “high resistance,” while specificity was defined as the percentage of draws accurately classified as “low resistance.”

LQAS-based AMR survey

Setting and population

The study was carried out in 7 outpatient facilities (Medan and Bandung, in Indonesia) and 4 inpatient wards (Medan), which were deliberately selected based on their participation in the conventional AMR survey or their willingness to participate as a new site (Web Table 1, available at https://academic.oup.com/aje). These study sites represent primary and secondary care clinics in the public and private sector, as well as tertiary referral hospitals. The procedures for screening and inclusion of patients and for culture and antibiotic susceptibility testing (AST) were similar to those of our conventional AMR survey (14). In brief, consecutive patients suspected of a UTI based on signs and symptoms were eligible to join the study, and they were enrolled after providing written informed consent. The definition of suspected UTI was based on the current guidelines from the US Centers for Disease Control and Prevention (Web Table 2) (16). Inclusion of patients in a particular facility or hospital ward continued until the required sample size for the number of clinical isolates was reached.

Study parameters

The criteria of the LQAS-based survey were defined during a workshop with clinicians in Medan and Bandung. This included the antibiotics to be assessed, the upper threshold of resistance above which an antibiotic should not be used for the empirical treatment of UTI (20% for each selected antibiotic), and the maximum allowable probability of misclassification (15%). The participants agreed on a single sampling plan for an LQAS definition of 5%–20%, resulting in a required sample size at each facility/ward of 44 isolates, and a decision rule of 5. Given that E. coli and K. pneumoniae cause the majority of UTIs, these 2 pathogens were combined for reaching the required sample size.

Data collection

Demographic and clinical information was collected through an interview directly before or after the clinical consultation in the outpatient setting, or during hospital admission using an electronic clinical report form preinstalled on a mobile phone. Laboratory data other than AST results were likewise collected through electronic clinical report forms, all of which were uploaded to a secure server at the end of each day.

Laboratory procedures

Enrolled patients were requested to provide a urine specimen (midstream or through sterile aspiration from catheter tube) for a urinary dipstick analysis. Any presence of leukocyte esterase and/or nitrite was considered a positive dipstick test result. Urine specimens with a positive dipstick test result were submitted for urine culture in the accredited hospital microbiology laboratories of Dr. Hasan Sadikin General Hospital (Bandung) and Adam Malik Hospital (Medan). We defined culture positivity when a minimum of 10³ colony-forming units per ml were present, to account for the considerable number of patients suspected to have used antimicrobial treatment prior to urine sampling, which could affect culture yield. This colony count is predictive for bladder bacteriuria of E. coli (17). Colonies suspected to be E. coli or K. pneumoniae were identified using standard biochemical tests and submitted to AST.

Antimicrobial susceptibility testing

We used the disc diffusion method for AST according to Clinical and Laboratory Standards Institute guidelines, the results of which were interpreted according to breakpoints from the guideline version of 2016 (18). An intermediate test result was considered resistant. European Committee on Antimicrobial Susceptibility Testing recommendations were used for fosfomycin testing (19). Inhibition zones were measured using the imaging and measuring tools of a digital camera developed by BD Kiestra (Drachten, the Netherlands).

Data analysis

Data analysis is based on the first 44 isolates in each of the sites. The primary outcome of the study was the site-specific LQAS classification for each of the antibiotics tested based on the LQAS definition with a lower threshold of 5% and an upper threshold of 20%, and a decision rule of 5. We assessed the time between first screening event and the final AST result for the 44th isolate as a measure for throughput time of the LQAS-based AMR survey.

Data quality

The conventional and LQAS-based surveys employed identical data quality strategies. We carried out weekly assessments of missing or inconsistent data, after which corrections were implemented by returning to the primary data. The quality of bacterial cultures, isolate identification, and AST was monitored through telemicrobiology, allowing for digital sharing and analysis of high-resolution images of primary cultures and susceptibility test results through the internet (20). This feature facilitated weekly virtual laboratory rounds between Amsterdam and Indonesia, in which quality issues with respect to culture and susceptibility testing could be identified and corrected.

LQAS cost study

The primary outcome in the cost study was the cost of obtaining an LQAS classification at a single facility. The complete clinical and laboratory process, broken down into a priori defined tasks, was ascertained at 2 deliberately selected outpatient facilities in Medan; tasks were observed and timed multiple times during a period of 3 consecutive weeks. We used a microcosting approach with a mixed bottom-up and top-down approach from a health-care provider’s perspective, the methodology of which was proposed by the Global Health Cost Consortium (21). The sources used for cost estimates are listed in Web Table 3. The cost of each task was projected on the participant and specimen flow in each of the 7 outpatient facilities in the prospective LQAS-based AMR survey to obtain a cost per site-specific LQAS classification (Web Figure 1).

We performed 3 sensitivity analyses: 1) full staff cost for the time employed (top-down approach) as opposed to actual time spent (bottom-up approach); 2) a reduction in the number of antibiotics tested from 15 to 5; and 3) the presence of concessional pricing (20% reduction) for antibiotic discs used in AST. Costs are expressed in US dollars, with an exchange rate of 13,300 Indonesian rupiah (May 1, 2017).

Ethics statement

All 3 study components were part of a single protocol, which was approved by the University of Sumatera Utara Faculty of Medicine Ethics Committee, H. Adam Malik General Hospital Research Committee, Universitas Padjadjaran Faculty of Medicine Ethics Committee, and Dr. Hasan Sadikin General Hospital Research Committee. Written informed consent was obtained from each participant.

The reporting of the study followed the Strengthening the Reporting of Observational Studies in Epidemiology guidelines for cross-sectional studies (22).

RESULTS

LQAS test characteristics

The prevalence of AMR in the conventional AMR survey in an outpatient setting carried out between April 2014 and May 2015 ranged from 2.4% (95% confidence interval: 0.6, 4.2) for fosfomycin to 85.4% (95% confidence interval: 81.2, 89.5) for ampicillin (Web Table 4). For all LQAS definitions, the sensitivity (adequate classification of high resistance) was above 98%, while the specificity (adequate classification of low resistance) was above 80%, except for the 2%–10% LQAS definition, with a specificity of 44% (Table 1). The corresponding operator curves depicted in Web Figure 2 show that hardly any draw is classified as “high resistance” when the true prevalence of AMR is below the lower LQAS threshold. Similarly, almost all draws are classified as “high resistance” when the true prevalence of AMR is above the upper LQAS threshold. The narrower the distance between lower and upper threshold of the LQAS definition, the steeper the operator curve and the smaller the range of the true prevalence of AMR at which misclassification occurs.

Table 1.

Sensitivity and Specificity of Different Lot Quality Assurance Sampling Definitions for Classifying Outpatient Clinics, Using Data From Conventional Antimicrobial Resistance Surveillance, Medan and Bandung, Indonesia, 2014–2016

LQAS Definition, %	Required Sample Size, No.	Sensitivitya, %	Specificityb, %
2–10c	76	100.0	44.1
5–20	44	99.9	85.0
10–20	112	100.0	98.9
10–30	37	99.6	85.2
20–50	23	98.8	80.7
30–50	53	99.9	87.1

Abbreviations: AMR, antimicrobial resistance; LQAS, lot quality assurance sampling.

^a Percentage of draws accurately classified as “high resistance.”

^b Percentage of draws accurately classified as “low resistance.”

^c The upper value indicates the upper threshold of resistance prevalence above which an antibiotic should not be used for the empirical treatment of patients suspected of a urinary tract infection. The lower and upper thresholds together indicate the range of the true but unknown AMR prevalence in which misclassification is allowed to occur.

LQAS-based AMR survey

Data collection took place from September 2016 to June 2017. A total of 4,029 participants were screened, of whom 2,736 (67.9%) were eligible, and 2,724 patients (99.6% of those eligible) consented to enrollment (Figure 1). Patient characteristics and the frequency of signs and symptoms are summarized in Table 2.

Figure 1.

Patient disposition in lot quality assurance sampling–based antimicrobial resistance survey, among outpatients and inpatients in Medan and Bandung, Indonesia, September 2016 to June 2017. Bottom row indicates number of isolates; all other rows indicate number of individuals.

Table 2.

Characteristics of Enrolled Participants in a Lot Quality Assurance Sampling–Based Antimicrobial Resistance Survey Among Outpatients and Inpatients in Medan and Bandung, Indonesia, 2016–2017

Characteristic	Outpatient Service						Inpatient Service
	Urology (n = 1,034)		Obstetrics/Gynecology (n = 681)		Internal Medicine (n = 298)		Internal Medicine (n = 175)		Neurology (n = 201)		Surgery (n = 221)		Obstetrics/Gynecology (n = 114)
	No.	%	No.	%	No.	%	No.	%	No.	%	No.	%	No.	%
Sex
Male	811	78.4	N/A		155	52.0	94	53.7	111	55.2	132	59.7	N/A
Missing	5	0.5	3	0.5	1	0.3	0	0.0	0	0.0	0	0.0	1	0.9
Age group, years
18–24	33	3.2	79	11.6	9	2.0	4	2.3	20	10.0	25	11.3	13	11.4
25–34	59	5.7	391	57.4	15	5.0	12	6.9	12	6.0	20	9.0	29	25.4
35–44	102	9.9	162	23.8	36	12.1	22	12.6	28	13.9	43	19.5	23	20.2
45–54	168	16.2	41	6.0	90	30.2	49	28.0	51	25.4	58	26.2	27	23.7
55–64	251	24.3	3	0.4	104	34.9	56	32.0	56	27.9	48	21.7	16	14.0
≥65	416	40.2	4	0.6	43	14.4	32	18.3	34	16.9	27	12.2	5	4.4
Missing	5	0.5	1	0.2	1	0.3	0	0.0	0	0.0	0	0.0	1	0.9
Screening symptoms
Dysuria	531	51.4	91	13.4	184	61.7	19	10.9	3	1.5	20	9.1	4	3.5
Frequency	787	76.1	548	80.5	122	40.9	27	15.4	5	2.5	20	9.1	5	4.4
Urgency	160	15.5	37	5.4	34	11.4	11	6.3	2	1.0	1	0.5	4	3.5
Suprapubic pain	475	45.9	423	62.1	150	50.3	131	74.9	158	78.6	178	80.5	85	74.6
Costovertebral pain	484	46.8	452	66.4	17	5.7	97	55.4	123	61.2	128	57.9	44	38.6
Pyuria	N/A	N/A	N/A	N/A	N/A	N/A	116	66.3	174	86.6	152	68.8	51	44.7
Hematuria	75	7.3	6	0.9	4	1.3	16	9.1	18	9.0	39	17.7	20	17.5

Abbreviation: N/A, not applicable.

A positive dipstick test result was obtained from 1,233 (61.3%) participants attending an outpatient facility, yielding 457 (37.1%) positive cultures. In hospitalized patients, a positive dipstick test result was obtained for 538 (75.7%) participants, yielding 241 (44.8%) positive cultures (Figure 1).

All sites received an LQAS classification of “high resistance” for ciprofloxacin, levofloxacin, and cotrimoxazol, the drugs of first choice for the empirical treatment of UTI in Indonesia (Figure 2). Nearly all sites were classified as “high resistance” for nitrofurantoin, amoxicillin/clavulanic acid, and cefixime, representing alternative oral antibiotics for UTI treatment. The antibiotics for which the majority of sites received an LQAS classification of “low resistance” were amikacin, fosfomycin, and carbapenems. The results indicate local variation in the LQAS classification for certain antibiotics. Fifty percent of the urology outpatient facilities were classified as “high resistance” for amikacin compared with none of the other outpatient facilities. All of the hospital wards were classified as “high resistance” for ertapenem, compared with 2 out of 7 outpatient clinics. Time between first screening event and generation of the 44th quality-assured AST result ranged from 47 to 112 days in outpatient facilities, and from 85 to 138 days in hospital wards (Figure 2).

Figure 2.

Lot quality assurance sampling classification according to site and antibiotic, with time to reach classification in outpatient clinics and inpatient wards in Medan and Bandung, Indonesia, September 2016 to June 2017. Dark grey indicates high resistance, and light grey indicates low resistance. IM, internal medicine; Neuro, neurology; OBGYN, obstetrics/gynecology.

Cost study

Costing data were collected in May 2017 at 2 outpatient facilities (one for internal medicine and one for obstetrics and gynecology). The average cost for a site-specific LQAS classification for 15 antibiotics was $466 (range, 401–514) (Figure 3A). The cost of consumables needed in the clinic and laboratory was the main factor determining this average cost (68.1%) (Figure 3B). Using time employed rather than time spent by staff in the calculation increased the cost for an LQAS classification by 27% to US$643. Reducing the number of antibiotics in AST procedures to 5, or a 20% reduction in price on antibiotic discs for AST, reduced the cost for an LQAS classification by 7% and 2%, respectively.

Figure 3.

Absolute and relative costs (US dollars) for site-specific lot quality assurance sampling (LQAS) classification in outpatient clinics in Medan and Bandung, Indonesia, May 2017. A) Absolute costs ($). B) Relative costs (%). IM, internal medicine; O1–O2, obstetrics clinics 1–2; U1–U4, urology clinics 1–4.

DISCUSSION

Our study shows that an LQAS-based AMR survey has excellent sensitivity for the identification of health-care settings with a high prevalence of AMR in UTI in Indonesia. The approach allows the identification of local variations in AMR prevalence while having a short throughput time and affordable costs.

These findings are welcome news in an era in which the importance of AMR surveillance as an essential tool for antimicrobial stewardship cannot be overstated. The shift from initial national laboratory-based AMR surveillance toward population-based surveillance of clinical syndromes, as formulated in the Global AMR Surveillance System (3), is feasible when using an LQAS approach. The classification framework can overcome multiple hurdles encountered in the conventional estimation framework used in assessing the prevalence of AMR at the population level, the main being the large sample size required and its associated long duration and high costs. Although the LQAS approach does not give a precise estimate of the prevalence of AMR, its binary result (low or high prevalence) fits well with empirical clinical management and treatment guidelines (9).

Insight into locally prevailing resistance patterns in specific clinical syndromes through an LQAS-based AMR survey optimizes local empirical therapy. In addition to improving clinical outcomes of patients, this curbs further selection and spread of AMR locally. These benefits cannot be obtained with laboratory-based or conventional population-based AMR surveillance because of selection bias and lack of locally relevant data, respectively.

LQAS-based AMR surveys can be implemented at sentinel sites where the strategy can be repeated at regular intervals to assess changing trends and impact of interventions or to identify early development of resistance after the introduction of new drugs. Such a utility is currently lacking and could be extremely valuable in settings where microbiology capacity is limited, as in many low- and middle-income countries, or in settings where empirical treatment is the norm, as in primary care settings around the globe. In addition, if sites for LQAS-based AMR surveillance are selected through a sampling scheme that takes into account the size of the population in different health facilities (proportional-to-population-size sampling), it is possible to provide an overall AMR prevalence estimate by combining the site-specific LQAS results (23). When using such an approach, local classifications that are most informative for clinical management, as well as regional or national estimates that are important for policy makers, can be obtained simultaneously.

In our 11 sites, the overall cost of an LQAS classification, including 15 antibiotics, ranged between $403 and $514. Whether or not this is affordable depends largely on who will need to bear these costs and whether these costs weigh up against the expected benefits. We did not collect cost estimates of conventional surveillance, precluding direct comparisons with the cost of LQAS-based AMR surveillance. However, nearly all activities in the LQAS-based surveillance need to be carried out for a conventional surveillance, for which the same staff and utilities would be used in both Medan and Bandung. The cost difference between conventional surveillance and LQAS-based surveillance will therefore be largely driven by differences in sample size. A conventional AMR prevalence survey estimating 20% resistance with a 5% absolute precision will be 9 times larger than the current LQAS-based surveillance, making the latter potentially cost-effective.

LQAS-based surveys in the field of drug resistance have been used previously. Studies showed that an LQAS approach could retrospectively identify local variation in drug-resistant tuberculosis that was “hidden” in the overall conventional estimates, and could prospectively be used to survey a population for the prevalence of drug-resistant tuberculosis (24–26). Within the field of human immunodeficiency virus, there has been a strong guidance to use threshold surveys to assess the prevalence of transmitted drug resistance. However, LQAS designs were dismissed—it was argued that these designs did not result in sample size reductions in areas with anticipated low prevalence of transmitted drug resistance (27). Instead, a truncated sampling strategy was proposed and implemented by the World Health Organization (27). Field reports of these surveys identified problems with obtaining adequate sample sizes in this approach (28, 29). The latest guidelines on measuring transmitted resistance to human immunodeficiency virus drugs do not mention the threshold surveys, while sample size calculations are based on a single estimate with a preferred precision, using cluster sampling to enroll health facilities (30, 31). We used an identical analytical approach as described in the present study to validate LQAS-based surveillance with conventional surveillance of the prevalence of AMR in patients with suspected UTI using data from a sentinel network of Dutch general practitioners (15).

The clinical workshops in Medan and Bandung defined an upper threshold for resistance of 20% irrespective of the antibiotic and setting. The appropriateness of empirical use of an antibiotic is defined by clinical syndrome, severity of symptoms, setting, and prior antimicrobial treatment, among other factors, which typically should result in variations in thresholds of resistance. However, Indonesian clinicians are well aware of the dire situation with regards to the prevalence of AMR in their country. Choosing lower thresholds, albeit perhaps more clinically relevant in other countries, would not provide any useful information on appropriate empirical treatment for most antibiotics to Indonesian clinicians.

Our study has limitations. Although the sites were chosen deliberately, the patient population derives from facilities that are representative for the health-care setting in Indonesia. Given the consecutive sampling from the eligible population over a period of at least 6 weeks, with very low frequency of nonconsent (n = 12, 0.4%), the study population can be regarded as representative of the overall patient population seen at these facilities. The studies were carried out using existing accredited laboratories but within a study context that included repeated training and quality monitoring. These should be in place before embarking on LQAS-based AMR surveys, because any incorrect AST result would be detrimental to obtaining adequate LQAS classifications.

E. coli and K. pneumoniae have intrinsic resistance profiles, which are lost when results are reported combined. However, the aim of informing empirical treatment for UTI does not require separating results of drug susceptibility tests for these 2 microorganisms, because the practitioner is not informed about the causative microorganism due to the absence of culture results at the time of treatment prescription.

Indonesian treatment guidelines recommend fluoroquinolones as the preferred drugs for the empirical treatment of uncomplicated UTI, despite the classification of “high resistance” for all outpatient facilities and inpatient wards (32). Indonesia is in a dire situation if effective empirical antibiotic choices for uncomplicated UTI are limited to fosfomycin or parenteral drugs. Over-the-counter use and inadequate prescription of antibiotics are clear driving forces of AMR in Indonesia (33, 34). Antimicrobial stewardship efforts are urgently needed, and smart strategies for AMR surveillance can contribute to such efforts and inform clinicians as well as policy makers. Studies in Indonesia and China have shown that educational training on antimicrobial stewardship for prescribers and patients alike can lead to a reduction in antibiotic prescriptions (35, 36).

As a response to the loud call for AMR surveillance, we propose further adaptation of LQAS-based AMR surveys to obtain timely, locally relevant, and essential data to steer empirical treatment guidelines and antimicrobial stewardship efforts. Being a sampling and analytical approach, its utility is generalizable to other health-care settings in both high-income and lower- and middle-income countries.

Abbreviations

AMR

antimicrobial resistance

AST

antibiotic susceptibility testing

LQAS

lot quality assurance sampling

UTI

urinary tract infection