Quality indicators assessing antibiotic use in the outpatient setting: a systematic review followed by an international multidisciplinary consensus procedure

Abstract

Objectives

Quality indicators (QIs) assessing the appropriateness of antibiotic use are essential to identify targets for improvement and guide antibiotic stewardship interventions. The aim of this study was to develop a set of QIs for the outpatient setting from a global perspective.

Methods

A systematic literature review was performed by searching MEDLINE and relevant web sites in order to retrieve a list of QIs. These indicators were extracted from published trials, guidelines, literature reviews or consensus procedures. This evidence-based set of QIs was evaluated by a multidisciplinary, international group of stakeholders using a RAND-modified Delphi procedure, using two online questionnaires and a face-to-face meeting between them. Stakeholders appraised the QIs’ relevance using a nine-point Likert scale. This work is part of the DRIVE-AB project.

Results

The systematic literature review identified 43 unique QIs, from 54 studies and seven web sites. Twenty-five stakeholders from 14 countries participated in the consensus procedure. Ultimately, 32 QIs were retained, with a high level of agreement. The set of QIs included structure, process and outcome indicators, targeting both high- and middle- to low-income settings. Most indicators focused on general practice, addressing the common indications for antibiotic use in the community (particularly urinary and respiratory tract infections), and the organization of healthcare facilities. Twelve indicators specifically addressed outpatient parenteral antimicrobial therapy (OPAT).

Conclusions

We identified a set of 32 outpatient QIs to measure the appropriateness of antibiotic use. These QIs can be used to identify targets for improvement and to evaluate the effects of antibiotic stewardship interventions.

Introduction

Antibiotics are an essential clinical and public health resource, but bacterial resistance is increasingly threatening their efficacy worldwide.^1,2 Up to 80%–90% of antibiotics are prescribed in the outpatient setting,^3,4 and significant inter-country variation exists regarding antibiotic consumption, with high levels of inappropriate use.^5–7 Since antibiotic use is associated with the emergence of bacterial resistance at both individual and population levels,^8,9 antibiotic stewardship interventions are crucial to promote responsible use of antibiotics.

A reliable measurement of the quality of health care is essential to trigger improvement.^10,11 A significant number of studies have assessed the quality of antibiotic prescribing in the outpatient setting, often using very different methods, underscoring the need for standardized and consensually accepted QIs for the appraisal of antibiotic use.¹² Quality indicators (QIs) are ‘a measurable element of practice performance for which there is evidence or consensus that it can be used to assess the quality, and hence change in the quality, of care provided’.^10,13,14 Ideally, they have a clear and direct association with relevant outcomes. QIs help health care providers and policy makers to set priorities for interventions to improve health care.

QIs are usually divided into three categories: (i) structure indicators, reflecting the organization of the health care system (e.g. availability of copy of essential drugs list or formulary); (ii) process indicators (e.g. doing relevant diagnostic tests before prescribing antibiotics); and (iii) outcome indicators, focusing on the consequences of interventions (e.g. hospital readmission).¹³

The Driving Reinvestment in Research and Development and Responsible Antibiotic Use (DRIVE-AB) project is a public–private consortium, funded by the EU Innovative Medicines Initiative (IMI).¹⁵ The development of a framework to define responsible antibiotic use is one of its objectives, together with a set of current validated QIs and quantity metrics to evaluate antibiotic use.

This study was part of the DRIVE-AB project and had as its primary aim the development of a set of generic QIs for the outpatient setting from a global perspective. According to the DRIVE-AB project objectives, this consensus procedure had to take into account different perspectives, including the medical community, public health and patients, antibiotic research and development (R&D), payers, policy makers, governments and regulators. The QIs should also account for diverse socioeconomic settings, thereby ensuring a global scope. The goal of DRIVE-AB was indeed to define at an overarching level (i.e. by including all relevant perspectives) what constitutes responsible antibiotic use.

Materials and methods

The set of QIs was developed through a systematic literature review, followed by a RAND-modified Delphi procedure.^10,16,17 This review is reported following the PRISMA statement.¹⁸ This literature review was conducted alongside five other literature reviews, on responsible use of antibiotics,¹⁹ variation in antibiotic use,²⁰ QIs for the inpatient setting,²¹ quantity metrics for the outpatient setting²² and quantity metrics for the inpatient setting.²³

Systematic review of the literature

Search strategy

We searched the MEDLINE database using the PubMed interface for titles and abstracts of articles describing QIs for outpatient antibiotic use published from the inception of MEDLINE until 12 December 2014. The search strategy keywords were defined with the help of a librarian from one of the participating centres, and were organized around seven key concepts (Figure S1, available as Supplementary data at JAC Online).

Inclusion and exclusion criteria

We included studies written in English that focused on systemic (i.e. not topical, vaginal or inhaled) antibiotic use in humans in the outpatient setting and that described QIs.

Papers on antiviral, antifungal and antiparasitic drugs were excluded, as well as articles describing antibiotic use in tuberculosis and in pathologies included in the Orphanet list of rare diseases.²⁴ An outpatient was defined as a non-hospitalized patient who visits a physician in an ambulatory care setting. We therefore included outpatient parenteral antimicrobial therapy (OPAT).

We included papers where indicators were labelled as such, based on published trials, guidelines, literature reviews or consensus procedures, and fulfilled our QI definition (see Introduction).

We excluded papers whose full-text could not be retrieved from any of the libraries of the participating centres (eight different catalogues) and from Google Scholar^®.

Screening process, data collection and analysis

All titles and abstracts were independently screened by two investigators (M. L. M. and C. P.) to search for potentially eligible papers, using the literature review management software DistillerSR^® (Evidence Partners, Ottawa, Ontario, Canada). The full text of articles was systematically screened if there was no abstract available, or if the abstract was insufficiently detailed to allow a proper assessment of the eligibility criteria.

All eligible full-text articles were independently screened by two reviewers (M. L. M. and C. P.), who selected relevant articles based on our predefined inclusion and exclusion criteria. The reference list of all included articles was also screened to look for potential additional papers.

One investigator (M. L. M.) performed data extraction using a standardized form and another reviewer (C. P.) double-checked all extracted data from the included articles.

Data on relevant QIs were collected, together with the study design and the setting of the study (high- versus middle- to low-income countries²⁵). We categorized the study design as follows: QIs based on an expert consensus/Delphi round (consensus-based indicators); QIs based on a literature review (review-based indicators); and QIs based on guidelines (guideline-based indicators).

QIs were classified as ‘Structure’, ‘Process’ and ‘Outcome’ indicators.¹³ We further specifically identified QIs adapted to the general practice setting and QIs specific to OPAT.

During the whole screening and data extraction any disagreement between the two investigators was resolved by discussion, using advice from a third expert if needed (I. C. G. or M. E. H.).

Web site search

A web site search was also performed (Table S1). The selection of relevant web sites was based on discussion and consensus among the study investigators. Web sites in English from 26 national and international infectious diseases societies, quality improvement and public health organizations were included. All web sites were screened by one reviewer (M. L. M.) using ‘indicator’ and/or ‘antibiotic/antimicrobial’ as search terms. Data extraction was performed with the same standardized form as that used for published papers.

RAND-modified Delphi procedure

After having established a list of QIs based on the literature review and the web site search, duplicate QIs were deleted and those addressing similar topics were grouped. Each indicator was phrased as a generic statement (e.g. ‘Antibiotics should be prescribed for bacterial infections’). If applicable, a QI was further detailed in terms of one or more numerator–denominator combinations found in the literature or on web sites.

The comprehensive list of QIs was presented to a multidisciplinary panel of stakeholders for a RAND-modified Delphi consensus procedure,^10,16,17 consisting of two surveys (first and second round) with a face-to-face meeting between them. International stakeholders were invited by e-mail to participate. The selection process of the stakeholders is described elsewhere.²¹ The complete list of stakeholders is presented in the Acknowledgements section and full details are available in Table S2.

First round

The list of QIs was converted into an internet-based questionnaire (Figure S2) using SurveyMonkey^® (Palo Alto, CA, USA). Respondents were asked to appraise the relevance of the indicators for assessing the quality of antibiotic use in the outpatient setting, using a nine-point Likert scale ranging from 1 (clearly not relevant) to 9 (clearly relevant). They also had a ‘cannot assess’ option, and a comments box for each QI.

QIs were: (i) selected if the median score was 8 or 9 with agreement; (ii) held for further discussion if the median score was 8 or 9 without agreement; or (3) rejected if the median score was <8. Agreement was defined as >70% of the scores being in the upper tertile (score 7–9). Stakeholders could also suggest new QIs for further discussion.

Consensus meeting

Newly suggested QIs, selected QIs with many comments and those ‘held for further discussion’ (category 2) after the first round were discussed during a face-to-face meeting, to which the same stakeholders who participated in the first round were invited. The discussion was led by an expert moderator who was otherwise not part of the study team. Before the meeting, all stakeholders received a detailed summary of the survey results describing their individual replies as well as the group median scores for each suggested QI. During the meeting, stakeholders were asked to accept, reject or rephrase each QI under discussion to reach consensus. Moreover, they could rephrase the QIs that were already selected after the first round. Study investigators did not get directly involved in the discussion.

Second round

All previously accepted, added and rephrased QIs were presented in a second internet-based questionnaire and sent to all stakeholders who participated in the first round, together with a detailed report of the previous phases. Stakeholders were asked to definitively accept or reject each QI. A ‘cannot assess’ option was offered, as well as the possibility to add comments. Indicators were accepted if >70% of respondents agreed with their selection.

Results

Systematic review of the literature

The literature search identified 3563 articles, of which 287 were considered eligible for full-text screening. After the application of inclusion and exclusion criteria, the screening of reference lists and the web site search, a final set of 54 articles and 7 web sites was retained (Figure 1). The majority of the papers (37/54, 69%) concerned high-income countries (Table S3).^26–86

Figure 1.

Flow chart of the systematic review of the literature.

Overall, 356 QIs were extracted and 120 were retained after removing duplicates. Indicators addressing similar topics were grouped, producing a final list of 43 QIs (29 concerning general practice and 14 concerning OPAT), with 113 numerator–denominator combinations (Table S4). This list was presented to the stakeholders for the first round of the consensus procedure.

RAND-modified Delphi procedure

The first round was completed between August and September 2015. Forty-three stakeholders were contacted and 23 of them (from 12 countries: Belgium, Canada, Chile, Denmark, France, Nigeria, Spain, Sweden, the Netherlands, UK, USA and Vietnam) agreed to participate (Table 1 and Table S2). After the first round, 27 QIs were selected, 10 were rejected and 6 were held for further discussion. Two QIs were newly suggested (Figure 2 and Table S4).

Table 1.

Stakeholders participating in the RAND-modified Delphi consensus procedure

Phase of the consensus procedure/ characteristics	Group A	Group B	Group C	Group D	Total
First survey
Number (%)	11 (48)	3 (13)	6 (26)	3 (13)	23
Area of expertise (n)	GP (3)	anti-AMR network (1)	pharmaceutical industry (6)	agencies (1 ECDC, 1 CDC)
	ID (2)	ethics (1)		Ministry of Health (1)
	microbiology (4)	patient society (1)
	pharmacology (2)
Place of work (n)	Africa (1)	Europe (3)	Europe (3)	Europe (2)
	Asia (1)		North America (3)	North America (1)
	Europe (6)
	North America (2)
	South America (1)
Face-to-face meeting
Number (%)	4 (40)	1 (10)	3 (30)	2 (20)	10
Area of expertise (n)	ID (2)a	anti-AMR network (1)	pharmaceutical industry (3)	international societies (1 ECDC, 1 CDC)
	microbiology (1)
	pharmacology (1)
Place of work (n)	Europe (2) Asia (1)	Europe (1)	Europe (1)	Europe (1)
	North America (1)		North America (2)	North America (1)
Second survey
Number (%)	11 (55)	2 (10)	5 (25)	2 (10)	20
Area of expertise (n)	GP (3)	anti-AMR network (1)	pharmaceutical industry (5)	international societies (1 ECDC, 1 CDC)
	ID (2)	Ethics (1)
	microbiology (4)
	pharmacology (2)
Place of work (n)	Africa (1)	Europe (2)	Europe (3)	Europe (1)
	Asia (1)		North America (2)	North America (1)
	Europe (6)
	North America (2)
	South America (1)

AMR, antimicrobial resistance; GP, general practice; ID, infectious diseases.

Group A, medical community; Group B, public health and patients; Group C, antibiotic R&D; Group D, payers, policy makers, government and regulators.

^aThese stakeholders participated in the consensus procedure on QIs and quantity metrics for antibiotic use in the inpatient setting that was performed in parallel to our study by Monnier et al.²¹ and Stanić Benić et al.²³ They were provided with the results of the first survey on QIs for antibiotic use in the outpatient setting and took part in the consensus meeting.

Figure 2.

Flow chart of the RAND-modified Delphi consensus procedure.

The face-to-face consensus meeting took place at Schiphol airport (the Netherlands) on 30 September 2015. Seven stakeholders were present and three participated by teleconference. Two of these stakeholders (from Greece and Slovenia) participated in the consensus procedure on QIs and quantity metrics for antibiotic use in the inpatient setting that was performed in parallel to our study in the framework of the DRIVE-AB project. They were provided with the results of the first survey and took part in the consensus discussion on outpatient QIs (OQIs) (Table 1 and Table S2). Thirty-three QIs (with 68 numerator-denominator combinations) were selected, with 6 QIs having been rephrased and 2 merged with other already existing QIs (Figure 2 and Table S4).

The second round was carried out between the end of December 2015 and February 2016. Twenty stakeholders participated (Table 1). Thirty-two QIs (with 67 numerator–denominator combinations) were selected, with an average consensus score of 94%. One QI was rejected (Figure 2 and Table S4).

Final set of OQIs

The final set of 32 OQIs is presented in Table 2. Twenty QIs addressed general practice, 11 addressed OPAT and 1 addressed both. Full details with all numerator–denominator combinations are available in Table S5.

Table 2.

The final set of 32 consensually validated quality indicators assessing antibiotic use in the outpatient setting

Outpatient quality indicator (OQI)	Type of indicator	References	Study designa
OQI-1 Antibiotics should be prescribed for (most) bacterial infections (e.g. acute pneumonia, urinary tract infections)	process	26–30	A
OQI-2 Antibiotics should not be prescribed for viral infections or (most) self-limiting bacterial infections (e.g. acute bronchitis, influenza, acute otitis media in children >2 years old)	process	26,28–42	A, B, C
OQI-3 Outpatients should receive antibiotic therapy compliant with guidelines; this includes, but is not limited to, indication, choice of the antibiotic, duration, dose and timing	process	26–31,34,36,39,40,43,44,46–51	A, B, C
OQI-4 Some antibiotics should be rarely prescribed	process	59	B
OQI-5 Acute upper respiratory infections and bronchitis should not be treated with antibiotics within the first 3 days, unless there is documented indication for treatment	process	29,30,60,61	A, C
OQI-6 Outpatients with acute tonsillitis/pharyngitis should undergo a group A streptococcal diagnostic test to decide whether or not they should receive antibiotics	process	32	C
OQI-7 Outpatients with an acute tonsillitis/pharyngitis and positive group A streptococcal diagnostic test should be treated with antibiotics	process	29,30	A
OQI-8 Antibiotics for an acute tonsillitis/pharyngitis should be withheld, discontinued or not prescribed if an outpatient presents a diagnostic test (rapid antigen test or throat culture) negative for group A streptococci	process	40	B
OQI-9 Prescribed antibiotics should be chosen from an essential list/formulary	process	47,48,50–52,62–73	C
OQI-10 Possible contraindications should be taken into account when antibiotics are prescribed	process	79,80	A, C
OQI-11 Antibiotics from the list of essential antibiotics should be available in health facilities that dispense antibiotics	structure	47,48,50,51,62,67,70,81	A, C
OQI-12 Key antibiotics should not be out of stock in health facilities that dispense antibiotics	structure	50,51	C
OQI-13 Antibiotics in stock should not be beyond the expiry date	structure	50,81	A, C
OQI-14 Antibiotics that are dispensed to outpatients should be adequately labelled (patient name, antibiotics name, when antibiotics should be taken)	structure	47,48,50,51,65,67,70,82	C
OQI-15 Antibiotics should be adequately conserved and handled in health facilities	structure	50,51	C
OQI-16 Health facilities should keep adequate records of dispensed key antibiotics	structure	50	C
OQI-17 A copy of the essential antibiotics list should be available in health facilities	structure	47,48,50,62,67,70	C
OQI-18 Standard antibiotic treatment guidelines should be available in health facilities	structure	50	C
OQI-19 Health facilities should have access to the Summary of Product Characteristics of prescribed antibiotics, written in a local language	structure	47	C
OQI-20 Antibiotics should not be sold without prescription	structure	50	C
OQI-21 Outpatients and OPAT patients with an antibiotic prescription should be educated on how to take it, on the dosage, on expected side effects, and on the natural history of the disease	process	47,48,50,51,60,62,65,67,70,75,76,82–86	A, B, C
OQI-22 The treatment plan should be agreed between the OPAT team and the referring clinician before start of treatment	process	84,85	B
OQI-23 All OPAT treatment plans should include dose, frequency of administration and duration of therapy	process	84	B
OQI-24 OPAT antibiotics should be correctly stored, prepared, reconstituted, dispensed and administered	structure	84,86	B
OQI-25 Administered doses of OPAT should be documented on a medication card	process	84	B
OQI-26 The first dose of a new antibiotic in an OPAT should be administered in a supervised setting	process	84	B
OQI-27 OPAT antibiotics should be regularly reviewed to optimize speed of intravenous-to-oral switch	process	84	B
OQI-28 Each OPAT centre should monitor quality indicators on OPAT antibiotics	structure	84,86	B
OQI-29 An expert in OPAT (physician, nurse, pharmacist) should work in each OPAT centre	structure	86	B
OQI-30 The OPAT plan should be communicated to the general practitioner at discharge	structure	85	B
OQI-31 The OPAT programme should be accredited or certified	structure	86	B
OQI-32 In an OPAT programme, clinical and/or microbiological outcomes, including treatment failure and adverse events (including Clostridium difficile infections), should be recorded	outcome	84,86	B

^aCategory of the study design: A, consensus-based indicators; B, review-based indicators; C, guideline-based indicators.

Discussion

Here we propose a set of 32 consensually validated QIs to assess antibiotic use in outpatients, targeting mostly general practice and OPAT, in both high-income and middle- to low-income settings. The selected QIs are purposefully generic.

DRIVE-AB goals were to come up with generic metrics for responsible use to be used worldwide; stakeholders from a broad area of expertise (who probably all view ‘responsible’ differently), and not only general practitioners (GPs), were thus asked to rate the relevance of these OQIs defining responsible use, for a broad range of infections in a broad range of countries with the aim of ending up with broadly supported QIs, from a global perspective. This makes our consensus procedure very different from those performed in the original included studies, as in those studies the QIs were very specifically defined, fitting a specific setting and, most of the time, a specific patient group or disease, and involved experts representing that setting (mostly GPs).

This list of QIs encompasses a wide range of clinical conditions, aiming at a comprehensive evaluation of antibiotic use. Only one outcome indicator (OQI-32, Table 2) was retained at the end of the consensus procedure. This is in line with the paucity of outcome indicators in other sets of QIs assessing antibiotic use.^17,26,87 They represent the endpoint of a chain of factors and their evaluation only indirectly estimates the quality of care provided and they are thus potentially influenced by many concurrent factors. In contrast, structure and process indicators provide a direct evaluation of quality in health care and thus allow direct identification of targets for intervention.

Our study is original and has several strengths. We performed a systematic review of QIs available in the literature, followed by a stepwise consensus procedure (RAND-modified Delphi procedure). This design has been previously applied to identify QIs in different health care domains, although the methodology varied significantly in published papers.⁸⁸ The method we chose has been applied in several settings^17,89,90 and complies with the recommendations suggested by Kötter et al.⁹¹ in their systematic review on guideline-based development of QIs, and by Boulkedid et al.⁸⁸ in their systematic review on use and reporting of the Delphi method. We think it ensures a robust content (‘Does the measurement represent every element of the construct?’) and face validity (‘Does the measurement tool appear to measure what we want to measure?’) for the selected QIs.

In the consensus procedure, we included stakeholders from different backgrounds, including medical professionals, patients’ representatives, R&D experts from the pharmaceutical industry, government, and relevant agencies such as the US CDC and the ECDC. Participants came from low-, middle- and high-income countries, from 12 countries and four different continents.

There are, however, some limitations that need to be acknowledged. We only searched the MEDLINE database for our systematic review and we did not systematically explore the grey literature.⁹² However, we did screen relevant web sites and the reference lists of all included articles.

Clinimetric properties of QIs were not assessed in our study, and need to be further validated in specific settings. Clinimetric properties are characteristics of indicators such as measurability (feasibility), applicability, acceptability, reliability, potential for improvement and sensitivity to change.^10,89,93 Our objective was to develop a list of generic QIs that can be assessed and adapted to national specificities and guidelines. The most critical aspect for some of the selected QIs is probably measurability, since in several cases patients’ demographic data and clinical information are needed to reliably assess the numerator and the denominator defining the indicator. This information is difficult to obtain on a large scale in many countries.^26,59

A few years ago, the European Surveillance of Antimicrobial Consumption Network (ESAC-Net) developed a set of so-called drug-specific quality indicators⁸⁷ for antibiotic use in the outpatient setting. However, these indicators were based on a quantitative evaluation of antibiotic consumption, and were actually more quantity metrics rather than QIs, since they can be meaningfully interpreted only in terms of comparison between different settings and different periods of time. The QIs we selected reflect the quality of care, having significance per se, without the need for comparison and not requiring the setting of thresholds (such as ‘quinolones should not exceed 10% of total antibiotic prescriptions’), which can be somewhat arbitrary. Thus, we think that this set of QIs and those developed by the ESAC-Net should not be considered as mutually exclusive, since they approach the phenomenon of antibiotic prescription in different ways and with substantially different methodology, thus being able to provide complementary information.

A systematic review of QIs in primary care has been recently published by Saust et al.⁹⁴ However, their literature search was less extensive than our systematic review. Moreover, they did not use a consensus procedure to validate the identified indicators. Also, many of the indicators they included are based on a quantitative evaluation of antibiotic consumption, similar to those from the ESAC-Net.^26,87 The WHO also provided detailed lists of QIs.^47,50,81 The WHO indicators target low- to middle-income countries, but they are generic and not focused on antibiotic treatments in outpatients. We nevertheless derived a relevant number of structure and process QIs (16 combinations of numerators and denominators, grouped in 13 QIs; for details see Tables S4 and S5) from these WHO indicators. Our consensus procedure confirmed that these QIs have high face validity and that they could represent a useful tool to assess the quality of antibiotic prescriptions in low-to- middle-income countries.

Regarding OPAT, we extracted QIs from two guidelines^84,86 and a critical appraisal of evidence.⁸⁵ To the best of our knowledge, there are no QIs for OPAT available in the literature, so we present here the first comprehensive set of validated QIs for OPAT. Outpatient parenteral antibiotic treatment is increasingly being used, mainly to optimize patient comfort and decrease risks and costs related to hospitalization,⁹⁵ but current evidence to guide its implementation is limited⁸⁵ and variability in practice is frequent.⁹⁶ Thus, a consensually validated set of QIs could be of great value to evaluate OPAT programmes and set targets for improvement.

Overall, our search identified a relatively low number of unique QIs targeting antibiotic use in the outpatient setting. Further investigation is needed to develop new QIs, including some focusing on diagnostic processes,⁹⁴ using an adequate methodology.^10,11,13 Assessing the relationship between process QIs and outcome indicators for appropriate antibiotic use (e.g. ecological consequences of antibiotic consumption) is also needed to further validate the QIs;⁹⁷ this was not done for the QIs that we identified in our literature review.

In conclusion, we listed a set of generic QIs intended to evaluate the quality of antibiotic use in the outpatient setting, in a wide range of clinical conditions, taking into account the perspectives of a wide range of stakeholders involved with antibiotics. These QIs can be used to measure the appropriateness of current antibiotic use at the individual, regional or supra-regional level. Thus they can be used to identify targets for improvement and to evaluate the effects of antibiotic stewardship interventions. Indeed, these are generic QIs that are globally applicable. These QIs need to be tested in practice, carefully evaluating their clinimetric properties in different settings and conditions.