Study reporting quality among interventions to reduce antibiotic use is a barrier to evidence-informed policymaking on antimicrobial resistance: systematic review

Abstract

Background

Countries are currently seeking evidence-informed policy options to address antimicrobial resistance (AMR). While rigorous evaluations of AMR interventions are the ideal, they are far from the current reality. Additionally, poor reporting and documentation of AMR interventions impede efforts to use evidence to inform future evaluations and policy interventions.

Objectives

To critically evaluate reporting quality gaps in AMR intervention research.

Methods

To evaluate the reporting quality of studies, we conducted a descriptive synthesis and comparative analysis of studies that were included in a recent systematic review of government policy interventions aiming to reduce human antimicrobial use. Reporting quality was assessed using the SQUIRE 2.0 checklist of 18 items for reporting system-level interventions to improve healthcare. Two reviewers independently applied the checklist to 66 studies identified in the systematic review.

Results

None of the studies included complete information on all 18 SQUIRE items (median score = 10, IQR = 8–11). Reporting quality varied across SQUIRE items, with 3% to 100% of studies reporting the recommended information for each SQUIRE item. Only 20% of studies reported the elements of the intervention in sufficient detail for replication and only 24% reported the mechanism through which the intervention was expected to work.

Conclusions

Gaps in the reporting of impact evaluations pose challenges for interpreting and replicating study results. Failure to improve reporting practice of policy evaluations is likely to impede efforts to tackle the growing health, social and economic threats posed by AMR.

Introduction

Antimicrobial resistance (AMR) poses an urgent threat to global health. Over the past 5 years, increasing attention to the health, social and economic threats posed by AMR have opened a rare policy window for achieving meaningful action on AMR.^1–7 To date, 100 countries have developed national action plans to combat AMR, although many of these plans are vague and most are devoid of specific policy directions.⁸ Having made commitments to tackle AMR and reduce inappropriate antimicrobial use, many countries are now actively searching for evidence-informed policy options to fulfil their obligations.

While rigorous evaluations of AMR interventions are the ideal, they are far from the current reality. A Cochrane review of hospital-level interventions to improve antibiotic prescribing found that the quality of the reporting among the 163 non-randomized studies was so poor that it was difficult for health professionals to use the research findings or implement interventions that were shown to be useful.⁹

This failure to adequately report intervention studies is not unexpected, though it is both scientifically and economically indefensible. Poor documentation is common in health research^10,11 and researchers have estimated that at least 50% of published health research is not sufficiently clear, complete or accurate for others to interpret or use.^12,13 Yet, reliable evidence is essential for decision-makers who must responsibly allocate scarce government resources to tackle AMR.

AMR has already rendered some infections untreatable using existing antimicrobials^14,15 and World Bank projections suggest that AMR could derail the Sustainable Development Goals by driving an estimated 24 million people into extreme poverty, exacerbating global economic inequality¹⁶ and potentially resulting in tens of millions of deaths.¹⁷ It is therefore of the utmost importance that policy interventions are guided by high-quality evidence rather than the principle of ‘it seemed like a good idea at the time’. Gaps in the reporting of impact evaluations pose challenges for interpreting and replicating study results and can lead decision-makers to design and implement policies without evidence. It is therefore of the utmost importance that studies clearly report on the development and elements of their intervention, their choices of measures and metrics, and their findings and conclusions. In this article, we apply the SQUIRE 2.0 (Standards for Quality Improvement Reporting Excellence) framework¹⁸ to critically evaluate reporting-practice gaps of policy intervention evaluations that aim to reduce human antimicrobial use.

Methods

Study design

We conducted a descriptive synthesis and comparative analysis to evaluate the reporting quality of AMR policy intervention studies. Two reviewers (S.R.V.K. and M.N./Ranjana Nagi) independently applied the SQUIRE 2.0 checklist to 66 studies identified in a systematic review of government policy interventions aimed at reducing the human use of antimicrobial drugs.^19,20 SQUIRE 2.0 is an 18-item checklist that is intended to guide the reporting of system-level interventions to improve healthcare. Full details of the systematic review’s methodology were published in advance in a protocol.^19,20

Eligibility

All studies were included if they clearly described a government policy intervention to reduce human antimicrobial use and applied a quantitative design to measure the impact.^19,20

Data selection

A search strategy encompassing seven electronic databases from medicine and the social sciences (MEDLINE, CINAHL, EMBASE, PAIS Index, CENTRAL, Web of Science and PubMed articles not indexed in MEDLINE) was developed in consultation with research librarians and searched from inception to January 2019 without language or date limits. Targeted web searching was used to identify grey literature and the ProQuest Dissertations and Theses database was used to identify dissertations. Subject-matter experts in each of WHO’s six regions were contacted to identify missing studies.

Articles were screened against three inclusion criteria: first, the evaluated intervention was a policy intervention defined as an intervention enacted by a government or government agency at the federal, state, provincial or municipal level, that aimed to change antimicrobial use through education, restriction, incentivization, coercion, training, persuasion, changing the physical or social context, modelling appropriate behaviour or reducing barriers to action;²¹ second, the study quantitatively evaluated the effect of the intervention; and third, the study assessed an outcome measure related to human antimicrobial use, including consumption, dosing, prescribing or sales of an antibiotic, antiviral, antiparasitic or antifungal drug. An initial screen of titles and abstracts was conducted independently by two reviewers (S.R.V.K. and Sara Jones/Archita Srivistava/Ranjana Nagi) as described in Supplementary data S1 (available at JAC Online). The full texts of potentially relevant studies were screened by two reviewers (S.R.V.K. and M.N./Ranjana Nagi) and disagreements were resolved by consensus.

Data extraction

To identify gaps in reporting practice, we assessed the quality of reporting against the SQUIRE 2.0¹⁸ tool; the 18 SQUIRE items are detailed in Table 1. We evaluated each of the systematic review’s English language studies against the SQUIRE 2.0 items; research abstracts and translated studies were excluded from this analysis because they lacked sufficient detail or the translation was of insufficient quality to grade them fairly on SQUIRE criteria. Two reviewers (S.R.V.K. and M.N./Sasha Van Katwyk) independently scored each study against the SQUIRE 2.0 criteria using a decision guide developed a priori(Supplementary data S2) based on the SQUIRE guidelines¹⁸ and SQUIRE explanation and elaboration documents.²² Each item was coded as ‘fully reported’ where complete information on all elements of the SQUIRE item was available, ‘not fully reported’ where there was incomplete information in the report on one or more elements of the SQUIRE item or ‘not applicable’ where the SQUIRE item was not relevant to the study in question. The reviewers kept notes on the missing or unclear elements in each study, which are available in Supplementary data S3. Disagreements were resolved by consensus.

Table 1.

SQUIRE 2.0 checklist items

SQUIRE item	Explanation in SQUIRE 2.0
Title	Indicate that the manuscript concerns an initiative to improve healthcare (broadly defined to include the quality, safety, effectiveness, patient-centredness, timeliness, cost, efficiency and equity of healthcare).
Abstract	a. Provide adequate information to aid in searching and indexing.
	b. Summarize all key information from various sections of the text using the abstract format of the intended publication or a structured summary, such as: background, local problem, methods, interventions, results and conclusions.
Problem description	Nature and significance of the local problem.
Available knowledge	Summary of what is currently known about the problem, including relevant previous studies.
Rationale	Informal or formal frameworks, models, concepts and/or theories used to explain the problem, any reasons or assumptions that were used to develop the intervention(s) and reasons why the intervention(s) was expected to work.
Specific aims	Purpose of the project and of this report.
Context	Contextual elements considered important at the outset of introducing the intervention.
Intervention	a. Description of the intervention(s) in sufficient detail that others could reproduce it.
	b. Specifics of the team involved in the work.
Study of the intervention	a. Approach chosen for assessing the impact of the intervention(s).
	b. Approach used to establish whether the observed outcomes were due to the intervention(s).
Measures	a. Measures chosen for studying processes and outcomes of the intervention(s), including rationale for choosing them, their operational definitions and their validity and reliability.
	b. Description of the approach to the ongoing assessment of contextual elements that contributed to the success, failure, efficiency and cost.
	c. Methods employed for assessing completeness and accuracy of data.
Analysis	a. Qualitative and quantitative methods used to draw inferences from the data.
	b. Methods for understanding variation within the data, including the effects of time as a variable.
Ethical considerations	Ethical aspects of implementing and studying the intervention(s) and how they were addressed, including, but not limited to, formal ethics review and potential conflict(s) of interest.
Results	a. Initial steps of the intervention(s) and their evolution over time (e.g. time-line diagram, flow chart or table), including modifications made to the intervention during the project.
	b. Details of the process measures and outcome.
	c. Contextual elements that interacted with the intervention(s).
	d. Observed associations between outcomes, interventions and relevant contextual elements.
	e. Unintended consequences, such as unexpected benefits, problems, failures or costs associated with the intervention(s).
	f. Details about missing data.
Summary	a. Key findings, including relevance to the rationale and specific aims.
	b. Particular strengths of the project.
Interpretation	a. Nature of the association between the intervention(s) and the outcomes.
	b. Comparison of results with findings from other publications.
	c. Impact of the project on people and systems.
	d. Reasons for any differences between observed and anticipated outcomes, including the influence of context.
	e. Costs and strategic trade-offs, including opportunity costs.
Limitations	a. Limits to the generalizability of the work.
	b. Factors that might have limited internal validity, such as confounding, bias or imprecision in the design, methods, measurement or analysis.
	c. Efforts made to minimize and adjust for limitations.
Conclusions	a. Usefulness of the work.
	b. Sustainability.
	c. Potential for spread to other contexts.
	d. Implications for practice and for further study in the field.
	e. Suggested next steps.
Funding	Sources of funding that supported this work. Role, if any, of the funding organization in the design, implementation, interpretation and reporting.

Analysis

We report the distribution of score by element as frequency and percentage. Overall scores, which were calculated by assigning one point for each element that was fully reported, are presented using a bar graph and summarized using median (IQR). A simple linear regression model was fitted to the total scores and year of publication to assess statistical significance of changes in reporting over time. We descriptively compared the reporting quality of studies by evaluation design.

Results

Description of included studies

A PRISMA flow diagram of the systematic review is available in Supplementary data S4. Sixty-six impact evaluations were identified that met the inclusion criteria (Supplementary data S5). Of the 66 included studies, 64 focused on interventions to reduce the use of antibiotics and 2 focused on interventions to reduce the use of antimalarials. These policy interventions spanned a wide range of topics, including: policies to improve hospital infection prevention and stewardship; policies to educate health professionals, policymakers and the public on sustainable antimicrobial use; policies to eliminate incentives for antimicrobial overuse and misuse; and policies to change features of the health system to limit antimicrobial overuse.²⁰ We found evaluations in four of the six WHO regions: Americas (n = 22), Western Pacific (n = 22), Europe (n = 20) and Africa (n = 2), but did not identify any evaluations from the South-East Asian or Eastern Mediterranean regions. The majority of studies focused on national-level policies (n = 42), with fewer studies focused on state/province (n = 10) and regional (n = 14) policies. Studies were published between 1997 and 2019.

The reporting quality varied across SQUIRE items, with 3% to 100% of studies including the recommended information for each item. The overall reporting quality of each study is shown in Figure 1 and the quality of reporting for each SQUIRE element is shown in Figure 2. Figure 3 shows the distribution of overall quality scores. We found a very slight trend towards better reporting over time (P = 0.013) although there remains substantial variation in quality scores in recent years (Figure 4).

Figure 1.

Reporting quality by element of the SQUIRE 2.0 checklist for the 66 included studies. Green (dark grey) represents fully reported items, yellow (light grey) represents items that were not fully reported and black indicates that the item was not applicable. This figure appears in colour in the online version of JAC and in black and white in the print version of JAC.

Figure 2.

Number and percentage of the 66 included studies that fully reported each SQUIRE 2.0 item. Dark grey represents studies that fully reported the criteria and mid grey represents studies that did not fully report the criteria. N/A, not applicable (indicated by light grey).

Figure 3.

Distribution of the overall study quality scores (i.e. the number of fully reported SQUIRE items) among the 66 included studies.

Figure 4.

Trend in overall quality scores (i.e. the number of fully reported SQUIRE items) by publication date among the 66 included studies.

Findings for key SQUIRE items

Several SQUIRE items were incompletely reported in more than half of eligible studies, including the abstracts (Item 2), rationale (Item 5), description of the intervention (Item 8), measures (Item 10), ethical considerations (Item 12), results (Item 13), interpretation (Item 15), limitations (Item 16), conclusions (Item 17) and funding (Item 18). The lowest scoring studies were uncontrolled before–after studies; however, there was no clear indication that studies with more rigorous designs were better reported. As Figure 3 shows, most studies fully reported approximately half of the SQUIRE items.

Only 28 articles were judged to have an abstract (Item 2) that provided adequate information for searching and indexing and summarized all key information from various sections of the text. Articles that failed to fully report this item often did not include either keywords or a summary of the background and importance of the problem.

Fewer than half (n = 16, 24%) of the included studies adequately reported the study rationale (Item 5); in particular, studies failed to report the reasons why the intervention was intended to work. Studies rarely used a formal theory or framework from the fields of behaviour change or implementation science. The 16 studies that reported a mechanism by which the intervention was intended to work used informal theories, as in Park et al.’s²³ 2005 rationale for why an intervention to limit physicians’ power to dispense medications would reduce antimicrobial use: ‘When physicians have a financial incentive to dispense medications, they are likely to prescribe more drugs. In previous research, dispensing doctors, in comparison with non-dispensing doctors, were found to prescribe greater numbers of drugs.’

Only one-fifth of the included studies (n = 13, 20%) reported the intervention in sufficient detail that it could be replicated (Item 7), with a complete description of the activities and tools used, the inputs, internal activities, outputs and mechanisms by which the components are expected to bring about change. Among these studies, several failed to report details of the personnel participating in the project, including the type of health professionals engaged and their experience, and the level of training received by personnel leading programmes and workshops. Many studies also neglected to report what the intervention contained, such as the subject or content of workshops, and how often and over what timespan the intervention was implemented. We noted that these details were most often missing when the evaluation was carried out retrospectively by a third party.

Fewer than half of included studies (n = 29, 44%) fully reported on the measures employed for studying the intervention process and outcomes (Item 10). Many studies failed to report the rationale for choosing their selected measures, the validity and reliability of their chosen measures or their approach to assessing the completeness and accuracy of data. However, most studies (n = 56, 85%) fully reported the methods employed in the analysis, including the methods employed for drawing inferences from the data and assessing variation in the data (Item 11). In evaluating the reporting quality of both items, we only considered whether the methods were clearly reported. It was not within the scope of this project to evaluate whether the methods described were of sufficient rigour to adequately answer the research question. Forty percent of the included studies (n = 26) used descriptive methods or an uncontrolled before–after design.

Four studies fully reported the SQUIRE item for the results section (Item 13, 6%). The most common omission in this section was a description of the initial steps of the intervention, along with modifications made to the intervention during the project and the unintended consequences and costs of the intervention. The small number of studies that adequately reported the steps of the intervention either used tables and timelines to depict intervention activities and dates (e.g. Belongia et al.,²⁴ D’Acremont et al.²⁵ and Wirtz et al.²⁶) or were single-timepoint interventions, such as enacting a new law, evaluated using interrupted time-series methods where the intervention date could be clearly indicated in a figure (e.g. Santa-Ana-Tellez et al.²⁷ and Mamdani et al.²⁸).

Despite only 4 articles fully reporting the SQUIRE elements for interpretation (Item 15, 6%), the majority of the elements that make up this item were well reported. Nearly all included studies reported the nature of the association between the intervention and outcomes, compared the results with findings from other publications and commented on the impact of the project on people and health systems. However, nearly all included studies failed to report on the costs of the intervention and any strategic trade-offs associated with the intervention; a puzzling oversight given that all included studies were policy interventions enacted by government authorities. Similarly, the full limitations of the study were only described by 24 of the included studies (Item 16, 36%). Although many studies reported on factors that might have limited the internal validity of the study (confounding, bias, etc.), they largely failed to report on the efforts made to minimize and adjust for these limitations. Many studies also failed to report on the generalizability of the research.

The conclusions section was the worst reported SQUIRE item. Only two studies fully reported all of the elements of the conclusion (Item 17, 3%). While most articles did report on the usefulness of their study, many failed to report on the implications of the research for practice in the field, to report on the potential for spread to other contexts, to report on the sustainability of the intervention or to suggest next steps.

Ethical considerations of implementing and studying the intervention and how they were addressed, including formal ethics review and potential conflicts of interest, were only reported by 30 of the included studies (Item 12, 45%). Similarly, only 28 studies fully reported the SQUIRE item on funding (Item 18, 42%). Many articles reported the source of their funding, but failed to report what role the funder had played in design, implementation, interpretation and reporting of the research.

Discussion

Principal findings

Although other systematic reviews have commented on the poor reporting practices in AMR studies,⁹ to the best of our knowledge this is the first study to systematically identify gaps in reporting practices. Our critical appraisal shows that the majority of impact evaluations of government policy to reduce antimicrobial use fully reported on approximately half of the key items identified in the SQUIRE reporting tool for health-systems interventions. Most troubling is our finding that only one-fifth of the included studies reported the intervention in sufficient detail that it could be replicated and one-quarter reported a theory or mechanisms by which the intervention was expected to change the use of antimicrobial drugs.

Research implications

Previous studies^10,11 have shown that poor documentation is common in public health research and the failure to adequately report AMR intervention studies is not unexpected. While our study shows a very slight trend towards better reporting over time, the variation in quality of recently published AMR evaluations is of concern, particularly given the proliferation of quality appraisal tools over the past several years. Several options exist to improve the quality of reporting in AMR studies. The SQUIRE 2.0 checklist is one among many frameworks that can be used to improve the reporting quality of AMR interventions. The TIDieR Checklist (Template for Intervention Description and Replication)²⁹ and TIDieR-PHP (Population Health and Policy)³⁰ are both useful tools to improve the reporting and description of interventions, as is the CONSORT statement³¹ and its many extensions.^32–35 More broadly, Chalmers et al.³⁶ have made several recommendations to improve replication of research and to minimize research waste, including the suggestion that research funders demand research protocols and develop a catalogue that describes research in progress. Research funders should consider requiring protocols and the use of reporting checklists to raise the quality of funded research projects. Detailed reporting of intervention components by governments would improve the quality of evaluations carried out by third parties.

Policy implications

Poor documentation and reporting of the components of AMR interventions poses a challenge for decision-makers. Faced with an urgent need to act on AMR, and in the absence of clear evaluations and reproducible interventions, decision-makers will be forced to design interventions without optimally using and learning from the available evidence. Poor documentation is common in health research^10,11 and the methodological and reporting gaps identified in this paper suggest that this type of ‘research waste’ is a common problem in AMR research. The 66 studies included in this report represent a vast (and unknown) amount of both government programme spending and research funding, yet more than half of the included studies provide sufficiently weak descriptions of the intervention that they cannot be replicated.

The failure to report—or to use—theories of behaviour change in the design of the interventions included here is troubling. Our findings of weak reporting for the rationale and description of the intervention are supported by research that suggests interventions planned without the use of theory are often unclear about the behaviours and outcomes being targeted and the means by which the interventions will achieve their intended effects.³⁷

Other identified gaps represent challenges for knowledge translation and shared learning in AMR. Before implementing interventions, most stakeholders are keen to understand the costs associated with an intervention, the intervention’s sustainability, its effects on different populations and whether the intervention had unintended consequences, such as increases in prescriptions for alternative antimicrobial drugs following the introduction of stricter regulations. These gaps suggest that, even in evaluations of government policy interventions, there is room for improved communication between evaluators and decision-makers to ensure that evaluations take into account key stakeholder questions.

Strengths and limitations

We systematically reviewed and critically evaluated the research literature on AMR policy interventions to rigorously identify gaps in reporting practice. Three research librarians from three different disciplines provided advice on our search strategy and experts from all WHO regions were contacted to ensure that all relevant literature was captured. For this evaluation we limited our analysis to study reports published in English to ensure that we were not unfairly penalizing authors for errors in translation.

However, we recognize that the 66 evaluations captured here are only those that were published in the peer-reviewed or grey literature and that these 66 evaluations do not comprise the full set of interventions launched globally. We suspect that these other interventions have not been evaluated with respect to antimicrobial use or the results of these studies have not been made public; as other examples are scarce, the included studies likely represent some of the best examples of reporting quality.

Conclusions

Operating at the intersection of science and global policy debates, shortcomings in the AMR evidence base pose problems for global AMR control efforts. The growing health and economic threat posed by AMR requires prompt policy action on the part of governments around the world and WHO has called for the development of the economic case for sustainable investment in AMR.³⁸ Yet, without better research evidence, we may falsely conclude that some interventions are both effective and cost-effective, when in reality they are both ineffective and inefficient. More needs to be done to ensure that gaps in reporting practice do not impede efforts to develop an evidence-informed approach to tackle the growing health, social and economic threats posed by AMR.