Clinical prediction models for multidrug-resistant organism colonisation or infection in critically ill patients: a systematic review protocol

Abstract

Introduction

Multidrug-resistant organisms (MDROs) are pathogenic bacteria that are the leading cause of hospital-acquired infection which is associated with high morbidity and mortality rates in intensive care units, increasing hospitalisation duration and cost. Predicting the risk of MDRO colonisation or infection for critically ill patients supports clinical decision-making. Several models predicting MDRO colonisation or infection have been developed; however, owing to different disease scenarios, bacterial species and few externally validated cohorts in different prediction models; the stability and applicability of these models for MDRO colonisation or infection in critically ill patients are controversial. In addition, there are currently no standardised risk scoring systems to predict MDRO colonisation or infection in critically ill patients. The aim of this systematic review is to summarise and assess models predicting MDRO colonisation or infection in critically ill patients and to compare their predictive performance.

Methods and analysis

We will perform a systematic search of PubMed, Cochrane Library, CINAHL, Embase, Web of science, China National Knowledge Infrastructure and Wanfang databases to identify all studies describing the development and/or external validation of models predicting MDRO colonisation or infection in critically ill patients. Two reviewers will independently extract and review the data using the Data Extraction for Systematic Reviews of Prediction Modelling Studies checklist; they will also assess the risk of bias using the Prediction Model Risk of Bias Assessment Tool. Quantitative data on model predictive performance will be synthesised in meta-analyses, as applicable.

Ethics and dissemination

Ethical permissions will not be required because all data will be extracted from published studies. We intend to publish our results in peer-reviewed scientific journals and to present them at international conferences on critical care.

PROSPERO registration number

CRD42022274175.

Strengths and limitations of this study.

• This systematic review will provide an overview of models predicting multidrug-resistant organisms (MDRO) colonisation or infection in critically ill patients, helping inform evidence-based recommendations.

• This systematic review will use the Prediction Model Risk of Bias Assessment Tool to evaluate the methodological quality of included studies.

• Meta-analysis and narrative summaries will be used for quantitative and qualitative evidence assessment, including pooled estimates, as suitable.

• The findings of this systematic review will provide a foundation for predicting and preventing MDRO using evidence-based methodology, helping to reduce the rates of infection in critically ill patients.

• Potential limitations of this review include heterogeneous data sources, for example, studies with varied designs, populations, MDRO and intensive care unit types and timelines, which may require further research to standardise.

Introduction

Multidrug-resistant organisms (MDROs) increase the risk of poor outcomes worldwide.¹ MDROs, including methicillin-resistant Staphylococcus aureus (MRSA), carbapenem-resistant Enterobacteriaceae (CRE), vancomycin-resistant Enterococcus (VRE) and extended-spectrum β-lactamase-producing Enterobacteriaceae (ESBL-EKP), etc, among others, are the leading causes of hospital-acquired infections.² A recent study, based on data from patients treated across 890 hospitals in the USA from 2012 to 2017, reported 622 390 cases of infection; MRSA and ESBL infections accounted for most of these cases.³ In China, the national report on bacterial resistance in 2020 found that the detection rates of erythromycin-resistant Streptococcus pneumoniae (ERSP), methicillin-resistant coagulase-negative Staphylococci (MRCNS) and carbapenem-resistant Acinetobacter baumanni (CRAB) were highest at tertiary hospitals.⁴ The intensive care unit (ICU) has the highest incidence of MDROs across all hospital departments. Even in developed countries, where infection control is well-organised, approximately 25% of ICU patients experience at least one hospital-acquired infection; the corresponding rate for developing countries is 50%.⁵ Meanwhile, in China, the detection rates of ERSP, MRCNS and CRAB in the ICU are estimated at 94.4%, 84.2% and 78.2%, respectively.⁴

MDROs increase morbidity and mortality risks, and extend hospitalisation duration.⁶ In 2015, there were 700 000 reported deaths due to MDRO infections globally; this number is expected to exceed 10 million by 2050.⁷ In addition, cumulative economic losses related to bacterial antimicrobial resistance have been reported as $100 trillion. Giraldi et al⁸ estimated that infections extended general hospitalisations and ICU stays by an average of 18.8 days and 21.2 days, respectively. Wang et al⁹ reported that the length of ICU stay in patients with MDRO infection was 26.0 days longer than that of those without infection. Hence, infection control and prevention are important in the ICU setting. Antibiotic use helps manage infection risk and spread.¹⁰ Nevertheless, it increases the risk of antimicrobial resistance, which is growing to pandemic proportions, hindering treatment progress.¹¹ According to the WHO, most antimicrobials were discovered in the 20th century, and the development of new antibiotics has been limited since then.¹²

Guidelines for the prevention and control of MDRO outline some non-pharmaceutical interventions.^13–16 They require that risk factors for MDRO be ascertained to support accurate treatment choices. As no single risk factor can reliably predict MDRO infection due to disease heterogeneity and complexity, clinical prediction models are used for risk assessments.¹⁷ Internally and externally validated prediction models may help identify critically ill patients at risk of MDRO, supporting suitable antibiotic prescriptions and infection control measures. For example, Wang et al¹⁸ reported that male sex, higher C-reactive protein levels and higher Pitt bacteraemia scores were independent predictors of MDRO colonisation or infection. In addition, Yoon et al¹⁹ showed that ICU readmission during hospitalisation, chronic obstructive lung disease, recent antibiotic treatment and recent vancomycin use were independent risk factors for VRE carriage at ICU admission. Meanwhile, Ochotorena et al²⁰ found that an Acute Physiology and Chronic Health Evaluation score of more than 15 points and hospitalisation duration of more than 4 days increased the risk of MRSA colonisation/infection. Finally, Li et al²¹ proposed that carbapenem-resistant Klebsiella pneumoniae (CRKP) colonisation or infection in the previous year, CD4/CD8 cell count ratio of less than 1, and parenteral nutrition duration of more than 48 hours were independent risk factors for CRKP infection. These evidence not withstanding, to the best of our knowledge, there is no globally endorsed prediction model for MDRO colonisation or infection, and no risk-classification tools are used for the prediction of MDRO colonisation or infection in critically ill patients in routine clinical practice. Therefore, a critical evaluation of studies proposing potentially relevant prediction models is warranted.

Review objectives

The aim of this systematic review is to evaluate the reporting and methodology of studies on models predicting MDRO colonisation or infection in critically ill patients. We will apply the Prediction Model Risk of Bias Assessment Tool (PROBAST) to assess the risk of bias in studies on model development and validation. The specific objectives of this review are to:

1. Summarise models predicting MDRO colonisation or infection in critically ill patients.

2. Critically assess the methodology of these models.

3. Qualitatively describe the relevant models.

4. Conduct meta-analyses, as suitable, to estimate the overall performance of each risk model for predicting MDRO colonisation or infection.

Methods and analysis

This protocol is presented according to the Preferred Reporting Items for Systematic reviews and Meta-Analyses Protocols (PRISMA-P) guidelines (online supplemental file 1).²² This systematic review is scheduled to be performed from April to December 2022.

Literature search

PubMed, CINAHL, Embase, Web of Science, Cochrane Library, China National Knowledge Infrastructure and Wanfang databases will be searched from inception until April 2022. Relevant unpublished studies and grey literature will be identified using Google, conference articles, shortlisted study reference lists, index-related articles on PubMed and existing relevant reviews.

The following search strategy with related key words was developed: (extended-spectrum beta-lactamase OR multidrug-resistan* OR extensively drug-resistan* OR antimicrobial-resistan* OR antibiotic-resistan* OR antibacterial-resistan* OR pandrug-resistan* OR carbapenem-resistant* OR colistin-resistant* OR polymyxin-resistant* OR methicillin-resistant* OR vancomycin-resistant*) AND (Acinetobacter baumannii OR Pseudomonas aeruginosa OR Escherichia coli OR Klebsiella pneumoniae OR Enterobacteriaceae OR Staphylococc* OR Enterococc* OR microorganism* OR bacteria) AND ((prediction model* OR predicted model* OR predictive model* OR risk model* OR risk prediction OR predicted factor* OR predictive factor* OR prognostic model* prognosis model* OR prognostic factor* OR scoring model*) AND (critical care OR intensive care unit* OR critical illness OR ICU OR intensive care OR critically ill) in English and (耐药OR耐抗生素OR耐细菌OR耐甲氧西林OR耐碳青霉烯OR耐万古霉素OR超广谱β内酰胺酶OR耐粘菌素OR耐多粘菌素) AND (细菌 OR 微生物OR肠杆菌OR肠球菌OR鲍曼不动杆菌OR铜绿假单胞菌OR肺炎克雷伯菌OR金黄色葡萄球菌 OR 结核分枝杆菌) AND (预测模型OR预警模型OR预判模型OR判别模型OR风险模型OR风险预测OR 风险评估OR预测因素 OR 评分模型OR评分系统) AND (ICU OR 重症监护 OR 监护室) in Chinese. We will use medical subject headings and free-text to identify prediction model studies. The search methods for databases are included in online supplemental file 2).

Eligibility criteria

Studies will be included in this review if they are primary experimental or observational studies on the development and/or validation of a multivariable prediction model for MDRO colonisation or infection in critically ill patients and were published any time before April 2022. Population, intervention, comparator, outcomes, timing and setting characteristics²³ are presented in table 1. Additional eligibility criteria include the use of comparative study designs such as clinical trials, cohort, case–control and cross-sectional studies and published in the English or Chinese language.

Table 1.

Eligibility criteria for the systematic review framed using the PICOTS^* system

Item	Definition
Population	Both male and female adult critically ill patients (aged ≥18 years) will be considered. The exclusion criteria are as follows: (1) ICU duration less than 24 hours; (2) MDROs detected before the patient entered the ICU or within the first 48 hours in the ICU.
Intervention	Any prediction model which predicts the risk of MDRO colonisation or infection in patients with critical illness, to distinguish critically ill patients with poor outcomes (who will develop multidrug-resistant bacterial infection), with reporting of at least two predictors will be included. Any disease caused by MDRO will be included. All types of MDROs, including MRSA, CRE, VRE, ESBL-EKP or others will be included.
Comparator	Not applicable.
Outcomes	The outcome (to be predicted) is MDRO cultured from any of the clinical specimens after 48 hours of admission to the ICU. MDRO infection is defined as the invasion of the body tissues by MDROs resulting in disease. Infectious diseases included but are not limited to bacteremia, pneumonia and infections of the skin and soft issue, urinary tract, bloodstream or abdomen. The legal communicable disease diagnostic criteria approved by countries or international organisations were applied to diagnose these infectious diseases. MDRO colonisation is defined as any patient who had MDRO positive culture results and with no symptoms of clinical infection found.
Timing	Predictive variables measured at any time point during the course of the MDRO colonisation or infection while patients were being treated in the ICU.
Setting	Any type of ICU.

CRE, carbapenem-resistant Enterobacteriaceae; ESBL-EKP, extended-spectrum β-lactamase-producing Enterobacteriaceae; ICU, intensive care unit; MDROs, multidrug-resistant organisms; MRSA, methicillin-resistant Staphylococcus aureus; PICOTS, population intervention, comparator, outcomes, timing of prediction and of outcomes and setting; VRE, vancomycin-resistant Enterococcus.

We will exclude studies using the following criteria: (1) conference abstracts, editorials, clinical case reviews, letters, commentaries, book chapters or systematic reviews; (2) studies involving other types of patients who are not critically ill; (3) studies on the associations between clinical variables and MDRO colonisation or infection and (4) studies in which the study setting was in the community.

Study selection

We will remove record duplicates using the automatic replay function in NoteExpress software and by-hand assessments after each database search. Two researchers (YW and YX), trained at the Joanna Briggs Institute, will independently screen the titles and abstracts to identify relevant studies using NoteExpress software. Full-text manuscripts will be retrieved and independently evaluated for inclusion. Disagreements will be resolved by consensus or a third-researcher (CY or FW) arbitration. The selection process will be presented in a PRISMA flow diagram (figure 1).

Figure 1.

PRISMA 2020 flow diagram for new systematic reviews which includes searches of databases, registers and other sources. *Consider, if feasible to do so, reporting the number of records identified from each database or register searched (rather than the total number across all databases/registers). **If automation tools were used, indicate how many records were excluded by a human and how many were excluded by automation tools. From: Page et al³⁸ PRISMA, Preferred Reporting Items for Systematic reviews and Meta-Analyses.

Data abstraction

At least two trained reviewers (Yi W and QY) will independently extract data from included studies. A standardised data extraction form will be created based on the Checklist for Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies (the CHARMS checklist).²⁴ This data extraction form will be piloted on five papers and amended by at least three reviewers (Yi W, FW and YX). Any revisions will be implemented based on group consensus. Data on the following study characteristics will be extracted: first author, year of publication, study design and characteristics, source of data, participant eligibility, recruitment, description and sample size, type of ICU, the number and/or incidence of predicted outcomes, the type of MDRO, infectious diseases, candidate predictors, missing data, modelling method and evaluation, risk ratios or ORs for the predictors (both overall and stratified), model performance and calibration (eg, calibration plot and Hosmer-Lemeshow test), discriminating capacity (eg, area under the receiver operating characteristic curve and Concordance Index) and model evaluation (eg, sensitivity, specificity, positive and negative predictive values), as well as the study discussion/conclusions. If the data of interest are missing or unclear, we will refer to any cited papers and contact corresponding authors to obtain the desired information. Disagreements will be resolved by consensus between the two reviewers or by arbitration by a third (CY or FW) researcher. The lead investigator (CY) will upload the data and records on a shared secure platform accessible to all investigators (Baidu Netdisk, Baidu Netcom Technology Corporation, Beijing, China).

Critical appraisal

Two reviewers (Ying W and YX) will independently appraise each included prediction model using the PROBAST instrument, a tool for assessing the risk of bias and applicability of diagnostic and prognostic prediction model studies, which was published in 2019.^{25 26} As well as serving clinical medical personnel who are considering using a prediction model, it is also used to help researchers develop a model or include models in a systematic review or meta-analysis.²⁷ In recent years, PROBAST has been used in systematic reviews of infection prediction models, like COVID-19 infection,²⁸ but unfortunately, it has not been fully applied in prediction models of MDRO colonisation or infection. PROBAST is widely used for quality evaluation of prediction models, therefore, this study will use this tool for critical appraisal. PROBAST includes four steps, which are described in detail to support assessment completion. The four domains are as follows: participants, predictors, outcome and analysis, and are divided into a total of 20 questions to support structured risk of bias assessments. Each domain is rated as at a ‘high’, ‘low’ or ‘unclear’ risk of bias. Any disagreement will be resolved by consensus and consultation with a third reviewer (CY/Yi W).

Statistical analysis

We will also produce a narrative summary of the included studies. A summary of the characteristics (eg, study design, population size, national location, year, participants’ characteristics, species of bacteria and statistical method) will be included. Counts and percentages will be used to describe categorical outcome data and risk of bias assessment findings. Continuous data, including sample size and predictor count, will be presented using means and SD, and medians and IQRs, for normally and non-normally distributed variables, respectively.

Meta-analytical methods will be used where data pooling is suitable. We will follow the recently published framework for the meta analysis of prediction models.^{23 29} We will group study results according to the species of bacteria (eg, MDRO, CRE and CRKP). To pool prediction findings from models developed for different strains of drug-resistant bacteria, a random-effects model will be used to obtain a summary estimate of model performance and calibration. As validation studies per model are likely to be few, they will be analysed using the C-statistic and 95% CIs in a random-effect models based on the restricted maximum likelihood estimation method.³⁰ Finally, 95% prediction intervals, which account for heterogeneity, will be assessed to provide a predicted range of C-statistic values to be used for reference by future validation studies. Heterogeneity will be calculated with the χ² test and I² test (<25%, low heterogeneity; 25%–50%, moderate heterogeneity and >50%, strong heterogeneity).³¹ A funnel plot will be generated to assess publication bias if more than 10 studies are included in a meta-analysis. All statistical analyses will be performed using R Statistical Software V.3.2.3 (R Core Team, R Foundation for Statistical Computing, Vienna, Austria) and Stata V.15.0 (Stata Corporation, College Station, TX, USA). We will use the R package ‘metamisc’ for the meta-analysis of prediction models, which is available from https://CRAN.R-project.org/package=metamisc.

Patient and public involvement

No patient or member of the public will be involved in the design, conduct, or reporting of this systematic review.

Ethics and dissemination

Ethical approval will not be required because this systematic review will be based on data extracted from previous studies. We plan to publish our findings in peer-reviewed journals dedicated to critical care medicine or nursing research. We also plan to present our results at the International Council of Nurses and at other conferences relevant to critical care.

Amendments

This systematic review protocol will be amended during the peer-review process.

Discussion

The rates of infections caused by MDROs (eg, MRSA, CRE, VRE and ESBL-EKP) are increasing. These infections lead to poor outcomes in critically ill patients.^32–35 Several models predicting MDRO infection have been developed,18–21 36 37 potentially supporting infection control and prevention measures. To the best of our knowledge, one systematic review has evaluated the evidence on models predicting ESBL colonisation or infection.¹⁷ This previous systematic review included studies published before April 2018 and focused on ESBL-EKP infection or colonisation. In contrast, this proposed systematic review has a broader scope, including all MDRO colonisation or infections acquired in the ICU, and will interrogate five English-language and three Chinese-language databases, as well as grey literature to ensure comprehensive coverage. There is a strong research team and sufficient time to ensure literature screening, quality evaluation and data extraction. Owing to the complex and scattered influencing factors, we will package the similarity factors, and conduct a meta-analysis to draw valuable conclusions, which will be completed with the help of a statistician and an evidence-based expert. This review will contribute to the understanding of the risk of MDRO colonisation or infection among critically ill patients. This review may also support evidence-based approaches to infection control and prevention that do not involve antibiotic use, helping improve outcomes.

Ethics statements

Patient consent for publication

Not required.