Accuracy of signs, symptoms and blood tests for diagnosing acute bacterial rhinosinusitis and CT-confirmed acute rhinosinusitis in adults: protocol of an individual patient data meta-analysis

Abstract

Introduction

This protocol outlines a diagnostic individual patient data (IPD) meta-analysis aimed at developing simple prediction models based on readily available signs, symptoms and blood tests to accurately predict acute bacterial rhinosinusitis and CT-confirmed (fluid level or total opacification in any sinus) acute rhinosinusitis (ARS) in adults presenting to primary care with clinically diagnosed ARS, target conditions associated with antibiotic benefit.

Methods and analysis

The systematic searches of PubMed and Embase of a review on the accuracy of signs and symptoms for diagnosing ARS in ambulatory care will be updated to April 2020 to identify relevant studies. Authors of eligible studies will be contacted and invited to provide IPD. Methodological quality of the studies will be assessed using the Quality Assessment of Diagnostic Accuracy Studies-2 tool. Candidate predictor selection will be based on knowledge from existing literature, clinical reasoning and availability. Multivariable logistic regression analyses will be used to develop prediction models aimed at calculating absolute risk estimates. Large unexplained between-study heterogeneity in predictive accuracy of the models will be explored and may lead to either model adjustment or derivation of separate context-specific models. Calibration and discrimination will be evaluated to assess the models’ performance. Bootstrap resampling techniques will be used to assess internal validation and to inform on possible adjustment for overfitting. In addition, we aim to perform internal–external cross-validation procedures.

Ethics and dissemination

In this IPD meta-analysis, no identifiable patient data will be used. As such, the Medical Research Involving Humans Subject Act does not apply, and official ethical approval is not required. Findings will be published in international peer-reviewed journals and presented at scientific conferences.

PROSPERO registration number

PROSPERO CRD42020175659.

Strengths and limitations of this study.

• Combining individual patient data (IPD) from multiple diagnostic accuracy studies allows for more statistical power and therefore more precise predictions.

• The study will be conducted according to the most recent guidance on the use of IPD meta-analyses of diagnostic modelling studies.

• Some potentially useful predictors may not be available in >50% of the included studies.

• There might be differences in measurement of index and reference standard tests between studies.

Introduction

Acute rhinosinusitis (ARS) is a very common infection and an important reason for consultations by adults in ambulatory care.^{1 2} The condition places a huge burden on society; it has been associated with impaired quality of life for patients and with high direct and indirect costs.³

Clinicians have to rely on clinical signs and symptoms such as nasal discharge or congestion, facial pain or pressure, and tooth pain for diagnosing ARS.³ However, these symptoms overlap considerably with other conditions, making accurate diagnosis challenging.

Despite most sinus infections being viral in origin⁴ and antibiotics having only marginal beneficial effects in otherwise healthy adults with uncomplicated ARS,^{5 6} antibiotic prescribing rates vary from around 50% in the Netherlands to up to 90% in the UK and the USA.^{2 7 8} Importantly, ARS is one of the conditions with highest antibiotic overprescribing rates,^{1 9} thereby exposing patients to avoidable side effects¹⁰ and the population to emerging antimicrobial resistance.¹¹

With antimicrobial resistance posing an increasing threat to public health globally,¹² effective strategies to encourage judicious use of antibiotics in common infections such as ARS are urgently needed. So far, educational programmes focusing on targeted or delayed antibiotic prescribing in primary care and patient information strategies have been effective in clinical studies,^13–15 but adoption of these strategies in day-to-day practice has proven challenging with antibiotic use in adults remaining at high levels^{2 16 17}

Among other reasons, clinicians tend to prescribe antibiotics because they feel unable to recognise patients that are most likely to benefit from antibiotics.¹⁸ Unfortunately, a meta-analysis of individual patient data (IPD) of 10 randomised controlled trials (RCTs) on the effectiveness of antibiotics in patients with clinically diagnosed ARS failed to identify signs and symptoms that predicted antibiotic benefit.⁵ Identification of specific signs and symptoms predictive of acute bacterial rhinosinusitis (ABRS) diagnosis has therefore been recognised as an important unmet research need.³ Placebo-controlled trial evidence indicates that this target condition is indeed associated with antibiotic benefit in adult outpatients. First, antibiotics are associated with improved cure rates in adult patients with clinically diagnosed ARS and elevated values of C reactive protein (CRP) or erythrocyte sedimentation rate (ESR),¹⁹ which in turn have been associated with positive bacterial culture of antral fluid.²⁰ Second, antibiotics lead to significantly faster and better recovery than placebo (number needed to treat to benefit=3) in those with evidence of a fluid level or total opacification in any sinus on CT,²¹ which in turn has been shown highly predictive of pus or mucopus by antral puncture.²² Importantly, antibiotics do not lead to improved recovery compared with placebo in outpatient adults with clinically diagnosed ARS and mucosal thickening but without fluid level or total opacification on CT scan.²³ This latter finding is consistent with those from a CT study of the common cold showing that 27 of 31 patients (87%) who had a CT performed within a few days after symptom onset had abnormalities of one or both maxillary sinuses, which were almost exclusively related to mucosal thickening.²⁴

Routine CT scanning or antral puncture in a common and generally self-limiting condition such as ARS is however neither feasible nor cost-effective. However, a simple model based on readily available signs, symptoms and blood tests that can accurately predict ABRS or CT-confirmed (fluid level or total opacification in any sinus) ARS has the potential to help clinicians identify patients most likely to benefit from antibiotics and thereby substantially reduce inappropriate antibiotic prescribing in this common condition. While previous work has indicated that such models are promising,²⁵ recent insights into sample size calculations for studies aimed at developing binary logistic prediction models^26–28 reveal that the sample sizes of the individual studies in this field^{22 29–32} do not reach the minimum sample size required to ensure precision of the model predictions and minimise the risk for overfitting. This problem can however be overcome when combining IPD from these studies. In this paper, we describe the protocol of a diagnostic IPD meta-analysis aimed at developing such prediction models.

Methods

The protocol is reported according to the Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols statement.³³ The IPD meta-analysis will follow the guidance on the use of IPD meta-analyses of diagnostic and prognostic modelling studies³⁴ and will be reported according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement for diagnostic test accuracy studies (PRISMA-DTA) and the PRISMA-IPD statement.^{35 36}

Patient and public involvement

Patients were not involved in development of the protocol.

Identification of relevant studies: a systematic review

A systematic review will be performed to identify and select any relevant studies. Study authors of eligible papers will be contacted and invited to provide IPD of their original study.

Study eligibility criteria

All studies on diagnostic accuracy of readily available signs, symptoms and blood tests for diagnosing ARS using a fluid level or total opacification in any sinus on CT scan and/or bacterial culture of antral fluid and/or bacterial culture of endoscopically guided middle meatal aspiration or swabs^{37 38} as the reference standard tests (table 1) will be eligible. Study participants must be adults (≥15 years) suspected by their physician of having uncomplicated ARS based on clinical signs and symptoms and presenting in a primary healthcare setting (ie, non-referred patients consulting their physician for the first time or patients presenting through self-referral in ambulatory care or an emergency department). No language restrictions will be applied. Studies involving children (<15 years), referred patients, hospitalised patients as well as those involving highly specialised populations (eg, those with immunodeficiency, odontogenic sinusitis or malignancy) will be excluded. To avoid verification bias, only studies in which all patients received the reference standard tests of interest (ie, CT scan, culture of endoscopically guided middle meatal aspiration or swabs or culture of fluid from antral puncture as part of clinical work-up) will be included.

Table 1.

Definition of index and reference standard tests

Reference standard tests	Definition
ABRS	Bacterial culture (any method) of fluid from antral puncture.
	Bacterial culture (any method) of endoscopically guided middle meatal aspiration.
	Bacterial culture (any method) of endoscopically guided middle meatal swabs.
CT-confirmed ARS	Presence of fluid level or total opacification in any sinus on CT scan.
Reference standard tests	Definition	Anticipated probability of outcome risk
Preceding URTI	History of preceding URTI as reported by patients or based on medical record.	Increase25
Previous diagnosis of ARS	History of previous ARS episode as reported by patients or based on medical record.	Decrease25
Maxillary pain	Pain (any, unilateral or bilateral) in maxillary sinus region as reported by patients.	Increase25
Pain in teeth	Pain in teeth as reported by patients.	Increase4 25
Anosmia	Loss of smell as reported by patients.	Increase25
Cacosmia	Sensation of bad smell as reported by patients.	Increase4
Double sickening	Worsening of symptoms after initial improvement of symptoms (‘two phases in the illness history’) as reported by patients or clinicians.	Increase4
Clinical impression	Overall clinical impression of illness as reported by the clinician.	Increase4
Purulent nasal discharge on examination	The presence of purulent nasal discharge on rhinoscopia anterior or endoscopy.	Increase4 25
CRP	CRP levels from blood samples collected by vena puncture (laboratory analysis) or fingerprick (using validated POCT devices).	Increase25
ESR	ESR levels from blood samples collected by vena puncture (laboratory analysis) or fingerprick (using validated POCT devices).	Increase20
Duration of illness (>10 days)	Symptom duration of more than 10 days as reported by patients.	Unknown*
Fever (>38°C)	Presence of fever (body temperature above 38°C) as reported by patients or measured by clinician.	Unknown*
Severe pain	Pain score >8/10 or equivalent scores using other pain rating scales as reported by patients.	Unknown*

*Candidate predictor selection based on clinical reasoning.

ABRS, acute bacterial rhinosinusitis; ARS, acute rhinosinusitis; CRP, C reactive protein; CT, computed tomography; ESR, erythrocyte sedimentation rate; POCT, point-of-care test; URTI, upper respiratory tract infection.

Search strategy

First, the reference list of a recent systematic review on the accuracy of signs and symptoms for diagnosing ARS in ambulatory care using any reference standard⁴ will be reviewed for relevant studies. Next, to increase the yield of potentially relevant studies, the systematic searches of PubMed and Embase of this review (online supplemental file 1) will be updated from 1 January 2015 to 1 April 2020. In addition, reference lists of all eligible studies as well as those from relevant systematic reviews will be screened for any further potential studies. Contributing review authors, experts in the field and the task force from the Danish Medicine Council working on the diagnosis and treatment of ARS will be asked if they have, or are aware of, any additional (published or unpublished) studies. Where necessary, individual study authors will be contacted.

Study selection

First, two review authors (RV and MHE) will independently review the reference list of the recent systematic review⁴ for eligible papers; where relevant, full texts will be retrieved. Next, two review authors (RV and MHE) will independently screen titles and abstracts of the unique records obtained from the electronic database searches and the reference lists of the relevant systematic reviews to assess their potential relevance for reviewing the full text. The same review authors will independently review the full text of all potentially eligible articles against the predefined inclusion and exclusion criteria. Any disagreements will be resolved by discussion.

Data extraction and management

Authors of eligible studies will be contacted via an email outlining the main study aim and inviting them to collaborate and share their deidentified, complete IPD of their original study in their preferred format. A data sharing agreement will be provided. Study data will be considered unavailable when none of the authors respond to multiple contact attempts or if all contacted study authors indicate they do not have access to the requested data and are unable to provide contact details of a colleague who does have access.

Retrieved IPD of individual studies will be checked by comparing key variables (eg, number of participants, descriptive analysis of demographic characteristics, index tests and reference standard test) with published data. We will contact collaborators for clarification if important discrepancies are identified. The level of missing data within each study will be assessed and discussed with collaborators with the aim to reduce missing data as much as possible. Finally, missing data will be investigated and appropriate methods for handling them, such as multiple imputation, will be considered.³⁹

All IPD data to be included in the meta-analysis will be converted to SPSS V.25 and R (R Foundation for Statistical Computing, Vienna, Austria). A database will be created containing study ID, patient ID, index tests (readily available signs, symptoms and blood tests including CRP, ESR and reference standard tests. Any critical differences in coding and/or measurement of index tests or the reference standard tests between studies will be discussed with the entire research team to decide how to resolve this issue most adequately. Harmonised data from the individual studies will be aggregated into single IPD sets with a study ID variable to identify participants from the same study. The aggregated database will be rechecked for accuracy.

Quality assessment of included studies

Two review authors (RV and MHE) will independently assess the methodological quality of the included studies using the Quality Assessment of Diagnostic Accuracy Studies-2 framework adapted to our research question.⁴⁰ This tool assess internal validity (risk of bias) and external validity (concerns about applicability) in four domains: patient selection, index test, reference standard test and flow and timing. Any disagreements will be resolved by discussion. If information regarding study quality is unclear or undisclosed, individual study authors will be contacted.

Data synthesis

Descriptive analysis and evidence synthesis

Study and participants’ characteristics will be provided for all studies that contribute IPD. Characteristics of eligible studies that did not contribute IPD will be considered to assess whether there is any evidence of selection bias.

IPD meta-analysis

Sample size considerations

Traditionally, sample size calculations for studies aiming to develop binary logistic prediction models use the widely known ‘10 events per variable’ rule of thumb.⁴¹ The validity of this rule of thumb has however been debated,²⁶ and recent research revealed alternative approaches to calculate the minimum sample size required when primary interest is in prediction accuracy.^{27 28}

An interactive sample size calculation tool (https://mvansmeden.shinyapps.io/BeyondEPV/) is now available to ensure predictive accuracy of a new prediction model with ≤30 candidate predictors.²⁴ Anticipating IPD availability for at least 375 participants (from two studies using CT scan as reference standard test)^{22 31} with an outcome proportion of 0.5 and the mean absolute prediction error set at 0.054, six candidate predictors could be included in our final model.

Alternatively, it has been suggested to identify the sample size and number of candidate predictors that minimises the amount of desired shrinkage (≤10%) during model development.²⁸ Sample size calculation using this approach requires prespecification of: (1) the number of candidate predictors, (2) outcome proportion and (3) the model’s anticipated Cox-Snell R-squared statistic, which reflects the signal:noise ratio (value between 0–1 with lower values indicating more difficulties in detecting patterns thereby increasing the potential for overfitting that ultimately leads to higher required sample sizes and fewer candidate predictors to be estimated reliably).²⁸ Anticipating IPD availability for at least 375 participants^{22 31} and an anticipated Cox-Snell R-squared statistic of 0.15, six candidate predictors can be included in our final model to target an expected shrinkage of 0.9 (which reflects an events per candidate predictor parameter of 27.4 assuming an outcome proportion of 0.5).²⁸

Selection of candidate predictors

Candidate predictors will be based on knowledge from existing literature and clinical reasoning. Previous research indicated that the following variables most likely predict the likelihood of ABRS or CT-confirmed ARS: (1) preceding upper respiratory tract infection, (2) previous diagnosis of ARS, (3) maxillary pain (any, unilateral and bilateral), (4) pain in teeth, (5) purulent nasal discharge on examination, (6) anosmia, (7) cacosmia, (8) overall clinical impression, (9) double sickening, (10) CRP and (11) ESR (table 1).4 20 22 25 32 42 Where possible, we will add the following, clinically sensible, predictors to the models to investigate their added diagnostic value: duration of illness (>10 days), fever (>38°C) and severe pain (table 1).

Continuous variables (CRP and ESR) will ideally be kept continuous and the functional relationship with the outcome will be assessed through restricted cubic splines.

Model development

The aggregated IPD dataset will be used to inform which predictors will be included in our multivariable logistic regression models.⁴³ Reasons for dropping variables will be based on non-availability in >50% of the studies and critical between-study heterogeneity in measuring a variable.

If potential relevant differences in how the reference standard has been performed between studies remain unsolved, we will explore whether this leads to differences in regression coefficients and the performance of models between studies. Meaningful heterogeneity in results will be reported and incorporated in our modelling strategy (eg, stratified models). Large unexplained between-study heterogeneity in predictive accuracy of the models will also be explored and discussed and may ultimately lead to either adjustment of the models (by dropping the variables that contribute most to the observed heterogeneity) or derivation of separate context-specific models.

The overall ability of the models to discriminate between patients with and without ABRS or abnormal CT finding will be quantified using the c-statistic. Calibration of the models will be assessed graphically with calibration plots.

Since an initial prediction model commonly shows a too optimistic discrimination and calibration, that is, overfitting, we will use bootstrap resampling techniques for internal validation and, where appropriate, to obtain an average shrinkage factor to adjust regression coefficients for overfitting. In addition, we aim to perform an internal–external cross-validation procedure⁴⁴ to estimate models’ diagnostic performance outside the context of the acquired IPD; every study will be left out once for validation of the models on the remaining studies. The final models will based on the entire pooled dataset and will generate continuous risk estimates. The final models will be presented in nomograms to allow for user-friendly decision making at an individual level⁴⁵ using Shiny web applications. We aim to use net benefit and decision curves to explore the clinical value of the models.⁴⁶

In exploratory analyses, we aim to apply other statistical approaches such as classification and regression trees, artificial neural networks, and fast and frugal trees to explore whether they generate additional insight.

Ethics and dissemination

There will be no identifiable patient data in any of the datasets. As such, the Medical Research Involving Humans Subject Act (WMO) does not apply to this study. The Medical Ethics Review Committee Utrecht, the Netherlands, reviewed the study protocol (protocol 20–331/C) and concluded that an official approval was not required.

All principal investigators will provide written confirmation that all participants included in the original studies had given full written informed consent. Data will be stored within a secured folder of the data management department of the Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht. On completion of the study, data will be stored for at least 15 years at a central drive of the data management department of the Julius Center and will be made available for the use by third parties on request and approval of the research team.

Research findings will be published in international peer-reviewed journals and will be presented at national and international scientific conferences.

Review registration and anticipated end date of study

The review has been registered in PROSPERO; CRD42020175659. The anticipated end date of data collection is January 2021, and the anticipated end date of the study is August 2021.

Discussion

With antibiotic overprescribing for ARS in adults estimated at 51%–54%,^{1 9} continuing efforts are needed to substantially reduce inappropriate antibiotic overprescribing in this condition. Tailored guidance based on identification of subgroups that are most likely to benefit from antibiotics has the potential to do so; such guidance for clinicians⁴⁷ led to reductions in inappropriate antibiotic prescribing for childhood acute otitis media with current overprescribing estimated at 4%.⁹ For ARS, such evidence-based guidance is still lacking,⁵ which leaves clinicians with a feeling of uncertainty and thereby creating a situation in which overuse of diagnostic tools and medications commonly occur.⁴⁸

The recently updated European Position Paper on Rhinosinusitis and Nasal Polyps has highlighted the most urgent unmet research needs in ARS. Among further RCTs evaluating the effectiveness of the full range of medications in ABRS, research into combinations of symptoms and signs predictive of ABRS has been recognised as top priority.³ The proposed diagnostic IPD meta-analysis aims to fill this gap by developing simple multivariable models based on readily available signs, symptoms and blood tests that can accurately predict ABRS or a fluid level or total opacification in any sinus on a CT scan in adults presenting to primary care with ARS, target conditions associated with antibiotic benefit.^19–21 Previous work from single studies indicated that such models are promising,²⁵ but combining IPD from multiple diagnostic accuracy studies allows for more statistical power and therefore more precise predictions. An initial scoping exercise also revealed that most of the candidate predictors were measured consistently across the individual studies that facilitate meta-analysis. Using state-of-the art diagnostic IPD meta-analysis methods, this project will ultimately reveal whether simple and easy-to-implement models can be derived to accurately predict ABRS or abnormal CT findings in adults (≥15 years) presenting to primary care with suspected, uncomplicated ARS to help clinicians to tailor antibiotic prescribing in this very common condition. A recent paper debunked common myths about risk threshold for prediction models.⁴⁵ To avoid inappropriate risk stratification of patients, we will calculate continuous risk estimates. Where appropriate, we aim to derive context-specific and clinical sensible risk thresholds by exploring test and treatment thresholds for ARS in a subsequent project.^{49 50}